# Two-dimensional double-kagome-lattice nitrogene: a direct band gap
semiconductor with nontrivial corner state

Wenzhang Li  College of Physics and Electronic Engineering, Center for
Computational Sciences, Sichuan Normal University, Chengdu, 610068, China  Qin
He  College of Physics and Electronic Engineering, Center for Computational
Sciences, Sichuan Normal University, Chengdu, 610068, China  Xiao-Ping Li
School of Physical Science and Technology, Inner Mongolia University, Hohhot
010021, China  Da-Shuai Ma  Institute for Structure and Function $\&$
Department of Physics, Chongqing University, Chongqing 400044, China  Botao Fu
<EMAIL_ADDRESS>College of Physics and Electronic Engineering, Center
for Computational Sciences, Sichuan Normal University, Chengdu, 610068, China

###### Abstract

Based on first-principles calculations, we predict that nitrogen atoms can
assemble into a single-layer double kagome lattice (DKL), which possesses the
characteristics of an intrinsic direct band gap semiconductor, boasting a
substantial band gap of 3.460 eV. The DKL structure results in a flat valence
band with high effective mass and a conduction band with small effective mass
comes from Dirac electrons. These distinctive band edges lead to a significant
disparity in carrier mobilities, with electron mobility being four orders of
magnitude higher than that of holes. The presence of flat band in DKL-
nitrogene can be further discerned through the enhanced optical absorption and
correlated effects as exemplified by hole-induced ferromagnetism.
Interestingly, DKL-nitrogene exhibits inherent second-order topological
states, confirmed by a non-trivial second Stiefel-Whitney number and the
presence of 1D floating edge states and 0D corner states within the bulk band
gap. Additionally, the robust N-N bonds and the lattice’s bending structure
ensure thermodynamic stability and mechanical stiffness. These attributes make
it exceptionally stable for potential applications in nano-devices.

##  I Introduction

In recent years, the exploration of two-dimensional (2D) element (Xenes)
beyond graphene has sparked a revolutionary shift in materials science and
condensed matter physicsBhimanapati  _et al._ (2015); Molle  _et al._ (2017);
Xie  _et al._ (2021); Mannix  _et al._ (2017). These atomically thin
substances exhibit exceptional properties, offering a wealth of opportunities
for novel electronic and optoelectronic devices. Among these 2D materials, the
group-VA XenesZhang  _et al._ (2018); Khan  _et al._ (2021); Xia  _et al._
(2019) (nitrogene, phosphoreneLi  _et al._ (2014); Liu  _et al._ (2014); Zhu
and Tománek (2014), arseneneMardanya  _et al._ (2016); Zhong  _et al._ (2018);
Kamal and Ezawa (2015, 2015), antimoneneShao  _et al._ (2018); Wu  _et al._
(2017); Wang  _et al._ (2015), and bismutheneLu  _et al._ (2014); Liu  _et
al._ (2011); Drozdov  _et al._ (2014)) that crystal in buckled hexagonal or
puckered rectangular lattice have garnered considerable attention due to their
intrinsic semiconducting band gapZhang  _et al._ (2016), high carrier
mobilityQiao  _et al._ (2014) and strongly anisotropic propertiesXia  _et al._
(2014). Despite the considerable progress in understanding 2D materials,
research concerning elemental compounds containing nitrogen remains remarkably
scarceFrost  _et al._ (2016). This scarcity is surprising given nitrogene
critical importance in various chemical and biological processes. The dearth
of studies exploring nitrogen-based single-element materials has hindered the
comprehensive understanding of their potential and intriguing properties.

Structure determines property, among diverse 2D crystal structures, the kagome
lattice holds particular signifcance. Kagome-formatted materials are
characterized by a lattice arrangement resembling a woven basket, possessing
fascinating flat-band and Dirac electron that set them apart from other 2D
crystalsLin  _et al._ (2018); Liu  _et al._ (2020); Kang  _et al._ (2020).
This unique electronic structure gives rise to a range of interesting
phenomena such as topological quantum state and many-body physicsYin  _et al._
(2018); Zhang  _et al._ (2020); Yin  _et al._ (2022), making them an enticing
avenue for further research. Motivated by the lack of research on nitrogen-
based single-element materials and the alluring properties of kagome-
structured materials, we endeavor to introduce a novel nitrogen-based
monolayer with a kagome-related lattice and explore the realm of this
fascinating material and unravel its extraordinary electronic and topological
characteristics.

In this work, we theoretically propose a kind of monolayer nitrogen that
crystals in a double kagome lattice (DKL), which we term DKL-nitrogene.
Notably, this material features Dirac electron in the conduction band and a
flat band in the valence band. This giant electron-hole asymmetric in the band
structure results in is a remarkable diversity in carrier mobilities for DKL-
nitrogene. The impact from the flat-band is further revealed in DKL-nitrogene
that it not only enhances optical absorption in the ultraviolet region but
also induces half-metal ferromagnetism under proper hole doping. Furthermore,
DKL-nitrogene is confirmed to be a second-order topological insulator, hosting
non-zero second Stiefel-Whitney number, 1D floating edge states, and 0D corner
states. Thus, DKL-nitrogene holds great promise for applications in
electronics and optoelectronics with the potential to unlock exotic quantum
phenomena and inspire future research in the field of 2D single-element
materials.

##  II Calculation method

The first-principles calculations are performed based on density functional
theory (DFT) and implemented in Vienna ab initio simulation package
(VASP)Kresse and Hafner (1993, 1994); Kresse and Furthmüller (1996, 1996) with
projector-augmented-wave (PAW)Blöchl (1994) and Perdewe-Burke-Ernzerhof
(PBE)Perdew  _et al._ (1996) exchange correlation potential. Additionally, the
HSE06 hybrid functionalsKrukau  _et al._ (2006) is considered for better
description of the band gap and optical absorption. A cut-off energy of 500 eV
and a $k$-mesh of $15\times 15\times 1$ are selectedMonkhorst and Pack (1976).
The convergence criteria for total energy and force are set at $10^{\small-6}$
eV and -0.001 eV/Å, respectively. The phonon spectrum is calculated using the
PHONOPY packageTogo and Tanaka (2015). The tight-binding (TB) Hamiltonian was
constructed via MagneticTB codeZhang  _et al._ (2023), and the topological
edge states were calculated based on Wanniertool codeWu  _et al._ (2018).

##  III Atomic structure and stability of DKL-nitrogene

In contrast to other pnictogens, nitrogen typically remains in its gaseous
state and only forms the nonmolecular solid phase under extreme conditions.
Recently, a monolayer nitrogen with buckled honeycomb structureÖzçelik  _et
al._ (2015) was predicted stable at room temperature, referred as “nitrogene”.
As depicted in Fig. 1(a), this hexagonal nitrogene features three-coordinated
A/B sublattices with a buckling height ($h$) of 0.70 Å, resembling blue
phosphorene. Theoretically, a standard kagome lattice represents the line
graph of the honeycomb latticeZhang  _et al._ (2019), which can be achieved by
placing an atom (red crossing in Fig. 1(a)) at the midpoint of each side of
the regular hexagon. However, the arrangement of atoms in the kagome lattice
violates the typically triple-coordination bonding rule of pnictogens,
resulting in structural instability. To restore the triple-coordination
bonding, each atom can be substituted with a binary component (A@B), leading
to the formation of a double kagome lattice (DKL)Cai  _et al._ (2023) or
diatomic kagome latticeLiu  _et al._ (2021), as illustrated in Fig. 1(b). This
DKL can also be directly obtained from the honeycomb lattice by simply
replacing every N-atom with a N-trimer. Interestingly, such DKL structures
have been discovered in several covalent organic frameworksJiang  _et al._
(2021) but have rarely been discussed in elemental materialsChen  _et al._
(2018); Zhu  _et al._ (2020).

In the optimized DKL-nitrogene, each N-atom remains representative of a three-
coordination of group-VA elements with two different bonding lengths
($l_{1}$,$l_{2}$) and bonding angles (${\theta}_{1}$, ${\theta}_{2}$) as shown
in Fig. 1(b). The optimized lattice constant of DKL-nitrogene is about 4.04 Å,
with intra-trimer and inter-trimer bonding length of 1.48 Å and 1.53 Å,
respectively, which is similar to that of $hex$-nitrogeneÖzçelik  _et al._
(2015). The intra-trimer angle ${\theta}_{1}$ is constrained to $60^{\circ}$
and the inter-trimer angle ${\theta}_{2}$ is about $111^{\circ}$. The buckling
height between top and bottom N-atoms is about 1.40 Å, significantly larger
than that of $hex$-nitrogene (0.70 Å), due to strong $sp^{3}$ hybridization.
It’s worthwhile that the crystal structure of DKL-nitrogene also bears a
resemblance to that of black phosphorene when viewed from the side, sharing a
similar puckered geometry.

Figure 1:  (a) Top and side views of $hex$-nitrogene. (b) Top and side views
of DKL-nitrogene. The A and B sublattices are located at different heights,
distinguished by blue and purple colors, respectively. The red crosses at the
bonding centers between two N-atoms form a typical kagome lattice. (c)
Electron local function diagram (ELF) of DKL-nitrogene. (d) Phonon dispersion
of DKL-nitrogene.

By visualizing the electron local function (ELF) in Fig. 1(c), a robust
covalent bonding nature between N-N atoms becomes evident, and the presence of
a lone pair electron on the N-atom is distinctly illustrated. The dynamic
stability of DKL-nitrogene is unequivocally affirmed by the phonon spectrum
illustrated in Fig. 1(d), wherein no imaginary frequencies are observed
throughout the entire Brillouin zone. Additionally, the thermal and mechanical
stability of DKL-nitrogene are firmly established through ab initio molecular
dynamics simulations in Fig. S1 and adhering to Born’s mechanical stability
criteriaHieu  _et al._ (2022) as elaborated in Tab. S1, and corresponding
Young’s modulus and Poisson’s ratio are shown in Fig. S3 in the supplementary
information.

##  IV Electronic structure of DKL-nitrogene

Figure 2: (a) Orbital-resolved band structure and density of states (DOS) of
DKL-nitrogene. (b) 3D view of the HVB and LCB of DKL-nitrogene. (c) Band
structure from TB model with parameters: $e_{0}=-0.32$, $t_{1}=3.0$,
$t_{2}=2.86$, $t_{5}=-0.18$. Here, $t_{n}$ denotes the $n$th order hopping
parameter. (d) Charge density of the HVB. (e) Charge density of the LCB.

The electronic band structure of DKL-nitrogene is determined through
calculations incorporating the HSE06 correction, as depicted in Fig. 2(a).
Unlike its hexagonal allotrope, which is recognized as an indirect insulator
(5.900 eV, HSE06)Özçelik  _et al._ (2015), DKL-nitrogene exhibits a distinct
behavior as a promising direct band gap semiconductor, with both the valence
band maximum (VBM) and conduction band minimum (CBM) at the $\Gamma$ point.
The calculated band gap is approximately 3.460 eV at the HSE06 level. Due to
its substantial direct band gap, DKL-nitrogen holds potential utility within
the visible and ultraviolet spectral ranges. When examining the orbital-
projected band structure, it becomes apparent that the highest valence band
(HVB) and the lowest conduction band (LCB) are predominantly composed of
$p_{z}$ orbitals, with minor contributions from $p_{x,y}$ and $s$ orbitals.
Intriguingly, despite the similar orbital composition, HVB and LCB exhibit
strikingly different dispersion behaviors. As clearly illustrated in Fig. 2(b)
(in a 3D view), the former conforms to a conventional conduction band pattern
with typically parabolic dispersion, while the latter shows nearly flat
dispersion with a small bandwidth of 0.590 eV.

To further elucidate the potential mechanism underlying this phenomenon, we
have plotted the 3D charge density for the HVB and the LCB along with 2D
cross-sectional views, as shown in Figs. 2(d) and 2(e), respectively. In the
case of the HVB, the three intra-trimer N-atoms form in-plane anti-bonding
states, with reduced charge density between them, while two inter-trimer
N-atoms form out-of-plane bonding states. Remarkably, these inter-trimer
bonding centers, as displayed in the 2D cross-sectional view, actually give
rise to a typical kagome lattice pattern, ultimately leading to the emergence
of a flat band. Conversely, for the LCB, the three intra-trimer N-atoms engage
in in-plane bonding states, resulting in trefoil-like charge patterns, while
two inter-trimer N-atoms form out-of-plane anti-bonding states with reduced
charge density between them. As a whole, the centers of the two trefoils on
the top and bottom planes create a buckled honeycomb lattice. This lattice
configuration can account for massive LCB, including the presence of the Dirac
cone at the K point.

Because the band edges are mainly formed by $p_{z}$-orbital, a simple single-
orbital toy model can be written as:

$\displaystyle H=\sum_{i,j}t_{n}c^{\dagger}_{i}c_{j}+h.c.$ (1)

, where the $c^{\dagger}_{i}$ and $c_{j}$ are the creation and annihilation
operators at sites $i$ and $j$, respectively. The $t_{n}$ is the $n$-th order
hopping term. In the supplementary information (Fig. S2), we extensively
discussed the impact of various hopping parameters on the band structure. By
fitting these parameters, we obtained the band structure of DKL-nitrogene, as
illustrated in Fig. 2(c). Six $p_{z}$ orbitals on the DKL form two sets of
bandsSethi  _et al._ (2023). One set (conduction band) involves two coupled
bands that cross with each other at $K$ point with the energy minimum point at
$\Gamma$ point whereby the band is quite dispersive. The other set (valence
band) involves four coupled bands that form a hourglass-like structure where
two middle bands cross each forming Dirac cone at $K$, which is further
sandwiched by two flat bands from the top and bottom sides. The weak
dispersion of flat highest valence band is dominated by 5-$th$ order hopping
term ($t_{5}$) in our TB model.

##  V Anisotropic and electronic-hole asymmetric carrier mobility

The carrier mobility of a material greatly affects its applicability as an
electronic device. With approximation of longitudinal acoustic wave, the
phonon-related carrier mobility of 2D structure can be evaluated by a general
formulaQiao  _et al._ (2014):

$\displaystyle\mu_{x,y}=\frac{e\hbar^{3}C_{x,y}}{k_{B}Tm_{x,y}^{*}(m_{x}^{*}m_{y}^{*})^{1/2}D_{x,y}^{2}},$
(2)

the $m_{x,y}^{*}$ represents the effective mass of carriers, $C_{x,y}$ stands
for the modulus of elasticity, and $D_{x,y}$ is the deformation potential
constant. Here the subscript $x$ and $y$ denote the zigzag and armchair
directions, respectively, and the temperature is set at 300 K. Thereinto, the
deformation potential is defined as,

$D_{x,y}=\frac{\partial E_{n\bm{k}}}{\partial{\varepsilon_{x,y}}},$ (3)

where $E_{n\bm{k}}$ represents the electronic eigenvalue for a band with index
$n$ and wave vector $\bm{k}$ at VBM or CBM, while ${\varepsilon_{x,y}}$
denotes to the uniaxial strain along $x$ or $y$ direction. The deformation
potential characterizes the energy shift in response to external strain
perturbations, primarily accounting for phonon scattering from breathing
modes. The effective mass $m^{*}$ is obtained by fitting band dispersion with
formula:

$\frac{1}{m_{x,y}^{*}}=\frac{1}{\hbar^{2}}\frac{\partial^{2}E_{n\bm{k}}}{\partial
k_{x,y}^{2}}.$ (4)

The modulus of elasticity is calculated as,

$C_{x,y}=\frac{1}{S_{0}}\frac{\partial^{2}E}{\partial\large\varepsilon_{x,y}^{2}},$
(5)

where $E$ is the total energy, and $S_{0}$ is the area of the unit cell.

Carriers | $m_{x}^{*}/m_{0}$ | $m_{y}^{*}/m_{0}$ |  $D_{x}$(eV) |  $D_{y}$(eV) |  $C_{x}$(Jm-2) |  $C_{y}$(Jm-2) |  $\mu_{x}$(cm2V-1s-1) |  $\mu_{y}$(cm2V-1s-1)
---|---|---|---|---|---|---|---|---
Electron | 0.34 | 0.34 | 1.30 | 1.42 | 95.15 | 94.78 | 10390.00 | 8674.21
Hole | 15.30 | 16.90 | 2.96 | 2.93 | 95.15 | 94.78 | 0.94 | 0.87
Table 1: Effective mass $m^{*}_{x,y}$, deformation potential constant
$D_{x,y}$, elastic modulus $C_{x,y}$ and carrier mobility $\mu_{x,y}$ of DKL-
nitrogene, where $x$ and $y$ represent the transport direction. The $m_{0}$ is
the free-electron mass.

Based on the calculated effective mass ($m^{*}$), elastic modulus constants
($C_{x,y}$), and deformation potential constant ($D_{x,y}$) for the CBM and
VBM at $\Gamma$ point, we have determined the mobilities of both electrons and
holes, as presented in Table. 1. One of the most striking features of the
mobility is the substantial asymmetry between electrons and holes, with
electron mobility being four orders of magnitude greater than hole mobility.
For instance, electron mobility ($\mu_{x}$/$\mu_{y}$) reaches a substantial
value of 1.04$\times$104/0.87$\times$104 cm2V-1s-1, while hole mobility is
only 0.94/0.87 cm2V-1s-1. We attribute this asymmetry in carrier mobility
primarily to the intrinsic electron-hole asymmetry within the unique band
structure. This band structure displays a dispersive conduction band with a
Dirac cone, resulting in a small effective mass ($m_{x}^{*}$/$m_{y}^{*}$) for
electrons (0.34/0.34$m_{0}$). In contrast, the valence band is exceedingly
flat, leading to a large effective mass for holes (15.30/16.90$m_{0}$), as
listed in Table. 1. Furthermore, aside from the difference in effective mass,
the deformation potential ($D_{x}/D_{y}$) of electrons is only 1.30/1.42 eV,
notably smaller than that of holes (2.96/2.93 eV). Therefore, the significant
contrast in effective mass and in deformation potential between electrons and
holes ultimately result in a substantial disparity in carrier mobility, as
clearly indicated by Eq. (2). This pronounced difference in carrier mobility
has the potential to enhance the efficiency of separating photo-generated
electrons and holesLi  _et al._ (2013), opening up broad applications in
electronic devices, optoelectronics, and photocatalysis.

##  VI Enhanced optical absorption of DKL-nitrogene

Based on the linear response theory and independent particle
approximationGajdoš  _et al._ (2006a), we calculate the frequency-dependent
dielectric function
${\varepsilon(w)}={\varepsilon_{1}(w)}+i{\varepsilon_{2}(w)}$. The
${\varepsilon_{2}(w)}$ represents the imaginary part of the dielectric
function that can be derived from the inter-band optical transition, and the
${\varepsilon_{1}(w)}$ is the real part of dielectric function that can be
derived from ${\varepsilon_{2}(w)}$ according to Kramers-Kronig
relationshipGajdoš  _et al._ (2006b). To avoid the influence from the vacuum
layer, it’s more reasonable to calculate the 2D optical absorbance $A(w)$ from
corresponding optical conductivity $\sigma_{2D}(w)$Matthes  _et al._ (2014,
2016). Thereinto, the 2D optical conductivity is derived from the
corresponding 3D counterpart through the equation,

$\sigma_{2D}(w)=L\sigma_{3D}(w),$ (6)

where $L$ represents the total length of unitcell along out-plane direction,
and the $\sigma_{3D}(w)$ is derived from DFT obtained dielectric function of
2D system with sufficient evacuum layer as

$\sigma_{3D}(w)=i[1-\varepsilon(w)]\varepsilon_{0}w,$ (7)

where $\varepsilon_{0}$ is the permittivity of vacuum and $w$ is the frequency
of the incident wave. Then, the normalized absorbance $A(w)$ of a two-
dimensional material at normal incidence can be expressed as

$A(w)=\frac{Re\widetilde{\sigma}}{|1+\widetilde{\sigma}/2|^{2}},$ (8)

where $\widetilde{\sigma}=\sigma_{2D}(w)/\varepsilon_{0}c$ is the normalized
2D optical conductivity ($c$ is the speed of light).

To elucidate the impact of the unique electronic structure on the optical
absorption coefficient, we conducted comparative calculations for all
predicted monolayer nitrogenes, including DKL-nitrogene,
$hex$-nitrogeneÖzçelik  _et al._ (2015), $oct$-nitrogeneZhang  _et al._ (2015)
and ZS-nitrogeneBondarchuk and Minaev (2017), whose crystal structure and band
structure are given in Fig. S4 in supplementary information. Remarkably, DKL-
nitrogene demonstrates significantly enhanced optical absorption in the range
from 200 nm to 600 nm. In particular, the DKL-nitrogene exhibits much larger
optical absorption capabilities in the near-ultraviolet range (200-370 nm),
when compared to nitrogen allotropes. Only ZS-nitrogene displays larger
optical absorption within the 370-600 nm range due to its smaller band gap
(2.451 eV). Furthermore, considering the diverse band gaps and band edge
characteristics of these four materials, we specifically compared their
optical absorption near the band edge, as illustrated in the inset of Fig.
3(a). Within 35 nm range, we observed that DKL-nitrogene exhibits the highest
absorption coefficient at the band edge position compared to the other three
structures. This phenomenon can be attributed to its unique band edge
electronic structures. Specifically, in DKL-nitrogene, the flat valence band
provides a high electron density, contributing to substantial light absorption
at the band edge.

##  VII Flat-band ferromagnetism induced by hole doping in DKL-nitrogene

The kagome lattice stands out due to its uniquely flat band structure, which
offers a platform for exploring intriguing many-body phenomena. For instance,
the valence band observed in DKL-nitrogene contributes to an extremely large
DOS just below Fermi level as depicted in Fig. 3(b). By manipulating hole
concentrations within this context, it is possible to potentially induce
electronic instabilities. These instabilities, in turn, could lead to various
outcomes, such as magnetic phase transitions or Wigner crystallizationChen
_et al._ (2018). To further verify this concept, our study delves into the
behavior of the spin magnetic moment and spin polarization energy as a
function of increasing hole density per unit cell ($n$), as presented in Fig.
3(c). Remarkably, the spin magnetic moment emerges even at exceedingly low
doping densities and quickly reaches a saturation value of 1.0 $\mu_{B}$. This
value remains constant up to $n$=1.2, after which it gradually diminishes to
zero. The spin polarization energy, defined as the energy difference between
the ferromagnetic (FM) and non-magnetic (NM) states, exhibits a prominent
positive peak across the entire range of doping levels. This peak strongly
suggests a pronounced energy preference for the ferromagnetic state.

The emergence of itinerant magnetism induced by doping can be effectively
understood through the lens of the Stoner criterionCao  _et al._ (2015):

$N(E_{\text{F}})I_{xc}>1,$ (9)

where $N(E_{\text{F}})$ represents the DOS at the Fermi energy in the non-
magnetic (NM) state, and $I_{xc}$ is the Stoner parameter. The Stoner
parameter can be approximated using the relationshipHortamani  _et al._ (2008)
$I_{xc}=\Delta_{xc}/\mathbf{M}$, where $\Delta_{xc}$ signifies the energy of
exchange splitting between the two spin channels, and $\mathbf{M}$ denotes the
total magnetic moment. In the case of DKL-nitrogene, $I_{xc}\simeq 0.65$.
Evidently, the condition $N(E_{\text{F}})I_{xc}>1$ is satisfied across the
entire filling range of the flat band, as illustrated in Fig. 3(b). Notably,
the saturation magnetic moment of 1 B per hole indicates a 100$\%$ spin
polarization, confirming the presence of half-metallic states. For instance,
when considering $n$=1.2, this state exhibits the most substantial spin
polarization energy, quantified at 65 meV per hole. As illustrated in Fig.
3(d), it becomes evident that only the spin-up band crossing the Fermi level
leads to the formation of a half-metallic ferromagnetic (FM) state.

Figure 3: (a) The optical absorption coefficient for DKL-nitrogene,
$hex$-nitrogene, $oct$-nitrogene and ZS-nitrogene. Inset is the absorption
coefficient around band gap. (b) Total DOS of DKL-nitrogene around Fermi
level. (c) Spin polarization energy and spin magnetic moment versus hole
doping concentration ($n$) for DKL-nitrogene. (d) Spin-polarized band
structures under $n$=1.2 per unitcell.

##  VIII Second-order topological insulator with edge and corner states

In the realm of 2D materials, a diverse range of compounds boasting
substantial band gaps are systematically categorized into second-order
topological insulators (SOTIs)Benalcazar  _et al._ (2017a, b). Setting them
apart from conventional 2D topological insulators that typically exhibit 1D
conductive edge states bridging the conduction and valence bands, these 2D
SOTIs require no SOC and uniquely harbor 1D gapped edge state as well as 0D
corner states. These remarkable corner states emerge specifically when these
materials are engineered into quantum dots with precisely tailored
geometriesSheng  _et al._ (2019); Liu  _et al._ (2019); Qian  _et al._ (2021).

Figure 4: (a) The Wilson loop of DKL-nitrogene. The number of crossing on
$\theta$=$\pi$ is 1, therefore the $w_{2}$ is 1. (b) The energy spectrum of 1D
nanoribbon with type-A edge configuration. The mid gap edge state is marked by
blue color. (c) The energy spectrum of 0D hexagonal flake (N=1152) composed of
type-A edges. The grey, blue and red dots stand for the bulk, edge and corner
states, respectively. (d) The charge spatial distribution of corner states.

To characterize SOTIs, a novel topological invariant known as the second
Stiefel-Whitney number ($w_{2}$) has been introducedAhn  _et al._ (2019); Zhao
and Lu (2017). For the DKL-nitrogene with inversion symmetry, $w_{2}$ can be
calculated directly using parity eigenvalues as follows:

$\displaystyle(-1)^{w_{2}}=\prod_{i=1}^{4}(-1)^{{\lfloor}N_{occ}^{-}(\Gamma_{i})/2{\rfloor}},$
(10)

where $N_{occ}^{-}$ denotes the number of occupied bands with odd parity at
the time reversal momentum ($\Gamma_{i}$), and ${\lfloor}{\rfloor}$ is the
floor function. The calculated $w_{2}$=1 for DKL-nitrogene indicates that it
is a 2D SOTI. Moreover, this non-zero second Stiefel-Whitney number is also
confirmed by the nested Wilson loop spectrumWieder and Bernevig (2018), as
shown in Fig. 4(a), where the spectrum exhibits a crossing point at $k_{x}=0$,
$\theta=\pi$, indicating $w_{2}=1$.

To unveil the non-trivial edge states and corner states, two types of
boundaries (type-A and type-B) with different atomic geometries are chosen, as
shown in Fig. 1(a). Firstly, for the 1D ribbon along the type-A direction,
only one branch of floating edge state exists in the middle of the bulk band
gap, as displayed in Fig. 4(b). Next, we explore the 0D corner state for a
finite-sized hexagonal flake (N=1152) containing six type-A edge
configuration. The discrete energy spectrum of DKL-nitrogene flake is shown in
Fig. 4(c), with bulk, edge, and corner states differentiated by the gray,
blue, and red colors, respectively. Above the edge states, there are six
degenerate corner states related by the $C_{6}$ symmetry, with their charge
being equally localized on the six corners of the hexagonal flake, as clearly
shown in Fig. 4(d). Therefore, considering doping this flake with integer
electrons to these 6-fold-degenerate corner statesHu  _et al._ (2022), each
corner will hold a fractionally quantized corner charge of $\frac{e}{6}$. On
the other hand, if we construct a hexagonal flake composed of uniform type-B
edges (N=1098), there exist three groups of six-degenerate corner states (see
Fig. S8 in the supplementary information). One group sits at zero energy
between the edge states, while the other two groups sit above and below the
edge states.

##  IX Discussion and conclusion

The distinctive structure of 2D materials enables us to finely manipulate
their electronic structures using external electric, light, and strain fields.
As a result, we have delved deeper into the manipulation of the band structure
and light absorption of DKL-nitrogene through strain engineering in Figs.
S5-S7 in supplementary information. We initially unveiled that DKL-nitrogene
can endure a maximum uniaxial strain of 10$\%$, a threshold comparable to that
of other two-dimensional materials in Fig. S5. Moreover, it exhibits an out-
of-plane negative Poisson’s ratio, reminiscent of black phosphorene’s
behaviorJiang and Park (2014). More importantly, the value of band gap can be
significantly controlled by strain while maintaining its intrinsic direct
band-gap characteristics in Fig. S6. Through incremental tensile strain, we
observed the band gap decreasing from 3.460 eV to 2.310 eV. Concurrently, the
peak absorption wavelength shifted from the near-ultraviolet range to the
visible light spectrum in Fig. S7. These exceptional and adjustable optical
characteristics form the basis for potential applications of DKL-nitrogene in
the field of optoelectronic devices.

In conclusion, we present a stable nitrogen-based monolayer structure
characterized by a unique double kagome lattice configuration, offering a
platform with intriguing properties. We unveil the asymmetric nature of the
parabolically dispersing conduction band and the nearly flat valence band is
revealed. Furthermore, we explore the implications of this flat-band
structure, emphasizing its role in enhancing optical absorption and
facilitating the emergence of flat-band-induced ferromagnetism. Beyond this,
we also uncover the underlying higher-order topology present in DKL-nitrogene
by calculating the second Stiefel-Whitney number and demonstrating the
existence of edge and corner states. With its exceptional mechanical, optical,
and topological attributes, the DKL-nitrogene may pave the way for possible
applications in various fields, particularly nano-device technology.

###### Acknowledgements.

This work is supported by the National Natural Science Foundation of China
(NSFC, Grants No. 12304086, No. 12204330, No. 12204074), and Dr. B. Fu also
the Sichuan Normal University for financial support (Grant No. 341829001). The
numerical computations were performed at the Hefei advanced computing center,
and this research was also supported by the High Performance Computing Center
of Sichuan Normal University.

## References

  * Bhimanapati  _et al._ (2015) G. R. Bhimanapati, Z. Lin, V. Meunier, Y. Jung, J. Cha, S. Das, D. Xiao, Y. Son, M. S. Strano, V. R. Cooper, _et al._ , ACS nano 9, 11509 (2015).
  * Molle  _et al._ (2017) A. Molle, J. Goldberger, M. Houssa, Y. Xu, S.-C. Zhang, and D. Akinwande, Nat. Mater. 16, 163 (2017).
  * Xie  _et al._ (2021) Z. Xie, B. Zhang, Y. Ge, Y. Zhu, G. Nie, Y. Song, C.-K. Lim, H. Zhang, and P. N. Prasad, Chem. Rev. 122, 1127 (2021).
  * Mannix  _et al._ (2017) A. J. Mannix, B. Kiraly, M. C. Hersam, and N. P. Guisinger, Nat. Rev. Chem. 1, 0014 (2017).
  * Zhang  _et al._ (2018) S. Zhang, S. Guo, Z. Chen, Y. Wang, H. Gao, J. Gómez-Herrero, P. Ares, F. Zamora, Z. Zhu, and H. Zeng, Chem. Soc. Rev. 47, 982 (2018).
  * Khan  _et al._ (2021) K. Khan, A. K. Tareen, Q. U. Khan, M. Iqbal, H. Zhang, and Z. Guo, Mater. Chem. Front. 5, 6333 (2021).
  * Xia  _et al._ (2019) F. Xia, H. Wang, J. C. Hwang, A. C. Neto, and L. Yang, Nat. Rev. Phys. , 1 (2019).
  * Li  _et al._ (2014) L. Li, Y. Yu, G. J. Ye, Q. Ge, X. Ou, H. Wu, D. Feng, X. H. Chen, and Y. Zhang, Nature Nanotech. 9, 372 (2014).
  * Liu  _et al._ (2014) H. Liu, A. T. Neal, Z. Zhu, Z. Luo, X. Xu, D. Tománek, and P. D. Ye, ACS nano 8, 4033 (2014).
  * Zhu and Tománek (2014) Z. Zhu and D. Tománek, Phys. Rev. Lett. 112, 176802 (2014).
  * Mardanya  _et al._ (2016) S. Mardanya, V. K. Thakur, S. Bhowmick, and A. Agarwal, Phys. Rev. B 94, 035423 (2016).
  * Zhong  _et al._ (2018) M. Zhong, Q. Xia, L. Pan, Y. Liu, Y. Chen, H.-X. Deng, J. Li, and Z. Wei, Adv. Funct. Mater. 28, 1802581 (2018).
  * Kamal and Ezawa (2015) C. Kamal and M. Ezawa, Phys. Rev. B 91, 085423 (2015).
  * Shao  _et al._ (2018) Y. Shao, Z.-L. Liu, C. Cheng, X. Wu, H. Liu, C. Liu, J.-O. Wang, S.-Y. Zhu, Y.-Q. Wang, D.-X. Shi, _et al._ , Nano Lett. 18, 2133 (2018).
  * Wu  _et al._ (2017) X. Wu, Y. Shao, H. Liu, Z. Feng, Y.-L. Wang, J.-T. Sun, C. Liu, J.-O. Wang, Z.-L. Liu, S.-Y. Zhu, _et al._ , Adv. Mater. 29, 1605407 (2017).
  * Wang  _et al._ (2015) G. Wang, R. Pandey, and S. P. Karna, ACS Appl. Mater. Inter. 7, 11490 (2015).
  * Lu  _et al._ (2014) Y. Lu, W. Xu, M. Zeng, G. Yao, L. Shen, M. Yang, Z. Luo, F. Pan, K. Wu, T. Das, _et al._ , Nano Lett. 15, 80 (2014).
  * Liu  _et al._ (2011) Z. Liu, C.-X. Liu, Y.-S. Wu, W.-H. Duan, F. Liu, and J. Wu, Phys. Rev. Lett. 107, 136805 (2011).
  * Drozdov  _et al._ (2014) I. K. Drozdov, A. Alexandradinata, S. Jeon, S. Nadj-Perge, H. Ji, R. J. Cava, B. A. Bernevig, and A. Yazdani, Nat. Phys. 10, 664 (2014).
  * Zhang  _et al._ (2016) S. Zhang, M. Xie, F. Li, Z. Yan, Y. Li, E. Kan, W. Liu, Z. Chen, and H. Zeng, Angew. Chem.-Int. Edit 55, 1666 (2016).
  * Qiao  _et al._ (2014) J. Qiao, X. Kong, Z.-X. Hu, F. Yang, and W. Ji, Nat. Commun. 5, 4475 (2014).
  * Xia  _et al._ (2014) F. Xia, H. Wang, and Y. Jia, Nat. Commun. 5, 4458 (2014).
  * Frost  _et al._ (2016) M. Frost, R. T. Howie, P. Dalladay-Simpson, A. F. Goncharov, and E. Gregoryanz, Phys. Rev. B 93, 024113 (2016).
  * Lin  _et al._ (2018) Z. Lin, J.-H. Choi, Q. Zhang, W. Qin, S. Yi, P. Wang, L. Li, Y. Wang, H. Zhang, Z. Sun, L. Wei, S. Zhang, T. Guo, Q. Lu, J.-H. Cho, C. Zeng, and Z. Zhang, Phys. Rev. Lett. 121, 096401 (2018).
  * Liu  _et al._ (2020) Z. Liu, M. Li, Q. Wang, G. Wang, C. Wen, K. Jiang, X. Lu, S. Yan, Y. Huang, D. Shen, _et al._ , Nat. Commun. 11, 4002 (2020).
  * Kang  _et al._ (2020) M. Kang, L. Ye, S. Fang, J.-S. You, A. Levitan, M. Han, J. I. Facio, C. Jozwiak, A. Bostwick, E. Rotenberg, _et al._ , Nat. Mater. 19, 163 (2020).
  * Yin  _et al._ (2018) J.-X. Yin, S. S. Zhang, H. Li, K. Jiang, G. Chang, B. Zhang, B. Lian, C. Xiang, I. Belopolski, H. Zheng, _et al._ , Nature 562, 91 (2018).
  * Zhang  _et al._ (2020) S. S. Zhang, J.-X. Yin, M. Ikhlas, H.-J. Tien, R. Wang, N. Shumiya, G. Chang, S. S. Tsirkin, Y. Shi, C. Yi, Z. Guguchia, H. Li, W. Wang, T.-R. Chang, Z. Wang, Y.-f. Yang, T. Neupert, S. Nakatsuji, and M. Z. Hasan, Phys. Rev. Lett. 125, 046401 (2020).
  * Yin  _et al._ (2022) J.-X. Yin, B. Lian, and M. Z. Hasan, Nature 612, 647 (2022).
  * Kresse and Hafner (1993) G. Kresse and J. Hafner, Phys. Rev. B 47, 558 (1993).
  * Kresse and Hafner (1994) G. Kresse and J. Hafner, Phys. Rev. B 49, 14251 (1994).
  * Kresse and Furthmüller (1996) G. Kresse and J. Furthmüller, Comp. Mater. Sci. 6, 15 (1996).
  * Kresse and Furthmüller (1996) G. Kresse and J. Furthmüller, Phys. Rev. B 54, 11169 (1996).
  * Blöchl (1994) P. E. Blöchl, Phys. Rev. B 50, 17953 (1994).
  * Perdew  _et al._ (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Krukau  _et al._ (2006) A. V. Krukau, O. A. Vydrov, A. F. Izmaylov, and G. E. Scuseria, J. Phys. Chem. C 125 (2006), 10.1063/1.2404663.
  * Monkhorst and Pack (1976) H. J. Monkhorst and J. D. Pack, Phys. Rev. B 13, 5188 (1976).
  * Togo and Tanaka (2015) A. Togo and I. Tanaka, Scripta Mater. 108, 1 (2015).
  * Zhang  _et al._ (2023) Z. Zhang, Z.-M. Yu, G.-B. Liu, Z. Li, S. A. Yang, and Y. Yao, Comput. Phys. Commun. 290, 108784 (2023).
  * Wu  _et al._ (2018) Q. Wu, S. Zhang, H.-F. Song, M. Troyer, and A. A. Soluyanov, Comp. Phys. Commun. 224, 405 (2018).
  * Özçelik  _et al._ (2015) V. O. Özçelik, O. U. Aktürk, E. Durgun, and S. Ciraci, Phys. Rev. B 92, 125420 (2015).
  * Zhang  _et al._ (2019) S. Zhang, M. Kang, H. Huang, W. Jiang, X. Ni, L. Kang, S. Zhang, H. Xu, Z. Liu, and F. Liu, Phys. Rev. B 99, 100404 (2019).
  * Cai  _et al._ (2023) Y. Cai, J. Wang, Y. Wang, Z. Hao, Y. Liu, L. Zhou, X. Sui, Z. Jiang, S. Xu, H. Ge, X.-M. Ma, C. Zhang, Z. Shen, Y. Yang, Q. Jiang, Z. Liu, M. Ye, D. Shen, Y. Liu, S. Cui, L. Wang, C. Liu, J. Lin, B. Huang, L. Wu, J. Zhuang, H. He, W. Zhang, J.-W. Mei, and C. Chen, Adv. Electron. Mater. 9, 2300212 (2023).
  * Liu  _et al._ (2021) H. Liu, S. Meng, and F. Liu, Phys. Rev. Mater. 5, 084203 (2021).
  * Jiang  _et al._ (2021) W. Jiang, X. Ni, and F. Liu, Accounts Chem. Res. 54, 416 (2021).
  * Chen  _et al._ (2018) Y. Chen, S. Xu, Y. Xie, C. Zhong, C. Wu, and S. B. Zhang, Phys. Rev. B 98, 035135 (2018).
  * Zhu  _et al._ (2020) J. Zhu, C. He, Y.-H. Zhao, and B. Fu, J. Mater. Chem. C 8, 2732 (2020).
  * Hieu  _et al._ (2022) N. N. Hieu, H. V. Phuc, A. I. Kartamyshev, and T. V. Vu, Phys. Rev. B 105, 075402 (2022).
  * Sethi  _et al._ (2023) G. Sethi, M. Cuma, and F. Liu, Phys. Rev. Lett. 130, 186401 (2023).
  * Li  _et al._ (2013) R. Li, F. Zhang, D. Wang, J. Yang, M. Li, J. Zhu, X. Zhou, H. Han, and C. Li, Nat. Commun. 4, 1432 (2013).
  * Gajdoš  _et al._ (2006a) M. Gajdoš, K. Hummer, G. Kresse, J. Furthmüller, and F. Bechstedt, Phys. Rev. B 73, 045112 (2006a).
  * Gajdoš  _et al._ (2006b) M. Gajdoš, K. Hummer, G. Kresse, J. Furthmüller, and F. Bechstedt, Phys. Rev. B 73, 045112 (2006b).
  * Matthes  _et al._ (2014) L. Matthes, O. Pulci, and F. Bechstedt, New J. Phys. 16, 105007 (2014).
  * Matthes  _et al._ (2016) L. Matthes, O. Pulci, and F. Bechstedt, Phys. Rev. B 94, 205408 (2016).
  * Zhang  _et al._ (2015) Y. Zhang, J. Lee, W.-L. Wang, and D.-X. Yao, Comp. Mater. Sci. 110, 109 (2015).
  * Bondarchuk and Minaev (2017) S. V. Bondarchuk and B. F. Minaev, Comp. Mater. Sci. 133, 122 (2017).
  * Cao  _et al._ (2015) T. Cao, Z. Li, and S. G. Louie, Phys. Rev. Lett. 114, 236602 (2015).
  * Hortamani  _et al._ (2008) M. Hortamani, L. Sandratskii, P. Kratzer, I. Mertig, and M. Scheffler, Phys. Rev. B 78, 104402 (2008).
  * Benalcazar  _et al._ (2017a) W. A. Benalcazar, B. A. Bernevig, and T. L. Hughes, Science 357, 61 (2017a).
  * Benalcazar  _et al._ (2017b) W. A. Benalcazar, B. A. Bernevig, and T. L. Hughes, Phys. Rev. B 96, 245115 (2017b).
  * Sheng  _et al._ (2019) X.-L. Sheng, C. Chen, H. Liu, Z. Chen, Z.-M. Yu, Y. X. Zhao, and S. A. Yang, Phys. Rev. Lett. 123, 256402 (2019).
  * Liu  _et al._ (2019) B. Liu, G. Zhao, Z. Liu, and Z. F. Wang, Nano Lett. 19, 6492 (2019).
  * Qian  _et al._ (2021) S. Qian, C.-C. Liu, and Y. Yao, Phys. Rev. B 104, 245427 (2021).
  * Ahn  _et al._ (2019) J. Ahn, S. Park, D. Kim, Y. Kim, and B.-J. Yang, Chin. Phys. B 28, 117101 (2019).
  * Zhao and Lu (2017) Y. X. Zhao and Y. Lu, Phys. Rev. Lett. 118, 056401 (2017).
  * Wieder and Bernevig (2018) B. J. Wieder and B. A. Bernevig, arXiv.1810.02373 (2018), 10.48550/arXiv.1810.02373.
  * Hu  _et al._ (2022) T. Hu, T. Zhang, H. Mu, and Z. Wang, J. Phys. Chem. Lett. 13, 10905 (2022).
  * Jiang and Park (2014) J.-W. Jiang and H. S. Park, Nat. Commun. 5, 4727 (2014).


2023

]Department of Computer Science and Mathematics
Lebanese American University
Beirut 1102 2801, Lebanon

# AraSpell: A Deep Learning Approach for Arabic Spelling Correction

Mahmoud Salhab  Faisal Abu-Khzam  [

###### Abstract

Spelling correction is the task of identifying spelling mistakes, typos, and
grammatical mistakes in a given text and correcting them according to their
context and grammatical structure. This work introduces "AraSpell," a
framework for Arabic spelling correction using different seq2seq model
architectures such as Recurrent Neural Network (RNN) and Transformer with
artificial data generation for error injection, trained on more than 6.9
Million Arabic sentences. Thorough experimental studies provide empirical
evidence of the effectiveness of the proposed approach, which achieved 4.8%
and 1.11% word error rate (WER) and character error rate (CER), respectively,
in comparison with labeled data of 29.72% WER and 5.03% CER. Our approach
achieved 2.9% CER and 10.65% WER in comparison with labeled data of 10.02% CER
and 50.94% WER. Both of these results are obtained on a test set of 100K
sentences.

###### keywords:

Arabic spelling correction, Seq2Seq, Deep Learning, Neural Networks.

111Available on GitHub at: https://github.com/msalhab96/AraSpell

##  1 Introduction

In recent years, the advances of Natural Language Processing (NLP) methods,
often applied to machine translation, speech recognition, question answering,
among others, have led to a wide adoption of NLP solutions in various fields
like health care nlphealthcare , manufacturing Prakash2019-oy , finance
xing2018a …etc, mostly focusing on English, with scarce attention on the
Arabic language.

Arabic is the 4th most-used language on the internet, and it is one of the six
official languages of the United Nations. It is spoken by people across 22
countries article ; 6841973 . Arabic can be categorized into three main
variants: (i) Classical Arabic (CA), which is mainly used in ancient and
theological texts but is still understood due to its use in the Holy Quran,
(ii) the Modern Standard Arabic (MSA), which is a modernized and simplified
version of CA, and (iii) Dialectal/colloquial Arabic (DA), where each region
has its own dialect. The most widely-understood variation among Arabic
speakers is MSA due to its wide adoption in education, media, and formal
communication across the different Arabic-speaking countries ALAYYOUB2018522 .

Spelling correction is the task of identifying the incorrect words in a
sentence and then correcting them according to certain criteria. Spelling
errors can greatly impact the output of sophisticated natural language
processing (NLP) and natural language understanding (NLU) models, and it can
greatly degrade the accuracy of these models. For example, Napoles et al.
inproceedings  show how error injection could badly affect the performance of
a dependency parsing system.

Spelling correction modules are vital components for many real-world NLP
applications such as Optical Character Recognition (OCR) systems where
spelling correction is employed in the post-processing phase to possibly
correct any misspelled word DBLP:journals/corr/abs-1204-0191 . Moreover, it is
used for search query correction to ensure the quality of search-based systems
Rachidi_arabicuser .

Most of the recent work in the Arabic spelling correction field has relied on
traditional approaches such as statistical and rule-based approaches, that
usually consist of two cascaded systems, namely error detection, and
correction, where a word would pass to the correction system if and only if an
error gets detected Moslem2020ArabiscCN .

This paper presents an end-to-end framework for automatic Arabic spelling
correction using various Seq2Seq models trained on more than 6.9 million
sentences, starting from a large number of correct sentences and introducing
various levels of errors via a novel error-injection scheme. The paper is
structured as follows. Literature review is presented in the next section,
followed by a section describing the proposed model, and the data used.
Section 4 presents the experiments and their results and we conclude in
Section 5 with a summary and potential future directions.

##  2 Literature Review

Automatic spelling correction received considerable attention from the
research community. Kukich Kukich1992TechniquesFA  divides the process of
spelling correction into three steps: (i) error detection, (ii) correction
candidate generation, (iii) candidate ranking. A hierarchical character tagger
model presented in Gao2021HierarchicalCT  for short text spelling error
correction, where they used a pre-trained language model at the character
level as a text encoder, and then predict character-level edits to transform
the original text into its error-free form with a much smaller label space,
and for decoding they proposed a hierarchical multi-task approach to alleviate
the issue of long-tail label distribution without introducing extra model
parameters. A word vector/conditional random field (CRF)-based detector was
proposed in Wang2015WordVR  to detect Chinese spelling errors and used a
language model for re-scoring.

The Long Short-Term Memory model (LSTM) proposed in 8904218  encodes the input
word at the character level, which also uses word and POS tag contexts as
features for Indonesian text. Similarly, Kinaci2018SpellingCU  trained a
recurrent neural network with dictionary words. For a given misspelled word,
they retrieve a candidate list from that dictionary word, then the list gets
expanded using a character-level bi-gram model and the trained model.

For Arabic, a set of correction rules is proposed in Nawar2014FastAR  such
that these rules get ranked by calculating their probability before they get
applied to the input text. In rulebased1  Shaalan et al. proposed a tool
capable of recognizing and suggesting correction of ill-formed input for
common spelling errors. It is composed basically of Arabic morphological
analyzer, lexicon, spelling checker, and spelling corrector.

A system of two components introduced in 6193415 : one to retrieve candidates
for misspelled words using Damerau–Levenshtein, and another one to correct
spelling errors using A* lattice search and 3-grams language model. Similarly,
a two-stage system was used in paper1 , where first a misspelled word is
detected via a morphological analysis system. The detected word is classified
as incorrect if it does not have analysis, and the authors used different
correction candidate generation and then used a Naive-Bayes classifier to pick
the most likely to be the correction for the incorrect word. Moreover,
Attia2016-qm  used a look-up table and character-level language model for
misspelling detection and then used candidate generation and ranking. The
candidate then get selected using a language model.

Shaalan, Khaled spellchecking1  proposed a look-up table and language modeling
for error detection followed by a candidate generation that uses edit distance
measured by Levenshtein distance from the misspelled word, then a noisy
channel model trained on one-billion words with knowledge-based rules to
assign scores to the candidate corrections and choose the best correction
independent of the context.

A neural network model using bidirectional long short-term memory (bi-LSTM)
with a polynomial network (PN) for error detection is proposed in
10.1145/3373266 . In Moslem2020ArabiscCN , Moslem et al. introduced a many-to-
one neural network-based context-sensitive spelling checking and correction
model, where they modeled the words that come both before and after the word
to be corrected as the conditional context in language model predictions.
Finally, Abandah et al. abandah  used stacked Long Short-Term Memory (LSTM)
modules for common soft Arabic spelling errors correction with stochastic
error injection for a limited number of characters to capture limited frequent
mistakes.

##  3 Methodology

Automatic spelling correction systems detect a spelling error and propose a
set of candidates for correction. Building such a system using traditional
approaches - such as rule-based approaches - requires intensive work to build
and debug. The advantage of deep learning stems from its ability to discover
the underlying patterns in the data. Despite the fact that deep learning based
systems outperform traditional systems in many fields, such systems require an
intensive amount of labeled data to generalize well and to be able to be used
in production. In reality, having labeled data is rare, and the manual data
labeling process is very hard, costly, and requires domain experts to do.
Given the fact that unlabeled data is hugely available, the ability to do
auto-labeling would have a great advantage to building automatic spelling
correction systems. If we are confident that the unlabeled data from an
initial source is accurate and trustworthy, we can build a labeled corpus as
we describe in 3.1 then train the model on that data.

Figure 1 shows an overview on how our approach is structured. Our toolchain
starts with data gathering, followed by data quality assurance, data cleaning
and data labeling (in the specified order). Lastly, Modeling is applied where
we build and train different Seq2Seq model architectures.

Figure 1: An overview of AraSpell project structure.

We now present the details of our proposed approach to solving Arabic spelling
correction using various Seq2Seq models and a new self-labeling method.

###  3.1 Data

For this work we used the Arabic Wikipedia 2021 dump222The data set is
available on Kaggle here https://www.kaggle.com/datasets/z3rocool/arabic-
wikipedia-dump-2021, the data set contains 711230 articles in various domains.

####  3.1.1 Data Preprocessing

The data comes in the form of articles. Some articles may contain different
languages, so the can have encoding and formatting issues as well as invalid
symbols, and might have different lengths.

Given a set of articles $\boldsymbol{A}=\\{a_{1},a_{2},\dots,a_{H}\\}$ and set
of ordered transformation functions $\mathcal{F}=(f_{1},f_{2},\dots,f_{T})$
applied on each article in $\boldsymbol{A}$ resulting a set of clean lines
$\boldsymbol{L}$.

Where $\boldsymbol{L}=\bigcup\limits_{h=1}^{H}f_{1}(f_{2}(\dots
f_{T}(a_{h})\dots))$ such that each line in $\boldsymbol{L}$ has a reasonable
length, contains valid characters only, contains correct and valid Modern
Standard Arabic (MSA) words. The transformation functions built and used can
be categorized into two groups:

  1. 1.

Cleaning, cleansing, and mapping: The articles are lengthy, vary in length,
contain Arabic diacritization, some characters in the articles represented in
various uni-codes, and lastly many articles contain non-Arabic content, for
that each article split down into short paragraphs, Arabic diacritization
omitted, any character repeated more than twice truncated into two characters
only, characters normalized as table 1 shows, and only valid Arabic characters
are kept.

  2. 2.

Filtering: We filtered the lines by dropping any line that either contains
numbers, contains a floating character -floating character is a word in the
sentence that has a length of one-, contains any citation content, contains
more than two unique words -unique words are the words that appear only once
in the corpus-, contains less than 3 or more than 20 words, or the number of
characters in the line is less than 15 or more than 128 characters.

utf8

Arabic Character | Original Unicode | Normalized to |  |
---|---|---|---|---
لا | \ufefb | \u644\u627 |  |
لأ | \ufef7 | \u644\u623 |  |
لإ | \ufef9 | \u644\u625 |  |
لآ | \ufef5 | \u644\u622 |  |

Table 1: Arabic normalized uni-codes

####  3.1.2 Artificial Data Generation

For the task of spelling correction, as with many supervised learning tasks, a
pair of correct (target) and incorrect (input) sequences are required. In
order to generate the incorrect sequences for the correct ones, we propose an
error injection method that takes a clean sequence and injects noise into it.

Given a set of cleaned lines $\boldsymbol{L}=\\{l_{1},l_{2},\dots,l_{N}\\}$
our objective is to find
$\boldsymbol{\hat{L}}=\\{\hat{l}_{1},\hat{l}_{2},\dots,\hat{l}_{N}\\}$ such
that the $\hat{l}_{n}$ is the corrupted/distorted version of $l_{n}$. For a
given clean line $l_{n}$, let G be a set of predefined error injection
operations such that $G=\\{g_{1},g_{2},\dots,g_{M}\\}$ with:

$\hat{l}\textsubscript{n}=\mathcal{G}_{n}(l\textsubscript{n})$ (1)
$\mathcal{G}_{n}(l\textsubscript{n})=g\textsubscript{n,1}(g\textsubscript{n,2}(\dots
g\textsubscript{n,$\phi_{n}$}(l\textsubscript{n})\dots))$ (2)
$g\textsubscript{n,$\phi_{n}$}\sim unif(G)$ (3) $\phi\textsubscript{n}=\lfloor
J_{n}.\psi\rfloor$ (4)

Where $\phi_{n}$ is the number of error injection operations of the nth line,
$J_{n}$ is the length of the nth line, and $\psi$ is the corruption/distortion
ratio or Character Error Rate (CER).
Any error injection operation in G could work under one of the below modes:

  * •

Insertion: a random valid character gets inserted randomly at a random
position in the sentence.

  * •

Deletion: a random character or pattern in the sentence gets removed.

  * •

Substitution: a random character in the sentence gets replaced by either a
random character or one of its keyboard key neighbors.

  * •

Transposition of two adjacent characters: two adjacent characters get swapped
randomly.

  * •

Mapping: a certain predefined pattern in the sentence gets replaced by one of
the predefined targets.

###  3.2 Proposed Models

Given a set of examples
$\\{(x_{1},y_{1}),(x_{2},y_{2}),\dots,(x_{N},y_{N})\\}$ where xn and yn are
the nth corrupted text, and the clean/original text respectively. Let xn =
$\\{<SOS>,x_{n}^{1},\dots,x_{n}^{K},<EOS>\\}$ the input sequence, let yn =
$\\{<SOS>,y_{n}^{1},\dots,y_{n}^{L},<EOS>\\}$ the output sequence, and let the
vocabulary $\mathbb{V}$ =
$\\{aleph,baa^{\prime},\dots,yaa^{\prime},<space>\\}$, $x_{n}^{k}$,
$y_{n}^{l}\in\mathbb{V}$, where SOS is the start-of-sentence token, and EOS is
the end-of-sentence-token. We want to model each output character $y_{n}^{l}$
as a conditional distribution over the previous characters and the input
$x_{n}$ using the chain rule:

$\prod_{l=1}^{L}P(y\textsubscript{n}\textsuperscript{l}\mid
x\textsubscript{n},y\textsubscript{n}\textsuperscript{0:l-1})$ (5)

As in machine translation, in the task of spelling correction, the model’s
input sequence $x_{n}$ usually does not match the output sequence $y_{n}$:
$\mid x_{n}\mid<\mid y_{n}\mid$ when the number of deletions is higher than
the number of insertions, $\mid x_{n}\mid>\mid y_{n}\mid$ when the number of
insertions is higher than the number of deletions, or $\mid x_{n}\mid=\mid
y_{n}\mid$ in case of swapping, replacement, or when the number of insertions
equals the number of deletions. We shall therefore model the conditional
distribution in (5) using Seq2Seq encoder-decoder. For that we conducted a
wide range of experiments on different Seq2Seq model architectures such as
RNN-based encoder-decoder and Transformers.

####  3.2.1 Attentional vanilla Seq2Seq using RNN

The attentional vanilla Sequence-to-Sequence RNN model consists of two sub-
modules: encoder and decoder with attention. The encoder takes the input
sequence $\boldsymbol{X}$ and transforms it into a high-level representation
$\boldsymbol{h}=(h_{1},\dots,h_{K})$, while the decoder with attention takes
$\boldsymbol{h}$, the previously predicted character, and the last hidden
state and produces a probability distribution over character sequences:

$\boldsymbol{h}=Encoder(\boldsymbol{X})$ (6)

$P(y\textsubscript{n}\mid\boldsymbol{X})=DecoderWithAttention(\boldsymbol{h},y\textsubscript{n-1},c_{n})$
(7)

$c\textsubscript{n}=Concat(softmax(QK^{T})V,h_{n})W_{p}$ (8)

Where $c\textsubscript{n}$ is $n\textsuperscript{th}$ context vector, the key
$K=\boldsymbol{h}W\textsubscript{k}$, value
$V=\boldsymbol{h}W\textsubscript{v}$, and query
$Q=h\textsubscript{n}W\textsubscript{q}$ such that Wq , Wk, Wv, and Wp are
weight projection matrices $W\textsubscript{q}\in\mathbb{R}\textsuperscript{d,
h\textsubscript{size}}$, $W\textsubscript{k}\in\mathbb{R}\textsuperscript{d,
h\textsubscript{size}}$, $W\textsubscript{v}\in\mathbb{R}\textsuperscript{d,
h\textsubscript{size}}$, and
$W\textsubscript{p}\in\mathbb{R}\textsuperscript{h\textsubscript{size},
d+h\textsubscript{size}}$, where d is the projection size.

Figure 2 visualizes the Attentional vanilla Seq2Seq using RNN model.

Figure 2: Attentional vanilla Seq2Seq using RNN.

####  3.2.2 Attentional Seq2Seq with stacked RNN blocks

During experiments, we found that the vanilla Seq2Seq model described in 3.2.1
is highly affected by the corruption ratio. As the corruption increases, the
model convergence becomes harder and slower and the model struggles to find
the proper alignment so the attention collapse, which led to weak performance.
We solved this issues by employing encoder-decoder using RNN blocks with
attention instead of using stacked RNNs, where each RNN block consist of Gated
Recurrent Unit (GRU) layer followed by dropout, feed-forward module, and layer
normalization.

The feed-forward module consists of 2 fully connected layers such that the
first up-scale the feature space by a factor of 2 and the second one scales it
back to the original feature space dimension. The encoder and the decoder with
attention work the same way as illustrated in (6), (7), and (8). Figure 3
visualizes the Attentional Seq2Seq with stacked RNN blocks model.

Figure 3: Attentional Seq2Seq with stacked RNN blocks.

####  3.2.3 Transformer

Transformer transformer  has become the go-to architecture in most of the NLP
applications. Due its success in NLP, it has also been adopted in Computer
Vision (CV) cv1 ; cv2 , Automatic Speech Recognition (ASR) sr1 ; sr2 , Speech
Synthesis st1 ; st2 , among many others.

The transformer consists of an encoder where the input $\boldsymbol{X}$ gets
mapped into high-level features $\boldsymbol{h}$, and a decoder with multi-
head-attention that attends on $\boldsymbol{h}$ given the previously predicted
samples $\boldsymbol{y\textsubscript{<n}}$.

The most notable advantage of Transformers over RNNs is that RNNs are auto-
regressive in nature, which makes it difficult to take full advantage of
modern fast computing devices such as Tensor Processing Units (TPUs) and
Graphical Processing Units (GPUs). Another advantage stems from the limitation
of the RNNs when encountering a very long sequence.

We used the model in its original architecture as mentioned in transformer
except for the feed-forward module, instead of up-scaling the feature space we
downscale it by a factor of 2.

##  4 Experiments

We have conducted a wide range of experiments, starting from how the
performance changes while the error injection percentage changes to how
different model architectures perform under the same error injection rate,
taking the vanilla RNN model 3.2.1 as the baseline and improving based on it.

To test the models’ performance, we used Character Error Rate (CER), Word
Error Rate (WER), Character Error Reduction Rate (CERR), and Word Error
Reduction Rate (WERR):

$WER=\frac{S+D+I}{N}$ (9)

$ERR=\frac{\overline{ER}-\widehat{ER}}{\overline{ER}}$ (10)

Where $\boldsymbol{S}$ is the number of substitutions, $\boldsymbol{D}$ is the
number of deletions, $\boldsymbol{I}$ is the number of insertions,
$\boldsymbol{N}$ is the number of words in the reference, $\boldsymbol{ERR}$
is the error reduction rate, $\boldsymbol{\overline{ER}}$ is the
original/reference CER/WER, and $\boldsymbol{\widehat{ER}}$ is the CER/WER
after correction. CER can be calculated the same as in (9), but the main
difference is that it works on the character level.

###  4.1 Experimental Setup

Data sets of different corruption ratios $\psi$ have been generated. Mainly
5%, 10%, 5% and 10% combined, and varied ratios between 2.5% and 10% are used
as described in 3.1.2. For the Vanilla RNN model, we used 4 layers, a hidden
size of 256, and an embedding size of 512. For the attentional Seq2Seq with
stacked RNN blocks model, we used 3 layers, a hidden size of 256, and an
embedding size of 512. For both models, during the training process, we used
gradient clipping with 1.0 max norm of the gradients and Adam optimizer with
exponential learning rate (lr), decay (11) with an initial learning rate of
10-4 and decay rate of 15*10-4.

$lr(step)=\text{Initial learning rate}*(1-\frac{\text{Decay
rate}}{100})^{\text{step}}$ (11)

For the transformer model we used four layers, 512 model dimensionality and 8
heads. For training, Adam optimizer is used with learning rate scheduler as
mentioned in transformer  with 4000 warm-up steps.

We trained our models on one machine with two NVIDIA 3080 TI GPUs, and we used
Kullback–Leibler divergence loss for all models. During training, we employed
different regularization techniques. We used dropout dropout  with a 10%
dropout ratio and label smoothing of value $\epsilon$=0.1 labelsmoothing .
During the training process of the vanilla RNN model, the model struggled to
build an alignment when the corruption ratio increased from $\psi$=5% to
$\psi$=10%, and it took 20X times to converge and build alignment. To achieve
that, we used a simple trick to speed the process up by pre-training the model
on low corruption ratio ($\psi$=5%) for few steps till the model learns to
build a proper alignment, then we retrained it on the data of high corruption
ratio.

###  4.2 Results

A test set of 100K sentences have been used. In each experiment, we tested the
WER, CER, WERR, CERR on $\psi$=5% and $\psi$=10%. Table 2 shows the CER and
WER results, while table 3 shows the original CER and WER calculated on the
original corrupted test data. Lastly, Table 4 shows the WERR, and CERR. We can
see that the transformer model outperforms all other models under all data
conditions. From tables 2 and 4 we can see that adding more data would help
the performance, as mixing the 5%, and the 10% data together yields better
performance across all models.

Model | CER (%) | WER (%)
---|---|---
$\psi=5\%$ | $\psi=10\%$ | $\psi=5\%$ | $\psi=10\%$
Transformer_0.05 | 1.24 | 4.15 | 5.35 | 18.38
Transformer_0.1 | 1.45 | 2.82 | 5.95 | $\boldsymbol{10.36}$
Transformer_mixed | $\boldsymbol{1.11}$ | $\boldsymbol{2.8}$ | $\boldsymbol{4.8}$ | 10.65
Transformer_varied | 1.22 | 3.16 | 5.41 | 12.35
RNNB_0.05 | 1.74 | 4.45 | 7.76 | 18.8
RNNB_0.1 | 1.86 | 4.01 | 7.8 | 15.41
RNNB_mixed | 1.67 | 4.06 | 7.51 | 16.55
RNNB_varied | 1.77 | 4.36 | 8.14 | 18.01
VanillaRNN_0.05 | 1.89 | 4.99 | 8.33 | 20.67
VanillaRNN_0.1 | 2.01 | 4.52 | 18.19 | 16.95
VanillaRNN_mixed | 1.86 | 4.53 | 7.84 | 17.36
VanillaRNN_varied | 2.01 | 4.97 | 8.76 | 19.49
Table 2: CER and WER tested on 5% and 10% corruption ratios, for a model Y_X, Y is the model name and X is the corruption ratio that Y is trained on. $\psi$ (%) |  $\overline{CER}$ (%) |  $\overline{WER}$ (%)
---|---|---
10 | 10.02 | 50.94
5 | 5.03 | 29.72
Table 3: CER and WER on the test set after introducing data corruption. Model | CERR (%) | WERR (%)
---|---|---
$\psi=5\%$ | $\psi=10\%$ | $\psi=5\%$ | $\psi=10\%$
Transformer_0.05 | 75.34 | 58.58 | 81.99 | 63.91
Transformer_0.1 | 71.17 | 71.85 | 79.97 | $\boldsymbol{79.66}$
Transformer_mixed | $\boldsymbol{77.93}$ | $\boldsymbol{72.05}$ | $\boldsymbol{83.84}$ | 79.09
Transformer_varied | 75.74 | 68.46 | 81.79 | 75.75
RNNB_0.05 | 65.40 | 55.58 | 73.88 | 63.09
RNNB_0.1 | 63.02 | 59.98 | 73.75 | 69.74
RNNB_mixed | 66.79 | 59.48 | 74.73 | 67.51
RNNB_varied | 64.81 | 56.48 | 72.61 | 64.64
VanillaRNN_0.05 | 62.42 | 50.19 | 71.97 | 59.42
VanillaRNN_0.1 | 60.03 | 54.89 | 38.79 | 66.72
VanillaRNN_mixed | 63.02 | 54.79 | 73.62 | 65.92
VanillaRNN_varied | 60.03 | 50.39 | 70.52 | 61.73
Table 4: CERR and WERR tested on 5% & 10% corruption ratios, for a model Y_X,
Y is the model name and X is the corruption ratio that Y is trained on. Figure
4: Attention generated during inference across all heads of the last decoder
layer of the transformer model.

utf8  Input Result إن الذب من المعادرن النفيسة إن الذهب من المعادن النفيسة
وقال محمد أن الاستمارات بلت ثلاث مليرات دولار وقال محمد أن الاستثمارات بلغت
ثلاث مليارات دولار ذبه الرجل مسراعا إلى الابة ذهب الرجل مسرعا إلى الغابة إلى
أيييين أنت ناظؤ إلى أين أنت ناظر الأؤدن عامصتها عمان الأردن عاصمتها عمان هل
سبق ورزت الدينة هل سبق ورزت المدينة يخج من البكرات مواد مصرة وغازات يخرج من
البكرات مواد مصرية وغازات وثد كنا نرتدي اليز الرمسي وقد كلنا نرتدي الزي الرسمي
الناق الذي يحط بسحل المحط الهاي النطاق الذي يحيط بسواحل المحيط الهادي يختلف
شكل المخط البركاني باختاف الماد الي يترلاكب منها يختلف شكل المخطط البركاني
باختلاف المواد التي يتراكب منها

Table 5: Samples generated during inference from the transformer model.

Figure 4 shows the attention generated from the transformer model trained on
the mixed data set across different heads during inference, and table 5 shows
testing samples during inference.

##  5 Concluding Remarks

We presented "AraSpell," a framework for Arabic spelling correction, and
introduced different Seq2Seq models with error injection schema. Our model was
trained on more than 6.9 million sentences and it has achieved a Character
Error Rate (CER) of 1.11%, and Word Error Rate (WER) of 4.8%, which resulted
in 77.93%, and 83.84% character and word error reduction rate respectively.
Moreover, the proposed model also achieved 2.9% CER and 10.65% WER resulting
in 72.05%, and 79.09% character and word error reduction rate, respectively.
Both results achieved on a test set of 100K sentences with 5%, and 10%
error/corruption injection rate.

A main feature of the proposed approach is the proper use of error injection.
For further improvements, in future work, we recommend adding more grammatical
mistakes for error injection and training on longer sentences to capture more
context. The approach is promising and can be put in practical use, especially
if further training is applied.

## Authors’ Contributions

Development, implementation and experimental analysis: MS. Writing and
reviewing: MS and FNA. Supervision: FNA.

## Supplementary Materials

Software and data produced as part of this study are available at the
following github repository: https://github.com/msalhab96/AraSpell

## Funding and/or Conflicts of interests/Competing interests.

No funding was received for conducting this study. Moreover, the authors have
no competing interests or conflict of interest to declare that are relevant to
the content of this article.

## References

  *   * (1) Olaronke, I., Olaleke, J.: A systematic review of natural language processing in healthcare. International Journal of Information Technology and Computer Science 08, 44–50 (2015). https://doi.org/10.5815/ijitcs.2015.08.07
  * (2) Prakash, K.: Advances in natural language processing - a survey of current research trends, development tools and industry applications. International Journal of Recent Technology and Engineering (2019)
  * (3) Xing, F.Z., Cambria, E., Welsch, R.E.: Natural language based financial forecasting: a survey. Artif Intell Rev 50, 49–73. https://doi.org/10.1007/s10462-017-9588-9
  * (4) Boudad, N., Faizi, R., Rachid, O.h.t., Chiheb, R.: Sentiment analysis in arabic: A review of the literature. Ain Shams Engineering Journal 9 (2017). https://doi.org/10.1016/j.asej.2017.04.007
  * (5) Alwajeeh, A., Al-Ayyoub, M., Hmeidi, I.: On authorship authentication of arabic articles. In: 2014 5th International Conference on Information and Communication Systems (ICICS), pp. 1–6 (2014). https://doi.org/10.1109/IACS.2014.6841973
  * (6) Al-Ayyoub, M., Nuseir, A., Alsmearat, K., Jararweh, Y., Gupta, B.: Deep learning for arabic nlp: A survey. Journal of Computational Science 26, 522–531 (2018). https://doi.org/10.1016/j.jocs.2017.11.011
  * (7) Napoles, C., Cahill, A., Madnani, N.: The effect of multiple grammatical errors on processing non-native writing, pp. 1–11 (2016). https://doi.org/10.18653/v1/W16-0501
  * (8) Bassil, Y., Alwani, M.: OCR post-processing error correction algorithm using google online spelling suggestion. CoRR abs/1204.0191 (2012) 1204.0191
  * (9) Rachidi, T., Bouzoubaa, M., Elmortaji, L., Boussouab, B., Bensaid, A.: ARABIC USER SEARCH QUERY CORRECTION AND EXPANSION
  * (10) Moslem, Y., Haque, R., Way, A.: Arabisc: Context-sensitive neural spelling checker. In: NLPTEA (2020)
  * (11) Kukich, K.: Techniques for automatically correcting words in text. ACM Comput. Surv. 24, 377–439 (1992)
  * (12) Gao, M., Xu, C., Shi, P.: Hierarchical character tagger for short text spelling error correction. In: WNUT (2021)
  * (13) Wang, Y.-R., Liao, Y.-F.: Word vector/conditional random field-based chinese spelling error detection for sighan-2015 evaluation. In: SIGHAN@IJCNLP (2015)
  * (14) Zaky, D., Romadhony, A.: An lstm-based spell checker for indonesian text. In: 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pp. 1–6 (2019). https://doi.org/10.1109/ICAICTA.2019.8904218
  * (15) Kinaci, A.C.: Spelling correction using recurrent neural networks and character level n-gram. 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), 1–4 (2018)
  * (16) Nawar, M.N., Ragheb, M.M.: Fast and robust arabic error correction system. In: ANLP@EMNLP (2014)
  * (17) Shaalan, Khaled and Khaled, and Allam, and Amin, and Gomah, and Abdallah: Towards automatic spell checking for arabic. (2003)
  * (18) Alkanhal, M.I., Al-Badrashiny, M.A., Alghamdi, M.M., Al-Qabbany, A.O.: Automatic stochastic arabic spelling correction with emphasis on space insertions and deletions. IEEE Transactions on Audio, Speech, and Language Processing 20(7), 2111–2122 (2012). https://doi.org/10.1109/TASL.2012.2197612
  * (19) Hassan, Y., Aly, M., Atiya, A.: Arabic spelling correction using supervised learning. (2014). https://doi.org/10.3115/v1/W14-3615
  * (20) Attia, M., Pecina, P., Samih, Y., Shaalan, K., Van Genabith, J.: Arabic spelling error detection and correction. Nat. Lang. Eng. 22(5), 751–773 (2016)
  * (21) Shaalan, K., Samih, Y., Attia, M., Pecina, P., Genabith, J.: Arabic word generation and modelling for spell checking. (2012)
  * (22) Alkhatib, M., Monem, A.A., Shaalan, K.: Deep learning for arabic error detection and correction. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 19(5) (2020). https://doi.org/10.1145/3373266
  * (23) Abandah, G., Suyyagh, A., Khedher, M.: Correcting arabic soft spelling mistakes using bilstm-based machine learning. International Journal of Advanced Computer Science and Applications 13, 815–829 (2022). https://doi.org/10.14569/IJACSA.2022.0130594
  * (24) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. CoRR abs/1706.03762 (2017) 1706.03762
  * (25) Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. CoRR abs/2005.12872 (2020) 2005.12872
  * (26) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. CoRR abs/2010.11929 (2020) 2010.11929
  * (27) Chen, X., Wu, Y., Wang, Z., Liu, S., Li, J.: Developing real-time streaming transformer transducer for speech recognition on large-scale dataset. CoRR abs/2010.11395 (2020) 2010.11395
  * (28) Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., Pang, R.: Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv (2020). https://doi.org/10.48550/ARXIV.2005.08100. https://arxiv.org/abs/2005.08100
  * (29) Li, N., Liu, S., Liu, Y., Zhao, S., Liu, M., Zhou, M.: Neural Speech Synthesis with Transformer Network. arXiv (2018). https://doi.org/10.48550/ARXIV.1809.08895. https://arxiv.org/abs/1809.08895
  * (30) Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., Qin, T., Liu, T.-Y.: MultiSpeech: Multi-Speaker Text to Speech with Transformer. arXiv (2020). https://doi.org/10.48550/ARXIV.2006.04664. https://arxiv.org/abs/2006.04664
  * (31) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
  * (32) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. CoRR abs/1512.00567 (2015) 1512.00567