# Electronic structure and magnetic correlations in trilayer nickelate
superconductor La4Ni3O10 under pressure

I. V. Leonov  M. N. Mikheev Institute of Metal Physics, Russian Academy of
Sciences, 620108 Yekaterinburg, Russia  Institute of Physics and Technology,
Ural Federal University, 620002 Yekaterinburg, Russia

###### Abstract

It has been recently shown that under pressure trilayer Ruddlesden-Popper
nickelate La4Ni3O10 (LNO) becomes superconducting below a critical temperature
$\sim$20 K, in addition to the infinite-layer and bilayer systems. Motivated
by this observation, we explore the effects of electron correlations on its
electronic structure and magnetic properties using the advanced DFT+dynamical
mean-field theory approach. Our results for the normal-state electronic
structure and correlation effects in LNO have much in common with the
infinite-layer and bilayer nickelates, with a remarkable site- and orbital-
dependent renormalizations of the Ni $3d$ bands and notable incoherence of the
Ni $d_{3z^{2}-r^{2}}$ states, caused by correlation effects. Our analysis of
the Fermi surface and magnetic correlations suggests the emergence of
competing spin and charge stripe states, implying the importance of in-plane
spin fluctuations to explain superconductivity in this material.

The recent discovery of superconductivity in the hole-doped “infinite-layer”
nickelates $R$NiO2 ($R$ – rare earth element) has renewed great attention to
the low-valence layered nickelates Li_2019 ; Hepting_2020 ; Zeng_2020 ;
Osada_2021 . In these mixed-valence systems, Ni ions adopt a square-planar
NiO2 structure, reminiscent the electronic states of Cu2+ ions in the hole-
doped cuprate superconductors. Being isoelectronic to the parent cuprate
superconductor CaCuO2, with a critical temperature up to $T_{c}\sim 110$ K
upon hole doping Azuma_1992 , the low-energy physics of hole-doped infinite-
layer nickelates is dominated by the Ni $d_{x^{2}-y^{2}}$ electrons Lee_2004 .
However, due to strong hybridization effects it is complicated by the presence
of the rare-earth $5d$ states at the Fermi level, yielding a self-doped
electronic structure and noncupratelike multiorbital Fermi surface Lee_2004 ;
Kitatani_2020 ; Chen_2022a ; Nomura_2022 ; Botana_2022 .

Experimental and theoretical studies of the infinite-layer nickelates suggest
a remarkable orbital-dependent localization of the Ni $3d$ states, implying
the importance of strong correlations Kitatani_2020 ; Chen_2022a ; Nomura_2022
; Botana_2022 . Interestingly, superconductivity sets in a mixed-valence state
near to Ni1.2+. While recent experiments provide evidence for sizeable
antiferromagnetic correlations, no static magnetic order has been detected yet
Lin_2021 ; Zhou_2022 ; Lin_2022 ; Fowlie_2022 . Based on the resonant
diffraction, the authors propose the formation of translational symmetry
broken states (spin and charge density wave states) Rossi_2021 ; Tam_2021 ;
Krieger_2021 , implying the complex interaction between spin and charge
degrees of freedoms Si_2020 ; Werner_2020 ; Karp_2020a ; Lechermann_2020b ;
Wang_2020 ; Nomura_2020 ; Leonov_2020 ; Leonov_2021 ; Lechermann_2022 ;
Kreisel_2022 ; Pascut_2023 ; Kitatani_2023 ; Slobodchikov_2022 ; Shen_2023 ;
Chen_2022 .

In 2023, superconductivity with a high critical temperature $\sim$80 K has
been experimentally determined in the bulk bilayer perovskite nickelate
La3Ni2O7, under pressure above 15 GPa Sun_2023 ; Hou_2023 ; Zhang_2023a ;
Liu_2023 . It belongs to the Ruddlesden-Popper (RP) series Lan+1NinO3n+1 with
$n=2$. In contrast to the infinite-layer nickelates, in the bilayer system the
Ni ions have a nominal Ni2.5+ ($3d^{7.5}$) electronic configuration. Moreover,
superconductivity sets in near to a structural phase transition to the
orthorhombic phase above $\sim$15 GPa, which is characterized by the absence
of tilting of oxygen octahedra Sun_2023 ; Hou_2023 ; Zhang_2023a ; Liu_2023 .
Superconductivity is found to compete with a weakly insulating behavior at
pressures below 15 GPa and a strange (bad) metal phase above $T_{c}$. Upon
cooling below 153 K (at ambient pressure), the bilayer nickelate shows anomaly
in the transport properties associated with the formation of spin and charge
density wave states Liu_2023 . Moreover, the experiments propose the
significance of orbital-selective correlations in the bilayer system,
consistent with the recent results of the electronic structure DFT+dynamical
mean-field theory (DFT+DMFT) and GW+DMFT calculations Shilenko_2023 ;
Zhang_2023 ; Lechermann_2023 ; Christiansson_2023 ; Yang_2023 ; Shen_2023b ;
Cao_2023 ; Chen_2023 ; Wu_2023 ; Ryee_2023 .

More recently, it has been demonstrated that under pressure above $\sim$15 GPa
the trilayer RP ($n=3$) nickelate La4Ni3O10 (LNO) reveals signatures of
superconductivity with $T_{c}$ about 20-30 K Sakakibara_2023 ; Zhu_2023 ;
Zhang_2023c ; Li_2023 ; Li_2024 . It shares the same structural motif as that
in the bilayer system (and high-$T_{c}$ curates) with the trilayer corner-
sharing NiO6 octahedra blocks separated by the La-O units. In contrast to the
bilayer nickelate, it has (at least) two structurally distinct Ni sites: in
the inner (Niin) and in the outer (Niout) NiO2 layers of the trilayer blocks.
Similarly to the bilayer system, under pressure above 15 GPa the trilayer
nickelate LNO exhibits a structural phase transition to a high-symmetry
tetragonal $I4/mmm$ phase, with no NiO6 octahedra tilting.

At ambient pressure, below $\sim$140 K LNO undergoes a metal-to-metal phase
transition associated with the formation of unusual intertwined spin and
charge density wave states Zhang_2020 . The pressure-driven transition is
accompanied by the suppression of (long-range) density wave ordering, with the
emergence of superconductivity above $\sim$15 GPa Sakakibara_2023 ; Zhu_2023 ;
Zhang_2023c ; Li_2023 ; Li_2024 . Moreover, experiments suggest the appearance
of a strange metal behavior in the normal state of LNO extending up to room
temperature Zhu_2023 . This implies the crucial importance of the effects of
electronic correlations, Fermi surface nesting, and charge and spin density
wave ordering to explain the properties of LNO. It is interesting that in
contrast to cuprates in which the trilayer systems $(n=3)$ hold the highest
$T_{c}$, in the trilayer nickelate $T_{c}$ is remarkably smaller than that in
the bilayer material. Nowadays, the effects of electronic correlations on the
electronic structure, magnetism, and superconductivity in the trilayer
nickelate are still poorly understood.

We address this issue in our present paper. In our work, we study the
electronic structure and magnetic properties of the high-pressure $I4/mmm$
phase of the trilayer La4Ni3O10 using the DFT+DMFT method Georges_1996 ;
Kotliar_2006 . We explore the effects of electron correlations on the
electronic structure and magnetic properties of this system. In our
calculations, we adopt the experimental lattice parameters and atomic
positions of LNO recently determined using synchrotron x-ray diffraction at 40
GPa Li_2023 . In DFT we employ generalized gradient approximation with Perdew-
Burke-Ernzerhof exchange functional as implemented in the Quantum ESPRESSO
package Giannozzi_2017 ; DalCorso_2014 . Using DFT+DMFT we compute the normal-
state electronic structure, Fermi surface, and magnetic correlations of the
paramagnetic (PM) phase of LNO under pressure. In our calculations we employ
the fully charge self-consistent DFT+DMFT method Pourovskii_2007  implemented
with plane-wave pseudopotentials Leonov_2020b . In order to take into account
the effects of electronic correlations in the partially filled Ni $3d$ shell
and charge transfer between the Ni $3d$, O $2p$, and La $5d$ valence states we
construct a basis set of atomic-centered Wannier functions for the Ni $3d$, O
$2p$, and La $5d$ orbitals using the energy window spanned by these bands
Anisimov_2005 .

We use the continuous-time hybridization expansion quantum Monte-Carlo
algorithm (CT-QMC) to treat the many-body and strong correlations effects in
the Ni $3d$ shell Gull_2011 . The effects of electron correlations in the Ni
$3d$ orbitals are treated by using the on-site Hubbard $U=6$ eV and Hund’s
exchange $J=0.95$ eV, taken in accordance with previous studies of the
infinite-layer and perovskite nickelates. The O $2p$ and La $5d$ valence
states are uncorrelated and are treated on the DFT level within the self-
consistent DFT+DMFT approach. We note that the tetragonal structure of LNO
contains two structurally distinct Ni sites: one Niin and two equivalent Niout
ions (one formula unit in the primitive cell). We therefore use a two-
impurity-site extension of DFT+DMFT to treat the effect of correlations in the
structurally distinct Ni sites. We employ the fully localized double-counting
correction evaluated from the self-consistently determined local Ni $3d$
occupations. In order to compute the k-resolved spectra and correlated Fermi
surfaces we use Padé approximants to perform analytic continuation of the
self-energy results on the real energy axis. Using DFT+DMFT we study the
normal-state electronic structure of PM LNO at a temperature $T=290$ K (our
calculations at $T=116$ K give quantitatively similar results).

Figure 1: k-resolved spectral functions of PM LNO obtained by DFT+DMFT at
$T=290$ K. The nonmagnetic DFT (PBE) results are shown with green broken
lines.

We begin with the electronic structure calculations of the high-pressure LNO.
In Fig. 1 we display the ${\bf k}$-resolved spectral functions obtained using
DFT+DMFT (for the crystal structure parameters taken at 40 GPa). The
corresponding orbital-dependent spectral functions are shown in Fig. 2.
Overall, the ${\bf k}$-resolved spectral functions of LNO remain similar to
those obtained within the nonmagnetic DFT. Our results show a remarkable
renormalization of the Ni $3d$ quasiparticle bands and incoherence of the
spectral weight caused by correlation effects. The partially occupied Ni
$d_{x^{2}-y^{2}}$ and $d_{3z^{2}-r^{2}}$ electronic states appear near the
Fermi level and form a quasiparticle band between -2 eV and 3 eV. The Ni
$t_{2g}$ states are fully occupied and are located at about -1.2 eV below the
$E_{F}$. The occupied O $2p$ states appear at -8 to -0.6 eV below $E_{F}$, and
exhibit strong hybridization with the Ni $3d$ and empty La $5d$ orbitals. The
electronic states originating from the La $5d$ orbitals are empty and appear
above 2 eV. This differs from a self-doped electronic structure behavior of
the infinite-layer systems Lee_2004 .

Figure 2: Orbital-dependent spectral functions of PM LNO obtained by DFT+DMFT
at $T=290$ K. The partial Ni $t_{2g}$, $x^{2}-y^{2}$, and $3z^{2}-r^{2}$
orbital contributions are shown (per orbital). The partial Ni $3d$ states are
magnified by a factor eight for better readability.

We observe that the electronic states of the Ni ions in the inner and outer
NiO2 layers exhibit only minor differences. We note that the Niin $e_{g}$
orbitals and the Niout $d_{3z^{2}-r^{2}}$ orbitals show a quasiparticle peak
at the Fermi level (at $T=290$ K), while the Niout $d_{x^{2}-y^{2}}$ states
are close to form a pseudo gap at $E_{F}$, with a quasiparticle peak located
to about -0.14 eV below $E_{F}$. Moreover, in close similarity to the bilayer
system, the k-resolved spectral functions of LNO show remarkably flat (and
strongly incoherent) band dispersions originating from the Ni
$d_{3z^{2}-r^{2}}$ orbitals located near the Fermi level at the Brillouin zone
(BZ) M-point (note that in our previous work Ref. Shilenko_2023  in Fig. 1 the
M and X points were mixed up by mistake). In addition, the Ni
$d_{x^{2}-y^{2}}$ orbitals form a nearly flat structure at about -0.14 eV
below $E_{F}$ at the X point on the $\Gamma$-X branch, suggestive of a van
Hove anomaly in the electronic spectra. Moreover, the same behavior is seen in
the electronic structure of the infinite-layer and bilayer nickelates, while
electronic correlations shift this anomaly closer to the $E_{F}$ (due to
bandwidth renormalizations). In overall, the calculated electronic structure
of LNO is very similar to that of the high-pressure bilayer nickelate, with
two extra Ni $e_{g}$ quasiparticle bands to appear near the Fermi level
Puggioni_2018 ; Li_2017 ; Sakakibara_2023 . In particular, we observe a weakly
dispersive band of the Ni $d_{3z^{2}-r^{2}}$ character to appear on the
$\Gamma$-Z branch below the Fermi level, which crosses the $E_{F}$ near the
$\Gamma$-point.

Similarly to that in the bilayer nickelate, the Wannier Ni $d_{x^{2}-y^{2}}$
and $d_{3z^{2}-r^{2}}$ orbital occupations in LNO are close to half-filling,
of $\sim$0.53 and 0.56 (per spin-orbit) for the Niin and 0.55 and 0.57 for the
Niout sites, respectively. That is, orbital polarization between the Ni
$d_{x^{2}-y^{2}}$ and $d_{3z^{2}-r^{2}}$ orbitals is rather weak and bandwidth
effects dominate over potential Jahn-Teller orbital splitting (for Ni3+ ions).
It is interesting to note that in the optimally doped $R$NiO2 the Ni
$d_{3z^{2}-r^{2}}$ orbital occupancy is considerably larger, $\sim$0.85
Leonov_2020 ; Leonov_2021 . In LNO the total Wannier Ni $3d$ occupations of
the Niin and Niout ions are nearly same, about 8.14 and 8.19, respectively.
This value sufficiently differs from the formal oxidation state Ni2.67+
($3d^{7.33}$), implying the importance of negative charge transfer effects
Zaanen_1985  (in contrast to the Mott-Hubbard regime of the infinite-layer
systems Lee_2004 ; Kitatani_2020 ; Chen_2022a ; Nomura_2022 ; Botana_2022 ).
Our results for the instantaneous local magnetic moments
$\sqrt{\langle\hat{m}^{2}_{z}\rangle}$ are 1.25 and 1.29 $\mu_{\mathrm{B}}$,
respectively. The fluctuating magnetic moments evaluated as
$M_{\mathrm{loc}}\equiv[k_{\mathrm{B}}T\int\chi(\tau)d\tau]^{1/2}$ (where
$\chi(\tau)\equiv\langle\hat{m}_{z}(\tau)\hat{m}_{z}(0)\rangle$ is the local
spin correlation function) are sufficiently smaller, about 0.36 and 0.43
$\mu_{\mathrm{B}}$. Moreover, in accord with a mixed-valence Ni ion
configuration our analysis of the weights of different atomic configurations
of the Ni $3d$ electrons (being fluctuating between various atomic
configurations within DMFT) gives 0.14, 0.54, and 0.29 for the $d^{7}$,
$d^{8}$, and $d^{9}$ configurations, respectively, for the Niin. For the Niout
ions these are 0.11, 0.54, and 0.31, respectively. Our analysis for the spin-
state configurations suggests strong interplay of the Ni $S=0$,
$S=\frac{1}{2}$, and $S=1$ states in the electronic structure of LNO. Thus,
the corresponding spin-state weights are 0.29, 0.41, and 0.28 for the Ni ions,
respectively, in close similarity to that in the bilayer system.

Figure 3:  Orbital-dependent the Ni $3d$ self-energies
$\mathrm{Im}[\Sigma(i\omega)]$ on the Matsubara axis and those analytically
continued on the real energy axis $\mathrm{Im}[\Sigma(\omega)]$ for the
crystallographically distinct Niin (top) and Niout (bottom) sites.

Next, we discuss the orbital-selective behavior and quasiparticle mass
renormalizations of the Ni $3d$ bands in LNO under pressure. We observe that
the electronic states of LNO obtained within DFT+DMFT show a remarkable
renormalization of the quasiparticle Ni $3d$ bands and orbital-selective
incoherence of the spectral weight (bad metal behavior). In particular, our
analysis of the Ni $3d$ self-energies on the Matsubara contour suggests a
typical Fermi liquid-like behavior with a substantial orbital-dependent
quasiparticle damping of the Ni $3d$ states (see Fig. 3). In fact, for the
Niin ions $\mathrm{Im}[\Sigma(i\omega_{n})]\sim 0.09$ and 0.11 eV for the Ni
$d_{x^{2}-y^{2}}$ and $d_{3z^{2}-r^{2}}$ quasiparticle states at the first
Matsubara frequency, at $T=290$K. The fully occupied Ni $t_{2g}$ states are
sufficiently coherent, with $\mathrm{Im}[\Sigma(i\omega_{n})]$ below 0.02 eV
at the first Matsubara frequency. At the same time, for the Niout ions the
quasiparticle damping is noticeably higher, about 0.12 and 0.15 eV for the
$d_{x^{2}-y^{2}}$ and $d_{3z^{2}-r^{2}}$ orbitals, respectively. Using Padé
extrapolation of the self-energy $\Sigma(i\omega)$ to $i\omega\rightarrow 0$
we obtain 0.03 and 0.04 eV for the Niin, and $\sim$0.05 for the Niout
$x^{2}-y^{2}$ and $3z^{2}-r^{2}$ states at the Fermi energy, respectively.

In addition, we evaluate the orbitally-resolved quasiparticle mass enhancement
factors as $m^{*}/m=[1-\partial\mathrm{Im}[\Sigma(i\omega)]/\partial
i\omega]|_{i\omega\rightarrow 0}$. Our DFT+DMFT calculations reveal
significant correlation effects within the Ni $3d$ orbitals that suggests
site- and orbital-dependent localization of the Ni $3d$ electrons. Our
analysis of the quasiparticle mass enhancement for the Niin ions gives nearly
orbital independent renormalisation factors $m^{*}/m\sim 1.9$ for both the Ni
$d_{x^{2}-y^{2}}$ and $d_{3z^{2}-r^{2}}$ orbitals. In contrast to this, for
the Ni ions in the NiO2 outer layer we obtain sufficient orbital
differentiation, $\sim$2.1 and 2.5 for the $d_{x^{2}-y^{2}}$ and
$d_{3z^{2}-r^{2}}$ orbitals, respectively. That is, the $d_{3z^{2}-r^{2}}$
states of the Niout ions are seen to be more correlated and incoherent-like
than the in-plane $d_{x^{2}-y^{2}}$ orbitals. The effective mass enhancements
of the Ni $t_{2g}$ states is sufficiently weaker, of $\sim$1.2 (for both Ni
ions). This behavior implies the importance of site- and orbital-dependent
localization of the Ni $3d$ electrons. In overall, the strength of correlation
effects is seen to be sufficiently weaker than that in the bilayer system (in
the bilayer system $m^{*}/m\sim 2.3$ and 3 for the Ni $d_{x^{2}-y^{2}}$ and
$d_{3z^{2}-r^{2}}$ orbitals, respectively, as obtained by DFT+DMFT with the
same Hubbard $U$ and Hund’s exchange $J$). It also differs qualitatively from
the results for $R$NiO2 which show strong orbital differentiation with
$m^{*}/m\sim 3$ and 1.3 for the Ni $d_{x^{2}-y^{2}}$ and $d_{3z^{2}-r^{2}}$
orbitals, respectively Leonov_2020 ; Leonov_2021 .

Overall, our estimates for the quasiparticle mass enhancement are in good
agreement with experimental estimates ($\sim$2-2.6) Li_2017 . Our findings are
also inline with a more narrow DFT bandwidth of the Ni $d_{3z^{2}-r^{2}}$
orbitals, by $\sim$17% less than that for the $d_{x^{2}-y^{2}}$ states, for
the Niout ions. We also check how the correlation effects depend on the choice
of the Hubbard $U$ and Hund’s exchange $J$. In fact, it was found that the
effective mass renormalization of the Ni $3d$ states depends sensitively upon
variation of the $U$ and $J$ values. Thus, for $U=4$ eV and $J=0.45$ eV we
obtain $m^{*}/m\sim 1.4$ and 1.5 for the Ni $x^{2}-y^{2}$ and $3z^{2}-r^{2}$
orbitals for the Niout ions, respectively. For $U=6$ eV and $J=0.45$ eV we
obtain $\sim$1.6 and 1.8, for $U=4$ eV and $J=0.95$ eV $\sim$1.6 and 1.9, and
for $U=8$ eV and $J=0.95$ eV $\sim$2.7 and 3.1, respectively. That is, our
estimates reveal a strong sensitivity of $m^{*}/m$ upon the Hund’s exchange
$J$, while dependence on the $U$ value is much weaker. This behavior is
consistent with that obtained for the bilayer system. This implies the
importance of multi-orbital Hund’s metal physics to explain the electronic
propertied of LNO.

Our results agree well with our analysis of the orbital-dependent local spin
susceptibility $\chi(\tau)$, evaluated within DFT+DMFT (see Supplemental
Materials suppl). Our DFT+DMFT calculations suggest the proximity of the Ni
$e_{g}$ states to orbital-selective localization. Moreover, the Ni $3d$
electron localization has a pronounced site selectivity. In fact, the Niout
$e_{g}$ states show more localized behavior than that for the Niin ions. In
particular, for the Niout ions the $d_{3z^{2}-r^{2}}$ orbitals show a slow
decaying behavior of $\chi(\tau)$ to 0.03 $\mu_{\mathrm{B}}^{2}$ at
$\tau=\beta/2$, while for the $d_{x^{2}-y^{2}}$ states it is $\sim$0.02
$\mu_{\mathrm{B}}^{2}$. For the Niin ions this orbital selective behavior is
less pronounced, $\chi(\tau)$ of about 0.01 $\mu_{\mathrm{B}}^{2}$ at
$\tau=\beta/2$ for both the $d_{3z^{2}-r^{2}}$ and $d_{x^{2}-y^{2}}$ orbitals.
Our results therefore suggest that magnetic correlations in LNO are at the
verge of a site- and orbital-dependent formation of local magnetic moments,
suggestive of Hund’s metal behavior Georges_2013 .

Figure 4:  Our results for the correlated Fermi surfaces for different $k_{z}$
calculated by DFT+DMFT at $T=290$ K.

Next, we compute the correlated Fermi surface (FS) using DFT+DMFT at $T=290$K.
To this end, we evaluate the ${\bf k}$-dependent spectral weight $A({\bf
k},\omega)=-1/\pi~{}\mathrm{Im}[\omega+\mu-H_{\bf k}-\Sigma(\omega)]^{-1}$ at
$\omega=0$. Here, $H_{\bf k}$ is the self-consistent Kohn-Sham Hamiltonian in
the Wannier basis set, $\Sigma(\omega)$ is the self-energy analytically
continued on the real energy axis. In Fig. 4 we show our results for the in-
plane quasiparticle FSs for different $k_{z}$. In agreement with previous
ARPES measurements, we obtain a large hole FS that closely resembles the FS of
optimally hole-doped cuprates, including its $x^{2}-y^{2}$ orbital character,
hole occupation level, and strength of correlation effects Li_2017 . We note
that correlation effects mainly result in orbital-dependent effective mass
renormalizations and incoherence of the spectral weight, while the FSs
topology is notably similar to that obtained within DFT.

Our results for the FSs bear a striking resemblance to those of the bilayer
nickelate (see, e.g., Refs. Shilenko_2023 ; Lechermann_2023 ). We observe two
electron FS pockets centered at the BZ $\Gamma$ (Z) point and three (two of
which are nearly degenerate) hole pockets at the M point. In overall, the
quasiparticle FSs are two-dimensional, except for the internal FS sheet
predominantly of the Niout $d_{3z^{2}-r^{2}}$ character centred at the
$\Gamma$ point, along the $k_{z}$ axis. The latter shows a remarkable
dependence on $k_{z}$, forming a hourglass-like FS. It is associated with a
weakly dispersive Niout $d_{3z^{2}-r^{2}}$ band on the $\Gamma$-Z branch,
crossing the $E_{F}$ near the $\Gamma$ point (see Fig. 1).

We note that the FSs exhibit strong orbital-dependent incoherence caused by
correlation effects. It is particularly notable for the hole pocket near the M
point originating from the shallow flat band of the Ni $d_{3z^{2}-r^{2}}$
character located near $E_{F}$. This FS pocket shows a remarkable $k_{z}$
dispersion (a large change of the size with $z$). We also notice a variation
of the shape of the large electron FS pocket of the Ni $d_{x^{2}-y^{2}}$
orbital character near the $\Gamma$ (Z) point, from a circle-like at $k_{z}=0$
to a square-like at $k_{z}=1/2$ ($\pi/z$), with no change of its size. In
comparison to the bilayer system, our results indicate a notably reduced two-
dimensionality of the electronic structure.

Moreover, our analysis of the FS suggests its high sensitivity towards hole or
electron doping, resulting in a reconstruction of the FS topology – a Lifshitz
transition. Thus, hole doping tends to suppress the internal FS sheet centred
at the BZ $\Gamma$ (Z) point. In contrast, by doping with electrons one would
diminish the strongly incoherent hole FS sheet near the M point. It would
affects both FS dimensionality and FS nesting. All this suggests high
sensitivity of the properties of LNO to stoichiometry and doping.

Figure 5:  Orbitally resolved static spin susceptibility $\chi({\bf q})$ of
LNO obtained by DFT+DMFT at $T=290$ K.

Obviously, the correlated Fermi surfaces of the high-pressure LNO exhibit
multiple in-plane nesting effects (see Fig. 4), implying the presence of
competing ordering effects in LNO. Therefore, we compute the momentum-
dependent static magnetic susceptibility $\chi({\bf q})$ in the particle-hole
bubble approximation within DFT+DMFT as $\chi({\bf
q})=-k_{B}T\mathrm{Tr}\Sigma_{{\bf k},i\omega_{n}}G_{\bf
k}(i\omega_{n})G_{{\bf k}+{\bf q}}(i\omega_{n})$. $G_{\bf k}(i\omega_{n})$ is
the local _interacting_ Green’s function for the Ni $3d$ states evaluated on
the Matsubara contour $i\omega_{n}$. In Fig. 5 we display our results for the
site- and orbital-dependent contributions to $\chi({\bf q})$. $\chi({\bf q})$
exhibits complex structure with multiple well-defined maxima, suggesting a
complex interplay between spin and charge density wave states in LNO. Our
results for the Niout ions exhibit three well-defined maxima at an
incommensurate wave vector $(0.3~{}0~{}0)$ on the $\Gamma$-X,
$(0.5~{}0.31~{}0)$ on the X-M, and $(0.31~{}0.31~{}0)$ on the $\Gamma$-M
branches of the BZ. The most pronounced instability is associated with that on
the $\Gamma$-M brach, competing with two minor instabilities at
$(0.5~{}0.31~{}0)$ and $(0.3~{}0~{}0)$. Most importantly, this result agrees
qualitatively well with that obtained for the bilayer system – in La3Ni2O7
$\chi({\bf q})$ also shows similar structure. This is due to the fact that the
FS topology (and hence nesting effects) in LNO in overall resembles that in
the bilayer nickelate. All this supports the picture of multiple (or
intertwined) spin and change density wave instability driven by the FS in
these nickelates. The later can in principle compete with the super-exchange
interactions, giving the complex interplay between electron correlations and
spin/charge stripe states.

While the electronic states of the Ni sites in the inner and outer NiO2 layers
show minor quantitative differences, $\chi({\bf q})$ reveals sufficiently
distinct behaviour for the Niin ions with different magnetic correlations. For
the Niin ions $\chi({\bf q})$ shows a rather complex behavior with multiple
structures, distinct from those in the double layer system. For Niin ions the
most pronounced instabilities is associated with a vectore close to that for
Niout, at $(0.29~{}0.29~{}0)$. Particularly interesting is the importance of a
ferromagnetic instability at the $\Gamma$ point.

Our results suggest the complex interplay of spin and charge states in the
high pressure phase of LNO, which can be important for understanding of
superconductivity in low-valence nickelates. We note that at ambient pressure
LNO exhibits the formation of unusual spin and charge density wave states
below a metal-to-metal phase transition. Moreover, the infinite-layer systems
Rossi_2021 ; Tam_2021 ; Krieger_2021 , hole-doped mixed-valence nickelates
(La,Sr)2NiO4 (with Sr $x=1/3$, Ni2.33+ ions) and square-planar La4Ni3O8 with
Ni1.33+ and La3Ni2O6 with Ni1.5+ ions all reveal similar behavior Lee_1997 ;
Bernal_2019 ; Zhang_2019 ; Zhang_2020 ; Lee_1997 . In hole-doped $R$NiO2
DFT+DMFT calculations suggest the possible formation of charge- (bond-)
disproportionated striped states Slobodchikov_2022 ; Shen_2023 ; Chen_2022 .
All this suggests the importance of density wave states (and most importantly
their fluctuations) to explain the electronic properties of LNO Tranquada_1995
; Keimer_2015  . Superconductivity observed in LNO can be associated with the
suppression of charge and spin density wave ordering under pressure, which
leads to a sharp increase of spin fluctuations. This implies the importance of
in-plane spin fluctuations to explain superconductivity in LNO under pressure.

In conclusion, using the DFT+DMFT approach we studied the normal-state
electronic properties of the trilayer nickelate superconductor. We note that
the electronic structure and correlation effects in the trilayer system share
the common features of the infinite-layer and bilayer nickelates. In agreement
with experimental estimates, our results show a remarkable renormalization of
the Ni $3d$ bands, implying the site- and orbital-selective localization of
the Ni $3d$ electrons. Our results suggest the emergence of competing stripe
states on a microscopic level, which affect the electronic structure and
superconductivity of this material. We propose the presence of strong in-plane
spin fluctuations in the trilayer system.

###### Acknowledgements.

We acknowledge support by the state assignment of Minobrnauki of Russia (theme
“Electron” No. AAAA-A18-118020190098-5).

## References

  * (1) D. Li, K. Lee, B. Y. Wang, M. Osada, S. Crossley, H. R. Lee, Y. Cui, Y. Hikita, and H. Y. Hwang, Nature 572, 624 (2019).
  * (2) M. Hepting, D. Li, C. Jia, H. Lu, E. Paris _et al._ , Nat. Mater. 19, 381 (2020).
  * (3) S. Zeng, C. S. Tang, X. Yin, C. Li, M. Li _et al._ , Phys. Rev. Lett. 125, 147003 (2020).
  * (4) M. Osada, B. Y. Wang, B. H. Goodge, S. P. Harvey, K. Lee, D. Li, L. F. Kourkoutis, and H. Y. Hwang, Adv. Mater. 33, 2104083 (2021).
  * (5) M. Azuma, Z. Hiroi, M. Takano, Y. Bando, and Y. Takeda, Nature (London) 356, 775 (1992).
  * (6) K.-W. Lee and W. E. Pickett, Phys. Rev. B 70, 165109 (2004).
  * (7) M. Kitatani, L. Si, O. Janson, R. Arita, Z. Zhong, and K. Held, npj Quantum Mater. 5, 59 (2020).
  * (8) A. S. Botana, K.-W. Lee, M. R. Norman, V. Pardo, and W. E. Pickett, Front. Phys. 9, 813532 (2022).
  * (9) H. Chen, A. Hampel, J. Karp, F. Lechermann, and A. J. Millis, Front. Phys. 10, 835942 (2022).
  * (10) Y. Nomura and R. Arita, Rep. Prog. Phys. 85, 052501 (2022).
  * (11) J. Q. Lin, P. V. Arribi, G. Fabbris, A. S. Botana, D. Meyers _et al._ , Phys. Rev. Let. 126, 087001 (2021).
  * (12) X. Zhou, X. Zhang, J. Yi, P. Qin, Z. Feng _et al._ , Adv. Mater. 34, 2106117 (2022).
  * (13) H. Lin, D. J. Gawryluk, Y. M. Klein, S. Huangfu, E. Pomjakushina, F. von Rohr, and A. Schilling, New J. Phys. 24, 013022 (2022).
  * (14) J. Fowlie, M. Hadjimichael, M. M. Martins, D. Li, M. Osada _et al._ , Nat. Phys. 18, 1043 (2022).
  * (15) M. Rossi, M. Osada, J. Choi, S. Agrestini, D. Jost _et al._ , Nat. Phys. 18, 869 (2022).
  * (16) C. C. Tam, J. Choi, X. Ding, S. Agrestini, A. Nag, B. Huang, H. Luo, M. García-Fernández, L. Qiao, K.-J. Zhou, Nat. Mater. 21, 1116 (2022).
  * (17) G. Krieger, L. Martinelli, S. Zeng, L. E. Chow, K. Kummer _et al._ , Phys. Rev. Lett. 129, 027002 (2022).
  * (18) L. Si, W. Xiao, J. Kaufmann, J. M. Tomczak, Y. Lu, Z. Zhong, and K. Held, Phys. Rev. Lett. 124, 166402 (2020).
  * (19) P. Werner and S. Hoshino, Phys. Rev. B 101, 041104(R) (2020).
  * (20) J. Karp, A. S. Botana, M. R. Norman, H. Park, M. Zingl, and A. Millis, Phys. Rev. X 10, 021061 (2020).
  * (21) F. Lechermann, Phys. Rev. B 101, 081110(R) (2020).
  * (22) Y. Wang, C.-J. Kang, H. Miao, and G. Kotliar, Phys. Rev. B 102, 161118(R) (2020).
  * (23) Y. Nomura, T. Nomoto, M. Hirayama, and R. Arita, Phys. Rev. Research 2, 043144 (2020).
  * (24) I. Leonov, S. L. Skornyakov, and S. Y. Savrasov, Phys. Rev. B 101, 241108(R) (2020).
  * (25) I. Leonov, J. Alloys Compd. 883, 160888 (2021).
  * (26) F. Lechermann, Phys. Rev. B 105, 155109 (2022).
  * (27) A. Kreisel, B. M. Andersen, A. T. Rømer, I. M. Eremin, and F. Lechermann, Phys. Rev. Lett. 129, 077002 (2022).
  * (28) G. L. Pascut, L. Cosovanu, K. Haule, and K. F. Quader, Commun. Phys. 6, 45 (2023).
  * (29) M. Kitatani, L. Si, P. Worm, J. M. Tomczak, R. Arita, and K. Held, Phys. Rev. Lett. 130, 166002 (2023).
  * (30) K. G. Slobodchikov and I. V. Leonov, Phys. Rev. B 106, 165110 (2022).
  * (31) Y. Shen, M. Qin, and G.-M. Zhang, Phys. Rev. B 107, 165103 (2023).
  * (32) H. Chen, Y.-F. Yang, G.-M. Zhang, Nat. Commun. 14, 5477 (2023).
  * (33) H. Sun, M. Huo, X. Hu, J. Li, Z. Liu _et al._ , Nature 621, 493 (2023).
  * (34) J. Hou, P. T. Yang, Z. Y. Liu, J. Y. Li, P. F. Shan _et al._ , Chin. Phys. Lett. 40, 117302, (2023).
  * (35) Y. Zhang, D. Su, Y. Huang, H. Sun, M. Huo _et al._ , arXiv:2307.14819.
  * (36) Z. Liu, H. Sun, M. Huo, X. Ma, Y. Ji, E. Yi, L. Li, H. Liu, J. Yu, Z. Zhang _et al._ , Sci. China: Phys. Mech. Astron. 66, 217411 (2023).
  * (37) D. A. Shilenko and I. V. Leonov, Phys. Rev. B 108, 125105 (2023).
  * (38) F. Lechermann, J. Gondolf, S. Bötzel, and I. M. Eremin, Phys. Rev. B 108, L201121 (2023).
  * (39) V. Christiansson, F. Petocchi, and P. Werner, Phys. Rev. Lett 131, 206501 (2023).
  * (40) Q.-G. Yang, D. Wang, Q.-H. Wang, Phys. Rev. B 108, L140505 (2023).
  * (41) Y. Zhang, L.-F. Lin, A. Moreo, and E. Dagotto, Phys. Rev. B 108, L180510 (2023).
  * (42) Y. Shen, M. Qin, and G.-M. Zhang, Chin. Phys. Lett. 40, 127401 (2023).
  * (43) Y. Cao and Y.-F. Yang, arXiv:2307.06806
  * (44) X. Chen, P. Jiang, J. Li, Z. Zhong, and Y. Lu, arXiv:2307.07154
  * (45) W. Wú, Z. Luo, D.-X. Yao, and M. Wang, arXiv:2307.05662
  * (46) S. Ryee, N. Witt, and T. O. Wehling, arXiv:2310.17465.
  * (47) H. Sakakibara, M. Ochi, H. Nagata, Y. Ueki, H. Sakurai _et al._ , arXiv:2309.09462.
  * (48) Y. Zhu, E. Zhang, B. Pan, X. Chen, D. Peng _et al._ , arXiv:2311.07353.
  * (49) M. Zhang, C. Pei, X. Du, Y. Cao, Q. Wang _et al._ , arXiv:2311.07423.
  * (50) J. Li, C. Chen, C. Huang, Y. Han, M. Huo _et al._ , arXiv:2311.16763.
  * (51) Q. Li, Y.-J. Zhang, Z.-N. Xiang, Y. Zhang, X. Zhu, H.-H. Wen, Chinese Phys. Lett. 41, 017401 (2024).
  * (52) J. Zhang, D. Phelan, A. S. Botana, Y.-S. Chen, H. Zheng _et al._ , Nat. Commun. 11, 6003 (2020).
  * (53) A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Rev. Mod. Phys. 68, 13 (1996).
  * (54) G. Kotliar, S. Y. Savrasov, K. Haule, V. S. Oudovenko, O. Parcollet, and C. A. Marianetti, Rev. Mod. Phys. 78, 865 (2006).
  * (55) P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. B. Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni _et al._ , J. Phys.: Condens. Matter 29, 465901 (2017).
  * (56) A. Dal Corso, Comput. Mater. Sci. 95, 337 (2014).
  * (57) L. V. Pourovskii, B. Amadon, S. Biermann, and A. Georges, Phys. Rev. B 76, 235101 (2007).
  * (58) I. Leonov, A. O. Shorikov, V. I. Anisimov, and I. A. Abrikosov, Phys. Rev. B 101, 245144 (2020).
  * (59) V. I. Anisimov, D. E. Kondakov, A. V. Kozhevnikov, I. A. Nekrasov, Z. V. Pchelkina _et al._ , Phys. Rev. B 71, 125119 (2005).
  * (60) E. Gull, A. J. Millis, A. I. Lichtenstein, A. N. Rubtsov, M. Troyer, and P. Werner, Rev. Mod. Phys. 83, 349 (2011).
  * (61) D. Puggioni and J. M. Rondinelli, Phys. Rev. B 97, 115116 (2018).
  * (62) H. Li, X. Zhou, T. Nummy, J. Zhang, V. Pardo, W. E. Pickett, J. F. Mitchell, and D. S. Dessau, Nat. Commun. 8, 704 (2017).
  * (63) J. Zaanen, G. A. Sawatzky, and J. W. Allen, Phys. Rev. Lett. 55, 418 (1985).
  * (64) See Supplemental Material at http://link.aps.org/supplemental/ for additional figures with our DFT+DMFT results for LNO.
  * (65) A. Georges, L. de’Medici, and J. Mravlje, Annu. Rev. Condens. Matter Phys. 4, 137 (2013).
  * (66) S.-H. Lee and S-W. Cheong, Phys. Rev. Lett. 79, 2514 (1997).
  * (67) O. O. Bernal, D. E. MacLaughlin, G. D. Morris, P.-C. Ho, L. Shu, C. Tan, J. Zhang, Z. Ding, K. Huang, and V. V. Poltavets, La4Ni3O8, Phys. Rev. B 100, 125142 (2019).
  * (68) J. Zhang, D. M. Pajerowski, A. S. Botana, H. Zheng, L. Harriger _et al._ , Phys. Rev. Lett. 122, 247201 (2019).
  * (69) J. M. Tranquada, B. J. Sternlieb, J. D. Axe, Y. Nakamura, and S. Uchida, Nature 375, 561 (1995).
  * (70) B. Keimer, S. A. Kivelson, M. R. Norman, S. Uchida, and J. Zaanen, Nature 518, 179 (2015).


# MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics
Transcription

###### Abstract

This paper makes several contributions to automatic lyrics transcription (ALT)
research. Our main contribution is a novel variant of the Multistreaming Time-
Delay Neural Network (MTDNN) architecture, called MSTRE-Net, which processes
the temporal information using multiple streams in parallel with varying
resolutions keeping the network more compact, and thus with a faster inference
and an improved recognition rate than having identical TDNN streams. In
addition, two novel preprocessing steps prior to training the acoustic model
are proposed. First, we suggest using recordings from both monophonic and
polyphonic domains during training the acoustic model. Second, we tag
monophonic and polyphonic recordings with distinct labels for discriminating
non-vocal silence and music instances during alignment. Moreover, we present a
new test set with a considerably larger size and a higher musical variability
compared to the existing datasets used in ALT literature, while maintaining
the gender balance of the singers. Our best performing model sets the state-
of-the-art in lyrics transcription by a large margin. For reproducibility, we
publicly share the identifiers to retrieve the data used in this paper.

##  1 Introduction

Empirical studies show that it is a challenging task even for human listeners
to recognize sung words, and this is more challenging than speech, due to a
number of performance, environment and listener related factors[1]. Thus the
automatic retrieval of sung words through machine listening, i.e. automatic
lyrics transcription (ALT), can potentially be impactful in easing some of the
time consuming processes involved in composing music, audio/video/music score
captioning and editing, lyrics alignment, music catalogue creation, etc.
Despite its potential, the current state of lyrics transcription is far from
being sufficiently robust to be leveraged in such applications.

With recent advances in automatic speech recognition (ASR) research and its
successful adaptation to singing data, considerable improvements have been
reported in ALT research [2, 3, 4]. In addition to this, newly released
datasets have accelerated the development of the research field[5, 6]. Through
these improvements, the prospect of applying ALT in the music industry has
become more realistic, assuming that progress continues. Although promising
results have been obtained for a cappella recordings[2, 4, 7], recognition
rates drop considerably in the presence of instrumental accompaniment[8, 9].

From the perspective of ASR, music accompaniment can be regarded as noise
since non-vocal music signals generally include minimal or no information
relevant to lyrics transcription, while their presence in the spectral domain
increases the confusions during prediction. For building more robust acoustic
models against noisy environments, the multistream approach in ASR was
introduced [10], inspired by how the acoustic signals are split into multiple
frequency bands and processed in parallel in the human auditory system[11].
While previous research suggested using multiresolution feature processing[12,
13] or reconstruction of a multi-band latent representation through
autoencoders[14] to achieve multistreaming ASR, the neural network
architecture recently introduced in [15], Multistreaming Time-Delay Neural
Network (MTDNN), proposes a simplified solution which is utilized in producing
the state-of-the-art for hybrid / Deep Neural Network - Hidden Markov Model
(DNN-HMM) based ASR[16, 17]. In this work, we propose a compact variant of
MTDNN, referred to as MSTRE-Net, where streams are diversified by having
different numbers of layers with the goal of reducing the number of trainable
parameters (i.e. model complexity), and thus the inference times and improving
the word recognition rates.

Additionally, we propose a number of other novel contributions for improving
lyrics transcription performance. We suggest combined training of the acoustic
model on both monophonic (e.g. DAMP-Sing! 300x30x2 [6]) and polyphonic (e.g.
DALI [5]) recordings, which is shown to improve performance for both cases.
Furthermore, we propose tagging monophonic and polyphonic utterances with
separate _music_ and _silence_ tokens explicitly. Our goal for this is to
generate alignments that are more robust against disruptions in the decoding
path, potentially caused by the musical accompaniment during the non-vocal
frames.

One major challenge in ALT research has been publishing reproducible results,
due to the lack of publicly available evaluation data[18]. Dabike and Barker
[2] shared manually verified annotations for a subset of DAMP which have been
utilized for evaluation in a cappella singing[4, 7]. The Jamendo (lyrics)
dataset [19] consists of 20 contemporary polyphonic music recordings released
under an open source license. Moreover, despite their limited nature in terms
of size and musical variability, Hansen[20] and Mauch[21] have been among the
two most commonly used evaluation sets for ALT. In addition to these, we
present a new test set with 240 polyphonic recordings having a larger span of
release dates and better singer gender balance in order to establish a more
comprehensive lyrics transcription evaluation.

The rest of the paper is structured as follows: we begin with a summary of
essential concepts in the state-of-the-art approach for hybrid-ASR. The
following section explains how the proposed MTDNN architecture is constructed.
Next, we give details of the data used in experiments, and introduce a new
evaluation set. Finally, we describe the experimental setup and present
results verifying our design choices through ablative tests.

##  2 Background

ALT can be considered as analogous to Large Vocabulary Continuous Speech
Recognition (LVCSR) for the singing voice. Similarly, the goal for ALT is
predicting the most likely word sequence, $\mathbf{w}$, given a stream of
acoustic observations, $\mathbf{O}$, which can be expressed in mathematical
terms as follows:

$\displaystyle\widehat{\mathbf{w}}=$
$\displaystyle=\operatorname*{argmax}_{\mathbf{w}}P(\mathbf{w}|\mathbf{O})$
$\displaystyle=\operatorname*{argmax}_{\mathbf{w}}P(\mathbf{w})p(\mathbf{O}|\mathbf{w})$
$\displaystyle=\operatorname*{argmax}_{\mathbf{w}}P(\mathbf{w})\sum_{\mathbf{Q}}p(\mathbf{O}|\mathbf{Q})P(\mathbf{Q}|\mathbf{w}),$
(1)

where elements of $\mathbf{Q}$ represent the phoneme classes111A phoneme is
the basic sonic unit of speech. In linguistics, words are considered to be
composed of sequences of phonemes.. In Equation 2, $p(\mathbf{O}|\mathbf{Q})$
is obtained via the acoustic model. Phonemes are converted to word labels
using a lexicon which defines a mapping between words and their phonemic
representations. The raw word posteriors are then smoothed with the language
model, $P(\mathbf{w})$ for obtaining grammatically more plausible output
transcriptions, $\widehat{\mathbf{w}}$.

According to the probabilistic approach of ASR, phonemes are represented with
HMMs where a transition between connected phone states occurs at every time
step[22]. In our system, we employ the Kaldi toolkit [23], an open-source ASR
framework that represents HMM states using Weighted Finite State Transducers
(WFST)[24]. In operation, a WFST graph is generated through composing
posteriors retrieved from the acoustic, language and pronunciation models. The
resulting directed paths of states form a lattice, a weighted acyclic
graphical structure which can represent multiple output hypotheses.

Figure 1: DNN-HMM based ASR at operation

###  2.1 Lattice-free Maximum Mutual Information

Most recent ALT systems utilize the state-of-the-art hybrid DNN-HMM framework
where the neural networks are trained in a sequence discriminative
fashion[25]. More specifically, the best performing lyrics transcribers to
date [2, 3, 4, 8, 7] use Lattice-free Maximum Mutual Information training (LF-
MMI)[26], where network parameters are tuned w.r.t. the MMI objective:

$\mathcal{F}_{\textit{MMI}}=\sum_{u}\log\frac{p(\mathbf{O}_{u}|Q_{u})^{\mathcal{K}}P(W_{u})}{\sum_{W}p(\mathbf{O}_{u}|Q)^{\mathcal{K}}P(W)}$
(2)

where $p(\mathbf{O}_{u}|Q_{u})$ is the probability of observing an acoustic
instance $O$ in the utterance $u$, in Markovian phone state $Q_{u}$, and the
$P(W)$’s are the word sequence probabilities[27]. Optimization w.r.t. MMI aims
at maximizing the shared information between the reference and target
sequences. More explicitly, the terms in the numerator are calculated per
utterance, whereas the denominator is computed over the entire training set.
Hence, the network parameters are updated to maximize the probabilities in the
numerator and minimize the denominator. In other words, the goal of MMI
training is to discriminate a certain acoustic observation with its given
utterance.

##  3 Multistreaming Time-Delay Neural Networks

(a) Single-stream TDNN ($\tau$=3)

(b) MTDNN with identical streams ($\tau$={3,6,9})

(c) MTDNN with distinct streams ($\tau$={3,6,9})

Figure 2: Different variants of TDNN architectures. From left-to-right, the
orange, yellow and cyan blocks represent the front-end 2-D CNN, TDNN streams
and final FC layers preceded by the purple softmax layer

The main body of MTDNN architectures consists of multiple streams of TDNN
layers trained in parallel, where each stream has a unique time dilation rate
($\tau$). Our proposed MTDNN variant differs from the original models [17, 16]
by having different numbers of layers in the TDNN streams, depending on $\tau$
(Figures 2(b) and 2(c)).

Prior to the TDNN streams, input features $\mathbf{x}$ are first processed by
a single stream 2-D Convolutional Neural Network (CNN) in the front-end of the
network,

$\mathbf{h}=\mathit{Stacked{\text{-}}2D{\text{-}}CNN}(\mathbf{x})$ (3)

where $\mathit{Stacked{\text{-}}2D{\text{-}}CNN}$ stands for the stack of 2-D
convolutional layers with $3\times 3$ kernels. Inspired by [4], we apply
subsampling on the height axis after each alternate layer with a factor of 2,
in order to get compact embeddings, $\mathbf{h}$, which are then fed into
multiple streams of TDNNs222A time-delay neural network consists of 1-D
convolutional layers where the convolution is applied with frames that are
dilated on the time axis[28]. In our architecture, we employ the factorized
variant of TDNN introduced in [29].. Each stream of TDNNs has a unique time
dilation rate, $\tau$, encoding information in different temporal resolutions,

$\mathbf{z}^{n}_{t}=\mathit{Stacked{\text{-}}TDNN}(\mathbf{h}|\tau=t,N=n),$
(4)

where $\mathbf{z}^{n}_{\tau}$ are the latent variables at the output of the
final ($N^{\mathit{th}}$) TDNN layer, and $t\in\mathbb{Z}$. These are
concatenated and projected to the classification ($\mathit{softmax}$) layer by
a pair of fully connected ($\mathit{FC}$) layers,

$a(s)=\mathit{softmax}(2\times\mathit{FC(Concat}(\mathbf{z}^{N}_{\tau_{1}},\mathbf{z}^{N}_{\tau_{2}},...,\mathbf{z}^{N}_{\tau_{K}}))),$
(5)

where $a(s)$ is the activation of the $\mathit{softmax}$ layer corresponding
to the phoneme state $s$ and $K$ is the number of TDNN streams. We decide the
number of layers per stream w.r.t. to the receptive field ($\mathit{RF}$) of
the nodes at the top TDNN layer,

$\mathit{RF}_{\mathbf{z}^{N}_{\tau}}=2\times l\times\tau\times N,$ (6)

where $l$ is the frame length of the acoustic feature vectors. Note that we
include an additional 1-D convolutional layer just before the TDNN streams.
This layer does not use dilation, in order not to skip any frames.

##  4 Data

###  4.1 Training Set

The acoustic models of the previously presented ALT systems in the literature
are built on either monophonic or polyphonic music recordings. In general,
monophonic models are trained on the $\textit{DAMP}^{\mathit{train}}$
dataset[2, 4, 7], while DALI is utilized for polyphonic models[9]. We merge
these two datasets, exploiting their size and musical variability. We curated
the polyphonic subset of the dataset on the recordings from the most recent
version (v2.0) of the DALI dataset [30], and included only those songs for
which the Youtube links were available and still in use at the time of the
audio retrieval.

###  4.2 Evaluation Sets

We perform model selection and optimization on the subsets of the DAMP and
DALI datasets, representing monophonic and polyphonic domains respectively.
For $\textit{DAMP}^{\mathit{test}}$, we use the test split introduced in [2].
For testing the lyrics transcription performance on polyphonic recordings, we
curated a new subset of DALI-v1.0, which we give the data selection procedure
below. Finally, we evaluate our best performing model on the three benchmark
datasets used in the literature, namely the Jamendo, Hansen and Mauch sets and
provide a comparison with the state of the art in Section 6.5.

Set | Words | Uniq. Words | # Utt. | # Rec | # Singers | Avg. Utt. Dur. | Total Dur.
---|---|---|---|---|---|---|---
LM-corpus | 13M | 100k | 2M | N/A | N/A | N/A | N/A
$\textit{DAMP}^{\mathit{train}}$ | 686k | 5.3k | 80k | 4.2k | 3k | 5sec | 112h
$\textit{DALI}^{\mathit{train}}$ | 1.1M | 25.5k | 227k | 4.1k | 1.5k | 2.48sec | 156h
$\textit{DAMP}^{\mathit{dev}}$ | 4k | 695 | 482 | 66 | 38 | 5.12sec | 41min
$\textit{DALI}^{\mathit{dev}}$ | 5.7k | 941 | 1.7k | 34 | 16 | 2.41sec | 48min
$\textit{DAMP}^{\mathit{test}}$ | 4.6k | 840 | 479 | 70 | 40 | 6sec | 48min
$\textit{DALI}^{\mathit{test}}$ | 62.8k | 4.2k | 240 | 240 | 160 | 233sec | 15.5h
Jamendo | 5.7k | 1k | 20 | 20 | 20 | 216sec | 72min
Hansen | 2.8k | 585 | 10 | 10 | 9 | 214sec | 35min
Mauch | 5.2k | 820 | 20 | 20 | 18 | 245sec | 82min

Table 1: Statistics of datasets used in experiments

For tuning the hyperparameters during evaluation, the language model scaling
factor and the word insertion penalty, we have used the combination of the
data from $\textit{DAMP}^{\mathit{dev}}$ split [2] and 20 recordings from
DALI-v2.0333This combined development set is denoted as dev in Section 6..

####  4.2.1 The DALI-test set

In this section, we give details of the curation procedure for the
$\textit{DALI}^{\mathit{test}}$ set. We began from the subset presented in
[31], which initially had 513 recordings and filtered it according to a number
of criteria. Numerous audio samples were not retrievable from the links
provided. We obtained the Youtube links through automatic search using
relevant key words. We discarded songs where the automatically retrieved
version was a live performance, had low audio quality or contained extra
background speech sections unrelated to its corresponding lyrics. For
consistency and fair evaluation, we did not include songs where the dominant
language was not English. We allowed for an artist to have at most 5 songs.
Among the remaining recordings, we manually selected a subset having a
relatively balanced distribution of singers’ gender, official release dates
over decades (see Figure 3) and variability in terms of singing styles, vocal
effects and music genre. Lyrics were initially obtained from the annotations
provided in [5] and manually verified following the steps explained in Section
5.1. The final version of $\textit{DALI}^{\mathit{test}}$ consists of 240
recordings, which sets the largest test set for lyrics transcription with
clean annotations. For open science, we publicly share the data identifiers,
cleaned lyrics annotations and a tutorial to retrieve the corresponding
Youtube links at “https://github.com/emirdemirel/DALI-TestSet4ALT”.

Figure 3: Songs per decade in ALT evaluation sets

##  5 Experimental Setup

###  5.1 Lyrics Preprocessing

Prior to being utilized for training, raw lyrics data automatically retrieved
from online resources (as in DALI) needs to be normalized, as the
transcription rules applied by lyrics providers are not standardized. We
remove all special ASCII characters except apostrophes. We convert numeric
characters to their alphabetic correspondence. All text is converted to upper
case. We observed several samples with erroneous hyphenation, explicit
syllabification and repeating letters (possibly indicating longer uttered
syllables or vowels). To cope with these, we apply automatic hyphenation
correction and canonicalization using the standard open-source NLTK
tools444These steps are potentially language specific.. The output lyrics are
then verified and corrected manually.

###  5.2 Language and Pronunciation Models

Lyrics often contain uncommon words that are not very likely to exist in
standard pronunciation dictionaries. For such words that are not in the
lexicon, or out-of-vocabulary (OOV) words, we generate pronunciations using a
pretrained grapheme-to-phoneme (G2P) converter[32]. In order to produce fair
and reproducible results, we generate pronunciations for the OOV words in the
evaluation sets as well, and do not skip these during evaluation. We utilize
the commonly used CMU English Pronunciation Dictionary555Link:
https://github.com/Alexir/CMUdict/blob/master/cmudict-0.7b. as the lexicon and
generate alternative pronunciations by duplicating the vowel phonemes for each
word pronunciation, inspired by the improvements observed in [33, 7]. A 4-gram
language model (LM) is constructed using the SRILM Toolkit [34]. We use the
combination of the lyrics corpus in [2]666This corpus contains lyrics from all
the songs of the artists included in the Billboard charts between 2015-2018,
plus the lyrics in $\textit{DAMP}^{\mathit{train}}$. and
$\textit{DALI}^{\mathit{train}}$. For scientific evaluation, we exclude any
songs which overlap with those in the evaluation sets.

###  5.3 Discriminating Instrumental and Silent Regions

The hybrid DNN-HMM ASR framework approaches the continuous word recognition
task essentially as a sequence decoding problem. Within this scope, the
presence of instrumental accompaniment, especially during non-vocal regions,
may disrupt the decoding path, potentially causing cumulative errors during
transcription and alignment. Traditionally, non-speech regions are represented
with a special silence token within the target class set during recognition.
Here, we propose using separate tokens for the non-vocal instances in
monophonic and polyphonic recordings. Prior to training, we associate these
tokens with their corresponding silence/music instances by explicitly adding
tags at the beginnings and the ends of the ground truth lyrics of each
utterance (see Table 2). These tags are represented as words in the lexicon
where their pronunciations correspond to the relevant silence token.

$\mathbf{w}$
---
Raw |  $w_{1}$ $w_{2}$ … $w_{N}$
DAMP |  $<$silence$>$ $w_{1}$ $w_{2}$ … $w_{N}$ $<$silence$>$
DALI |  $<$music$>$ $w_{1}$ $w_{2}$ … $w_{N}$ $<$music$>$

Table 2: Music / silence tagging w.r.t. the dataset

For silence and music tagging, we exploit our training data. As we know the
recordings in DAMP and DALI are monophonic and polyphonic respectively, we
apply tagging w.r.t. the dataset. The pronunciations of these pseudo-word tags
are represented with distinct phonemes in the lexicon.

###  5.4 Generating Phoneme Alignments

Since the neural network optimization is performed w.r.t. phoneme posteriors
(as explained in Section 2.1), we need to extract their timings, i.e.
alignments. To generate these, we train a triphone Gaussian Mixture Model
(GMM) - HMM model on “singer adaptive” features [35], following the standard
Kaldi recipe777We execute the GMM-HMM pipeline at
https://github.com/emirdemirel/ALTA, which is almost the same procedure as the
standard librispeech recipe, with tuned hyperparameters for singing data.. At
this stage, we compute per-word pronunciation probabilities following the
steps in [36], and retrain another triphone model using the updated lexicon
transducer. Using this new model, we apply forced alignment[22] on the
training data for generating the phoneme and word alignments.

###  5.5 Neural Network Training

DNN training is based on the Kaldi - chain recipe. In the feature space, we
use 40-band filterbank features extracted with a hop size of 10ms and window
size of 30ms. To achieve singer-adaptive training, we utilize i-Vectors[37]
which represent the singer identity information via global embedding vectors.
Frame subsampling is applied with a factor of 3 in this training scheme where
each subsampled frame in the input of the neural network is considered to
represent $l=3\times 10$ms = $30$ms of context. The data is fed into the
network as audio chunks of 4.2 seconds (140 frames) in minibatches of 32. We
apply a decaying learning rate with beginning and final rates of $10^{-4}$ and
$10^{-5}$ respectively. Stochastic gradient descent is used as the optimizer.
The training is done for 6 epochs.

##  6 Results

We report lyrics transcription results based on word error rate (WER). We
begin by comparing performances obtained using DAMP and/or DALI in training
the acoustic model. Then we test our proposed idea of discriminating silence
and accompaniment instances by using separate tokens, and perform experiments
testing different topologies of the MTDNN architecture. To boost the
performance further, we train a final model on augmented data and provide a
comparison of our results with previously published models.

###  6.1 Multi-Domain Training

According to Table 3, the model trained on $\textit{DAMP}^{\mathit{train}}$
performs relatively well on $\textit{DAMP}^{\mathit{test}}$, however its
performance drops dramatically on polyphonic recordings. On the other hand,
much better recognition rates are observed on $\textit{DALI}^{\mathit{test}}$
when a polyphonic model is used, but then the polyphonic model performs poorly
on a cappella recordings. Finally, using recordings from both the monophonic
and polyphonic domains results in improved performance on both polyphonic and
monophonic test sets, although the improvement is marginal on the monophonic
$\textit{DAMP}^{\mathit{test}}$ set.

###  6.2 Music / Silence Modeling

Next, we test whether the explicit music/silence tagging improves
transcription results. At this stage, we use a single stream architecture
($M^{\mathit{single}}_{8}$ in Table 4). Tagging is applied only in
constructing the GMM-HMM model and for generating alignments. The
music/silence tags were removed during neural network training. Table 3 shows
that alignment with music/silence tags did result in considerably improved
recognition results for polyphonic recordings, but no improvement was evident
for the monophonic case.

Train Set  | $\textit{DAMP}^{\mathit{test}}$ | $\textit{DALI}^{\mathit{test}}$
---|---|---
DAMP | 17.64 | 78.42
DALI | 61.95 | 59.19
DAMP \+ DALI | 17.14 | 53.86
+ music/sil tag  | 17.29 | 47.00

Table 3: Multi-domain training and music/silence tagging results

###  6.3 Neural Architecture Design

Here, we test various parameterizations of the multistreaming architecture. In
this stage, we did not use the explicit music and silence tagging for training
the models. As mentioned in Section 3, we diversify each stream of TDNNs in
terms of the number of hidden layers and/or their dimensions. In addition to
achieving improved performance, the goal of these modifications is to exploit
the temporal context to its full extent. For this, we calculate the number of
TDNN layers included w.r.t. the resulting
$\mathit{RF}_{\mathbf{z}^{N}_{\tau}}$.

In all MTDNN variants tested, we use 3 TDNN streams with $\tau\in\\{3,6,9\\}$.
We begin with finding the optimal number of TDNN layers for the stream with
the smallest $\tau$. For rapid experimentation, we use single-streaming TDNN
models ($M^{single}_{N}$ in Table 4). According to Table 4, using 9 layers
sets the optimal setup for $\tau=3$, having
$\mathit{RF}_{\mathbf{z}^{N=9}_{\tau=3}}=1620$ms. Note that further increasing
the number of TDNN layers to 10
($\mathit{RF}_{\mathbf{z}^{10}_{\tau=3}}=1800$ms) did not result in improved
recognition, and the model complexity was much higher (Figure 4). Therefore,
we chose as our baseline a single-stream model with 9 TDNN layers.

,  Stream Layers Dimension dev $\textit{DAMP}^{\mathit{test}}$
$\textit{DALI}^{\mathit{test}}$ $M^{\mathit{single}}_{7}$ 3 7 512 28.06 17.08
54.52 $M^{\mathit{single}}_{8}$ 3 8 512 28.05 17.14 53.44
$M^{\mathit{single}}_{9}$ 3 9 512 27.68 17.08 52.25 $M^{\mathit{single}}_{10}$
3 10 512 27.67 17.21 53.58 $M^{\mathit{multi}}_{9,a}$ 3-6-9 (9,9,9)
(512,512,512) 26.69 16.75 51.38 $M^{\mathit{multi}}_{9,b}$ 3-6-9 (9,4,3)
(512,512,512) 26.65 16.45 49.32 $M^{\mathit{multi}}_{9,c}$ 3-6-9 (9,9,9)
(512,256,172) 27.13 16.08 52.54 $M^{\mathit{multi}}_{9,d}$ 3-6-9 (9,4,3)
(512,256,172) 27.38 16.62 51.92

Table 4: Experiments on NN design

Next, we perform ablative tests on four variants of the multistream
architecture (notated as $M^{\mathit{multi}}_{9,\\{a,b,c,d\\}}$). Model
$M^{\mathit{multi}}_{9,a}$ have identical TDNN structures (except for $\tau$),
whereas the variants $M^{\mathit{multi}}_{9,\\{b,c\\}}$ have reduced $N$ or
hidden dimensions respectively w.r.t. $\tau$ at each stream. Both dimensions
of model reduction are applied on $M^{\mathit{multi}}_{9,d}$. In models
$M^{\mathit{multi}}_{9,\\{b,d\\}}$, we reduced the number of layers, $N$ for
the streams with larger $\tau$ to keep $\mathit{RF}_{\mathbf{z}^{N}_{\tau}}$
similar across all streams. $M^{\mathit{multi}}_{9,\\{b,d\\}}$ have 4 and 3
layers at the streams with $\tau=6$ and $\tau=9$ having $\mathit{RF}$ values
of 1440 and 1620ms respectively. On the other hand, adding one more layer on
the streams with $\tau=6,9$ would result in having
$\mathit{RF_{\mathbf{z}^{N}_{\tau}}}\geq$ 1800ms which is shown above to be
suboptimal in the single-stream case (see results for
$M^{\mathit{single}}_{10}$).

###  6.4 Model Selection

The proposed multistreaming setups except $M^{\mathit{multi}}_{9,c}$
outperformed their single-stream counterpart, $M^{\mathit{single}}_{9}$,
particularly on $\textit{DALI}^{\mathit{test}}$. The best results are achieved
with $M^{\mathit{multi}}_{9,b}$ which has unique $N$ layers across all streams
with the same hidden layer dimension.

To increase confidence in model selection, we investigate other operational
aspects of the tested models. In Figure 4, we compare the number of trainable
parameters which is a variable related to model complexity, and the real-time
factor (RTF) that measures how fast a model operates during inference. We
compute RTF’s based on the inference times across all the data used in
evaluation. We repeat this 5 times and report the mean of all iterations per
model. These iterations are performed on an
Intel${}^{\text{{\textregistered}}}$ Xeon${}^{\text{{\textregistered}}}$ Gold
5218R CPU.

Figure 4: Num. trainable parameters (left) & RTFs (right)

According to Figure 4, our best performing model $M^{\mathit{multi}}_{9,b}$
has the second largest number of trainable parameters. Its model complexity is
however much lower than that of $M^{\mathit{multi}}_{9,a}$, the architecture
presented in [16]. In terms of run time, all multistreaming models performed
faster than single-stream models, with $M^{\mathit{multi}}_{9,b}$ being among
the fastest. This shows that our compact variant has a reduced inference time
with an improved recognition rate as hypothesized in Section 1.

###  6.5 Comparison with the State of the Art

At this last step, we train a final model combining the music / silence aware
alignment with the best performing MTDNN architecture,
$M^{\mathit{multi}}_{9,b}$. To boost the performance further, we apply data
augmentation via speed perturbation with the factors of 0.9 and 1.1, tripling
the size of the training data. In Table 5, we compare our final model with
other lyrics transcribers reported in the literature. We retrained the
acoustic models in [2]
($M_{\text{\cite[cite]{[\@@bibref{}{dabike2019automatic}{}{}]}}}$), and [4]
($M_{\text{\cite[cite]{[\@@bibref{}{demirel2020automatic}{}{}]}}}$), using the
corresponding publicly shared repositories.
$M_{\text{\cite[cite]{[\@@bibref{}{gupta2020automatic}{}{}]}}}$ is based on
the pretrained acoustic model shared at
https://github.com/chitralekha18/AutoLyrixAlign. The same language model is
used in constructing the decoding graphs for all the models in Table 5. Note
that
$M_{\text{\cite[cite]{[\@@bibref{}{dabike2019automatic}{}{}]},\cite[cite]{[\@@bibref{}{demirel2020automatic}{}{}]}}}$
are trained on DAMP (monophonic) and
$M_{\text{\cite[cite]{[\@@bibref{}{gupta2020automatic}{}{}]}}}$ is trained on
DALI (polyphonic) datasets. We used the best performing language model scaling
factor when reporting the results in Table 5. We were not able to generate
results on $\textit{DALI}^{\mathit{test}}$ using
$M_{\text{\cite[cite]{[\@@bibref{}{gupta2020automatic}{}{}]}}}$ due to the
model being highly memory intensive, as also reported in [8].

In addition to these, we provide a comparison with the state of the art. The
best WER score reported on $\textit{DAMP}^{\mathit{test}}$ is based on
$M_{\text{\cite[cite]{[\@@bibref{}{demirel2020automatic}{}{}]}}}$ and applies
rescoring on the word lattices generated after the first-pass decoding using
an RNNLM[38]. We did not apply RNNLM rescoring as we did not achieve
consistent improvements across different test sets according to our empirical
observations. For a fair comparison, we also include the best results in [4]
achieved via n-gram LMs (the scores in paranthesis in Table 5). On Jamendo,
the best WER scores were reported in [8] where the inference was performed on
source separated vocals. For Hansen and Mauch datasets, the best results are
provided as reported in [9]888Note that the reason for the WER difference
between $M_{\text{\cite[cite]{[\@@bibref{}{gupta2020automatic}{}{}]}}}$ and
the scores reported in [9] is due to the bigger language model we used,
despite both models having the same acoustic model.. In order to evade
optimistic results, we have discarded the overlapping songs between Hansen,
Mauch and $\textit{DALI}^{\mathit{train}}$ during training the final model.

| WER
---|---
| $\textit{DAMP}^{\mathit{test}}$ | $\textit{DALI}^{\mathit{test}}$ | Jamendo | Hansen | Mauch
$M_{\text{\cite[cite]{[\@@bibref{}{dabike2019automatic}{}{}]}}}$ | 16.86 | 67.12 | 76.37 | 77.59 | 76.98
$M_{\text{\cite[cite]{[\@@bibref{}{gupta2020automatic}{}{}]}}}$ | 56.90 | N/A | 50.64 | 39.00 | 40.43
$M_{\text{\cite[cite]{[\@@bibref{}{demirel2020automatic}{}{}]}}}$ | 17.16 | 76.72 | 66.96 | 78.53 | 78.50
S.O.T.A |  14.96 (17.01) [4] | N/A | 51.76 [8] | 47.01 [9] | 44.02 [9]
MSTRE-Net | 15.38 | 42.11 | 34.94 | 36.78 | 37.33

Table 5: Comparison with the state-of-the-art.

The results above show that MSTRE-Net outperforms all of the previously
presented models on the polyphonic sets, with more than 15% , 7% and 6%
absolute WER improvements achieved on the Jamendo, Hansen and Mauch datasets
compared to the previous state of the art respectively. Notably, we achieved
less than 50% WER on the large $\textit{DALI}^{\mathit{test}}$ set indicating
more than half of the words across 240 songs were correctly predicted. Our
model also has the best results on $\textit{DAMP}^{\mathit{test}}$ achieved
via n-gram LM.

##  7 Conclusion

We have introduced MSTRE-Net, a novel compact variant of the multistreaming
neural network architecture, which outperforms previously proposed automatic
lyrics transcription models. Our model achieved these results with lower model
complexity and inference time. In addition, we showed that recognition rates
improved across all evaluation sets after leveraging both polyphonic and
monophonic data in training the acoustic model. We proposed a novel data
preprocessing method for generating alignments prior to neural network
training which resulted in considerably better word recognition rates from
polyphonic recordings compared to the baseline approach. Finally, we curated a
new evaluation set that is more comprehensive and varied, while having a much
larger size compared to the previous test data used in research. For
reproducibility and open science, the identifiers and a tutorial on making use
of this data will be shared with the research community.

Our final model outperformed the previously reported best ALT results by a
large margin, setting the new state-of-the-art. Through these results, we have
taken an important step in increasing the potential and the possibility for
ALT being an applicable technology in both Music Information Retrieval
research and the music technology industry.

## References

  * [1] P. A. Fine and J. Ginsborg, “Making myself understood: Perceived factors affecting the intelligibility of sung text,” _Frontiers in Psychology_ , vol. 5, p. 809, 2014.
  * [2] G. R. Dabike and J. Barker, “Automatic lyrics transcription from karaoke vocal tracks: Resources and a baseline system,” in _Interspeech_ , 2019.
  * [3] C. Gupta, R. Tong, H. Li, and Y. Wang, “Semi-supervised lyrics and solo-singing alignment,” in _International Society for Music Information Retrieval Conference (ISMIR)_ , 2018.
  * [4] E. Demirel, S. Ahlbäck, and S. Dixon, “Automatic lyrics transcription using dilated convolutional neural networks with self-attention,” in _International Joint Conference on Neural Networks (IJCNN)_ , 2020.
  * [5] G. Meseguer-Brocal, A. Cohen-Hadria, and G. Peeters, “DALI: A large dataset of synchronized audio, lyrics and notes, automatically created using teacher-student machine learning paradigm,” in _International Society for Music Information Retrieval Conference (ISMIR)_ , 2019.
  * [6] “Smule sing! 300x30x2 dataset,” accessed April, 2021, https://ccrma.stanford.edu/damp/.
  * [7] E. Demirel, S. Ahlbäck, and S. Dixon, “Computational pronunciation analysis in sung utterances,” in _European Conference on Signal Processing (EUSIPCO)_ , 2021.
  * [8] ——, “Low resource audio-to-lyrics alignment from polyphonic music recordings,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [9] C. Gupta, E. Yılmaz, and H. Li, “Automatic lyrics transcription in polyphonic music: Does background music help?” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020.
  * [10] H. Bourlard and S. Dupont, “A new ASR approach based on independent processing and recombination of partial frequency bands,” in _Fourth International Conference on Spoken Language Processing (ICSLP)_ , 1996.
  * [11] J. Allen, “How do humans process and recognize speech?” _IEEE Transactions on Speech and Audio Processing_ , vol. 2, no. 4, pp. 567–577, 1994\.
  * [12] H. Hermansky and P. Fousek, “Multi-resolution RASTA filtering for TANDEM-based ASR,” in _Interspeech_ , 2005.
  * [13] Z. Tüske, R. Schlüter, and H. Ney, “Acoustic modeling of speech waveform based on multi-resolution, neural network signal processing,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018.
  * [14] S. H. Mallidi, T. Ogawa, K. Veselỳ, P. S. Nidadavolu, and H. Hermansky, “Autoencoder based multi-stream combination for noise robust speech recognition,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [15] K. J. Han, R. Prieto, and T. Ma, “State-of-the-art speech recognition using multi-stream self-attention with dilated 1D convolutions,” in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2019.
  * [16] K. J. Han, J. Pan, V. K. N. Tadala, T. Ma, and D. Povey, “Multistream CNN for robust acoustic modeling,” in _Interspeech_ , 2020.
  * [17] J. Pan, J. Shapiro, J. Wohlwend, K. J. Han, T. Lei, and T. Ma, “ASAPP-ASR: Multistream CNN and self-attentive SRU for SOTA speech recognition,” in _Interspeech_ , 2020.
  * [18] A. M. Kruspe, “Training phoneme models for singing with “songified” speech data.” in _International Society for Music Information Retrieval Conference (ISMIR)_ , 2015.
  * [19] D. Stoller, S. Durand, and S. Ewert, “End-to-end lyrics alignment for polyphonic music using an audio-to-character recognition model,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019.
  * [20] J. K. Hansen, “Recognition of phonemes in a-cappella recordings using temporal patterns and mel frequency cepstral coefficients,” in _Ninth Sound and Music Computing Conference (SMC)_ , 2012.
  * [21] M. Mauch, H. Fujihara, and M. Goto, “Integrating additional chord information into HMM-based lyrics-to-audio alignment,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 1, pp. 200–210, 2011.
  * [22] M. Gales and S. Young, “The application of hidden Markov models in speech recognition,” _Foundations and Trends in Signal Processing_ , 2008.
  * [23] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, and P. Schwarz, “The Kaldi speech recognition toolkit,” in _IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2011.
  * [24] M. Mohri, F. Pereira, and M. Riley, “Weighted finite-state transducers in speech recognition,” _Computer Speech & Language_, 2002.
  * [25] K. Veselỳ, A. Ghoshal, L. Burget, and D. Povey, “Sequence-discriminative training of deep neural networks,” in _Interspeech_ , 2013.
  * [26] D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI.” in _Interspeech_ , 2016.
  * [27] L. Bahl, P. Brown, P. De Souza, and R. Mercer, “Maximum mutual information estimation of hidden Markov model parameters for speech recognition,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 1986.
  * [28] V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [29] D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. Yarmohammadi, and S. Khudanpur, “Semi-orthogonal low-rank matrix factorization for deep neural networks.” in _Interspeech_ , 2018.
  * [30] G. Meseguer-Brocal, R. Bittner, S. Durand, and B. Brost, “Data cleansing with constrastive learning for vocal note event annotations,” in _International Society for Music Information Retrieval Conference (ISMIR)_ , 2020.
  * [31] A. Vaglio, R. Hennequin, M. Moussallam, G. Richard, and F. d’Alché Buc, “Multilingual lyrics-to-audio alignment,” in _International Society for Music Information Retrieval Conference (ISMIR)_ , 2020.
  * [32] J. R. Novak, D. Yang, N. Minematsu, and K. Hirose, “Phonetisaurus: A WFST-driven phoneticizer,” in _International Workshop on Finite State Methods and Natural Language Processing_ , 2012.
  * [33] C. Gupta, H. Li, and Y. Wang, “Automatic pronunciation evaluation of singing.” in _Interspeech_ , 2018.
  * [34] T. Alumäe and M. Kurimo, “Efficient estimation of maximum entropy language models with N-gram features: An SRILM extension,” in _Eleventh Annual Conference of the International Speech Communication Association_ , 2010\.
  * [35] T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, “A compact model for speaker-adaptive training,” in _Fourth International Conference on Spoken Language Processing_ , 1996.
  * [36] G. Chen, H. Xu, M. Wu, D. Povey, and S. Khudanpur, “Pronunciation and silence probability modeling for ASR,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [37] G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, “Speaker adaptation of neural network acoustic models using i-vectors,” in _IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2013.
  * [38] H. Xu, T. Chen, D. Gao, Y. Wang, K. Li, N. Goel, Y. Carmiel, D. Povey, and S. Khudanpur, “A pruned RNNLM lattice-rescoring algorithm for automatic speech recognition,” in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018\.