# Tuneable electron-magnon coupling of ferromagnetic surface states in PdCoO2

F. Mazzola<EMAIL_ADDRESS>SUPA, School of Physics and Astronomy,
University of St Andrews, St Andrews KY16 9SS, United Kingdom  C.-M. Yim
SUPA, School of Physics and Astronomy, University of St Andrews, St Andrews
KY16 9SS, United Kingdom  V. Sunko  SUPA, School of Physics and Astronomy,
University of St Andrews, St Andrews KY16 9SS, United Kingdom  Max Planck
Institute for Chemical Physics of Solids, Nöthnitzer Straße 40, 01187 Dresden,
Germany  S. Khim  Max Planck Institute for Chemical Physics of Solids,
Nöthnitzer Straße 40, 01187 Dresden, Germany  P. Kushwaha  CSIR-National
Physical Laboratory, Dr K S Kishnana Marg, New Delhi-110012, India  Max Planck
Institute for Chemical Physics of Solids, Nöthnitzer Straße 40, 01187 Dresden,
Germany  O. J. Clark  L. Bawden  SUPA, School of Physics and Astronomy,
University of St Andrews, St Andrews KY16 9SS, United Kingdom  I. Marković
SUPA, School of Physics and Astronomy, University of St Andrews, St Andrews
KY16 9SS, United Kingdom  Max Planck Institute for Chemical Physics of Solids,
Nöthnitzer Straße 40, 01187 Dresden, Germany  D. Chakraborti  SUPA, School of
Physics and Astronomy, University of St Andrews, St Andrews KY16 9SS, United
Kingdom  Max Planck Institute for Chemical Physics of Solids, Nöthnitzer
Straße 40, 01187 Dresden, Germany  T. K. Kim  Diamond Light Source, Harwell
Campus, Didcot, OX11 0DE, United Kingdom  M. Hoesch  Diamond Light Source,
Harwell Campus, Didcot, OX11 0DE, United Kingdom  DESY Photon Science,
Deutsches Elektronen-Synchrotron, Notkestraße 85, 22607 Hamburg, Germany  A.
P. Mackenzie  Max Planck Institute for Chemical Physics of Solids, Nöthnitzer
Straße 40, 01187 Dresden, Germany  SUPA, School of Physics and Astronomy,
University of St Andrews, St Andrews KY16 9SS, United Kingdom  P. Wahl  SUPA,
School of Physics and Astronomy, University of St Andrews, St Andrews KY16
9SS, United Kingdom  P. D. C. King<EMAIL_ADDRESS>SUPA, School of
Physics and Astronomy, University of St Andrews, St Andrews KY16 9SS, United
Kingdom

###### Abstract

Controlling spin wave excitations in magnetic materials underpins the
burgeoning field of magnonics. Yet, little is known about how magnons interact
with the conduction electrons of itinerant magnets, or how this interplay can
be controlled. Via a surface-sensitive spectroscopic approach, we demonstrate
a strong and highly-tuneable electron-magnon coupling at the Pd-terminated
surface of the delafossite oxide PdCoO2, where a polar surface charge mediates
a Stoner transition to itinerant surface ferromagnetism. We show how the
coupling can be enhanced 7-fold with increasing surface disorder, and
concomitant charge carrier doping, becoming sufficiently strong to drive the
system into a polaronic regime, accompanied by a significant quasiparticle
mass enhancement. Our study thus sheds new light on electron-magnon
interactions in solid-state materials, and the ways in which these can be
controlled.

##  I Introduction

Low-dimensional systems offer enormous potential for stabilising and
controlling novel magnetic states and textures [1; 2]. However, magnetic
fluctuations are known to destabilise long-range order in two-dimensional (2D)
systems (the famous Mermin-Wagner theorem [3]). While magnetic anisotropy can
allow long-range order to develop again [4; 5], fluctuations can still be
expected to play a crucial role. This necessitates their fundamental study,
and potentially provides new opportunities in which to control magnonic
excitations for spintronic technologies. 2D magnetic systems can be realised
in a top-down approach, by exfoliating few/single layers from a bulk van der
Waals magnet [1; 6; 7], or can be realised in a bottom-up approach, by
realising magnetic order in a thin-film or surface geometry. The delafossite
oxide, PdCoO2, has recently been demonstrated as a model host of the latter
[8].

While it is a non-magnetic Pd $d^{9}$ metal in the bulk [9], the polar crystal
structure (Fig. S7a) leads to pronounced charge carrier doping at the surface
[10; 8; 11; 12]. For the Pd-termination, the resulting self-doping is
electron-like, which acts to shift a large peak in the unoccupied density of
states towards the Fermi level. This in turn triggers a Stoner instability,
generating a 2D ferromagnetic surface layer, as predicted by density-
functional theory [11; 8] and confirmed from electronic structure [8] and
anomalous Hall measurements [13]. This thus provides a model environment,
accessible to spectroscopic probes, in which to study the influence of
magnetic excitations on the electronic structure of a 2D magnet. Here, we use
angle-resolved photoemission (ARPES) and scanning-tunnelling microscopy and
spectroscopy (STM/S) to investigate this system, finding evidence for a strong
and highly-tuneable electron-magnon coupling.

Figure 1: Electronic structure of the Pd-terminated ferromagnetic surface of
PdCoO2. (a) Schematic of the crystal structure (side view) of PdCoO2. After
sample cleaveage, a CoO2 surface termination (left) and a Pd termination
(right) are both present. (b) ARPES spectra ($h\nu=80$ eV along
$\bar{\Gamma}$-M̄ and $h\nu=82$ eV along $\bar{\Gamma}$-K̄) show the
superposition of spectral weight arising from both terminations. The labelled
$\alpha$-$\beta$ and $\gamma$-$\delta$ bands derive from the Pd-terminated
surface, and represent exchange-split pairs by a surface ferromagnetism. (c)
STM topographic image of a Pd-terminated region. (d) The STM quasiparticle
interference map (bottom) from a Pd-terminated region is in good agreement
with the Fermi surface measured by ARPES (top), determined by the $\gamma$ and
$\delta$ bands. Signatures of such electron-like bands are also visible in
energy-dependent QPI measurements (e) for intra-$\gamma$ and intra-$\delta$
band scattering, reported together with the corresponding fits. Additional
spectral weight at 0.1 eV binding energy can be attributed to intra-$\alpha$
band scattering, in good agreement with the flat top of the $\alpha$ band
observed by ARPES.

##  II Results

###  II.1 Electronic structure of the Pd-terminated ferromagnetic surface

Figure S7 shows an overview of the surface electronic structure of PdCoO2 as
measured by angle-resolved photoemission spectroscopy (ARPES) and scanning
tunneling microscopy (STM). The crystal hosts two distinct surface
terminations (Fig. S7a): a Pd-terminated surface and a CoO2-terminated one.
These would be expected with approximately equal probability, with a typical
cleaved sample having both types present. Consistent with this, our measured
ARPES spectra shown in Fig. S7b exhibit signatures of the electronic states
associated with both surface terminations. A pair of heavy hole-like bands
around the Brillouin zone centre derive from the CoO2-terminated surface [10;
12], and will not be considered further here. Several additional low-energy
states are observed, which we have previously attributed as deriving from the
Pd-terminated surface [8]. Of particular relevance here are the two pairs of
exchange-split ferromagnetic surface states labelled ($\alpha$,$\beta$) and
($\gamma$,$\delta$) in Fig. S7b [8]. The very flat band top of the spin-
majority $\alpha$-band is visible $\approx 100$ meV below $E_{\mathrm{F}}$.
Its high associated density of states would sit at the Fermi level in the non-
magnetic state; this is responsible for triggering a Stoner transition to
itinerant surface ferromagnetism, [8; 11] pushing this flat band below
$E_{\mathrm{F}}$ as observed here.

To confirm that these key electronic states are indeed derived from the Pd-
terminated surface, we have performed STM measurements to selectively probe a
single surface termination (Fig. S7c) [14]. While the intrinsic defect density
in the bulk is extremely low [15], we find a number of defects are present in
our STM measurements. We tentatively attribute these to Pd vacancies at the
cleaved surface [16]. Clear quasiparticle interference (QPI) patterns are
visible around these, permitting a local measurement of the surface electronic
structure. The Fourier transform of the QPI patterns, shown in Fig. S7d,
exhibits two clear concentric circles, whose wavevector is in good agreement
with intra-band scattering of the $\gamma$ and $\delta$ Fermi surfaces
observed by ARPES. An energy-dependent cut through the QPI data (Fig. S7e)
confirms this assignment from the electron-like character of these pockets,
while an additional intense QPI signal at 100 meV below the Fermi level,
peaked around $\mathbf{q}=0$ and with little evident dispersion, is consistent
with the flat top of the $\alpha$ band observed in the ARPES.

###  II.2 Quasiparticle dynamics

Having confirmed the electronic structure of the ferromagnetic Pd-terminated
surface of PdCoO2, we will focus now on spectroscopic signatures of marked
electronic interactions evident for these surface states. Fig. 2a shows in
detail the $\gamma$ and $\delta$ bands. While the minority-spin $\delta$-band
appears to be well described as a simple parabolic band, the $\gamma$-band
exhibits several anomalies, or ‘kinks’, in its dispersion. These are visible
in the raw data, and in fits to momentum distribution curves (MDCs) shown as
blue markers in Fig. 2a. Typically, such spectral kinks are signatures of an
electron-boson coupling, which can be described via an electronic self-energy.
To investigate this further, we have extracted the self-energy from a self-
consistent analysis of our ARPES measurements (see Methods), and report the
Kramers-Kronig-consistent real and imaginary parts of the electron-boson self-
energy in Fig. 2b [17]

Two steep rises in the real-part of the electron-boson self-energy are visible
at $\hbar\omega_{1}\approx 50$ meV and at $\hbar\omega_{2}\approx 130$ meV.
These are in approximate agreement with phonon energies reported in the
literature for PdCoO2 [18; 19; 20]. The low-energy kinks of our measured data
can thus be satisfactorily attributed to electron-phonon coupling. However,
there is an additional broad hump in the extracted real part of the electron-
boson self energy which extends out to $>300$ meV. This is an implausibly high
energy to be due to a phonon mode (well above the highest phonon mode energies
of $\approx 150$ meV expected for PdCoO2 [18; 19; 20]). Indeed, a model self-
energy calculation including only coupling to phonon modes fails to capture
the high-energy part of the extracted experimental self energy for any
electron-phonon coupling strength (the best fit is shown as the grey lines in
Fig. 2b; see also Supplementary Fig. 2). We thus conclude that there must be a
third boson mode which is active here, with a characteristic energy scale of
$>200$ meV, and which exhibits a marked coupling to the electronic sub-system.
The high associated energy scale of this boson mode suggests that it may have
a magnetic origin: in other itinerant ferromagnets, the spin-wave spectra are
known to extend to similarly high energies of, for example, $>300$ meV in Fe,
$>500$ meV in Co and $\approx 500$ meV in Ni [21; 22].

Figure 2: Electron-boson coupling. (a) Electronic structure measurements
($h\nu=90$ eV, measured along the $\bar{\Gamma}$-K̄ direction) showing the
$\gamma$ and $\delta$ bands. The non-interacting, bare band, dispersion is
taken as a $\mathbf{k}\cdot\mathbf{p}$ band, set to match the Fermi
wavevectors of the experimental dispersions as described in the Methods. (b)
The real and imaginary parts of the electron-boson self energy extracted from
the data and from a Migdal-Eliashberg calculation as described in the Methods.
A two-phonon model (grey lines) fails to describe the experimentally
determined self energy, while a three-mode model (red line, with coupling to
two phonons plus a magnon mode) is in much better agreement. The arrows in (b)
show the characteristic mode energies. The real and imaginary parts of the
self-energy retain causality through Kramers-Kronig transformation.

Figure 3: Surface-dependent variation of electron-magnon coupling strengths.
(a) Electronic structure measurements of the $\gamma$ and $\delta$ bands as
function of hole-doping. A pronounced increase in the band renormalisation is
observed from the left to the right measurement, which were performed on
different patches of the same sample (S1 and S2) and on different samples (S3
and S4). (b) Fits of MDCs (shown here as a function of $k_{y}-k_{\mathrm{F}}$;
shifts in $k_{\mathrm{F}}$ of the $\gamma$ band between the samples are not
resolvable experimentally)

show a strong increase in effective mass at the Fermi level (decrease in the
Fermi velocity). By tracking the binding energy of the flat portion of the
$\alpha$ band (see Supplementary Fig. 3), we find that this is correlated with
an increasing hole doping (c). Fitting the extracted self energy for the these
and additional samples, we find that the increased Fermi velocity (red)
results due to a large increase in electron-magnon coupling strength, while
the electron-phonon coupling strength does not vary significantly. (d) Large-
scale topography measured by STM shows a large concentration of impurities
(likely Pd vacancies) distributed across the cleaved Pd-terminated surface,
which likely give rise to the variable hole doping determined above.

Previous ARPES measurements have observed signatures of the coupling of
electrons with such magnetic excitations in elemental magnetic metals [23; 24;
25; 26], including for their two-dimensional surface electronic states [27;
28], with spectroscopic signatures and energy scales very similar to those
observed here. This can be understood via a process similar to the observation
of electron-phonon coupling in ARPES: upon photoemission from a majority-state
band, the resulting photohole can be filled by an electron with opposite spin
accompanied by the emission or absorption of a magnon. Nonetheless, the
different dispersion relations of phonons and magnons leads to differences in
the functional form of the corresponding self-energy (see Methods). While we
cannot describe our measured self-energies solely using electron-phonon-based
models, extending them to include electron-magnon coupling leads to excellent
agreement with our experimentally-determined self-energies (Fig. 2b). The
determined characteristic mode energy (left as a free parameter in fits to our
experimental self-energies) is at 245 meV, while it exhibits a moderate
coupling strength, $\lambda_{\mathrm{el-mag}}=0.55\pm 0.05$, similar to, but
slightly higher than, the coupling strength to the two phonon modes at lower
energies: $\lambda^{(1)}_{\mathrm{el-ph}}=0.30\pm 0.02$ and
$\lambda^{(2)}_{\mathrm{el-ph}}=0.25\pm 0.03$. These values are in rather good
agreement with calculated electron-paramagnon and electron-phonon coupling
constants, respectively, in bulk elemental Pd [29; 30; 31], which is itself
thought to be close to a ferromagnetic instability.

Intriguingly, we find a strong sample-to-sample and spatial variation of the
strength of the electron-magnon coupling. These are summarised in Fig. 3a (see
also Supplementary Fig. 3). The electronic structure remains qualitatively the
same as that shown in Fig. S7. However, the measured Fermi velocity extracted
from fits to momentum distribution curves (Fig. 3b) decreases by a factor of
$\approx 3$ across these samples. At the same time, we observe a shift of the
binding energy of the flat portion of the $\alpha$-band towards the Fermi
level by $\approx{40}$ meV (Supplementary Fig. 4). The latter likely arises
due to surface Pd vacancies. These are readily apparent in our STM
measurements (Fig. 3d), and would act to partially counteract the polar
surface charge, in turn leading to a reduction of the electron-like doping at
the surface with respect to the hypothetical bulk-like surface termination.

For the STM measurements presented in Fig. 3d, we estimate a surface Pd
vacancy concentration of $\sim\\!2\%$. This would lead to a nominal surface
carrier density of 0.48 electrons/unit cell (el/u.c.), in good agreement with
the Luttinger count of the ARPES-measured Fermi surfaces from sample S1 (Fig.
2a). Assuming a rigid band shift with increasing Pd vacancy concentration, we
estimate from tight-binding analysis (see Supplementary Fig. 5) that the
variations observed across our samples here correspond to variations in the
surface electron concentration of a further 0.02 el/u.c., corresponding to a
further 2% of vacancies in the surface Pd layer. While high concentrations of
Pd vacancies would lead to exposed areas of CoO2-terminated surface, the low
concentrations of Pd vacancies observed here are thus best considered as
isolated point defects, consistent with our STM measurements and providing a
natural source of charge carrier doping.

###  II.3 Tuneable electron-magnon coupling

Taken together with the change in Fermi velocity, the above findings indicate
a dramatic enhancement of the many-body renormalisations of the surface
electronic structure with hole doping from surface Pd vacancies. From model
fits to the electronic self energy from our ARPES measurements, we find that
it is the magnon mode identified above which exhibits a strongly varying
coupling strength, while the electron-phonon coupling remains unchanged within
experimental error (Fig. 3c, see also Supplementary Fig. 6). The enhancement
of this electron-magnon coupling is dramatic, reaching coupling strengths on
the order of $\lambda\approx 3$, where a weak-coupling Migdal-Eliashberg
picture would be expected to break down.

Figure 4: Magnonic polarons. (a) Measured $\gamma$-band dispersion for the
sample with strongest electron-magnon coupling (S4, same data of Fig. 3a) but
with incresed saturation and different color scale). The $\gamma$ band is now
strongly renormalised, with the coherent quasiparticle band having an occupied
bandwidth of only $\approx 100$ meV (measurement taken at $h\nu=80$ eV). (b) A
replica of this band is evident, shifted to higher binding energies by the
characteristic magnon energy of $\approx 240$ meV, indicative of polaron
formation. (c) Energy distribution curves taken at the momentum value
indicated by the green line in (a) show how this replica band feature is
independent of photon energy, pointing to a strong intrinsic electron-magnon
coupling regime.

Indeed, for the sample marked S4 in Fig. 3, our weak-coupling models fail to
adequately describe the extracted dispersion from our ARPES measurements. We
show the measured dispersion again in Fig. 4 with a different contrast. It is
clear that the quasi-particle band is strongly renormalised, having an
occupied band width of only $\approx 100$ meV, significantly reduced compared
to the $>400$ meV occupied band width for sample S1 seen in Fig. 2. Within a
Migdal-Eliashberg approach, this would imply a quasiparticle residue
$Z=m_{0}/m^{*}<0.25$, a rather low value, and outside the regime of
applicability of this approximation. Consistent with this, instead of simply
generating a kink in the dispersion, we note that this strong electron-magnon
coupling leads to a replica of the quasiparticle band, evident in our
measurements as a weak dispersive feature with the same dispersion as the
quasiparticle band, but shifted to higher binding energies by $\approx 230$
meV (Fig. 4b). This is similar to the replica features observed due to strong
coupling with phonons [32; 33; 34; 35; 36] or plasmons [37; 38], generating
polaronic states. The separation between the quasiparticle band and the
replica band observed here is equal to the characteristic mode energy of the
magnon mode determined from fitting our data for lower coupling strengths
(Fig. 3), and so we attribute this as a shake-off replica due to the strong
electron-magnon coupling here. Its spectroscopic signatures are largely
independent of photon energy (Fig. 4c). Together with the large quasiparticle
mass renormalisation observed, this rules out that the replica band results
from extrinsic photoelectron energy loss [39], and instead indicates that the
intrinsic electron-magnon coupling becomes strong enough to drive the system
into a polaronic regime.

##  III Discussion

Our measurements above indicate a dramatic enhancement of electron-magnon
coupling due to increased disorder (vacancy concentration) of the cleaved
surface. This can impact the coupling of magnons to the itinerant electrons in
multiple ways. First, with the concomitant increased hole doping, the flat top
of the $\alpha$ band is shifted towards the Fermi level (Fig. 3c).
Correspondingly, the onset of the Stoner continuum from finite $\mathbf{q}$
spin-flip electron-hole excitations will be shifted to lower energies. The
collective magnon mode will thus enter the Stoner continuum more rapidly,
becoming Landau-damped and leading to a decreased quasiparticle lifetime, and
thus enhanced coupling strength, as we observe here. We also note that the
magnetism predicted in this surface layer by density-functional theory is only
stabilised by a surface relaxation which increases the surface Pd-O bond
length [8]. The stability of the magnetic order, and as a consequence the
strength of the electron-magnon coupling, is thus likely to be extremely
sensitive to structural distortions and defects. Indeed, we find that the
linewidth at the Fermi level also increases with hole doping here, a clear
indicator of an enhanced electron-impurity scattering pointing to increased
surface disorder which accompanies the increased electron-magnon coupling
strength.

The detailed mechanisms by which the electron-magnon coupling becomes enhanced
here require further detailed theoretical study. Nonetheless, our experimental
results point to a surprisingly-large response of this coupling to small
changes in carrier concentration and surface disorder. This firmly establishes
the surface states of PdCoO2 as a model system in which to investigate the
coupling of itinerant electrons to spin excitations, of fundamental importance
to understand the limits of stability of magnetic order in 2D. We hope that
the findings presented here motivate future studies aimed at gaining true
deterministic control over the large changes in electron magnon coupling which
we have observed. In this respect, we note the enormous recent progress on the
growth of thin films of PdCoO2 [40; 41; 42; 43], in some of which signatures
of the ferromagnetism of the Pd-terminated surface have already been reported
in transport measurements [13]. Such thin films would be ideally suited for
studies of gate-tuning of surface doping (or even surface disorder) via, e.g.,
ionic liquid gating. Our results above show that for changes in the surface
carrier concentration accessible to such gating techniques, the electron-
magnon coupling at PdCoO2 surfaces can be driven from a weak- to a strong-
coupling polaronic regime, opening tantalising new possibilities for studying
the creation and control of spin-polarons.

##  IV Materials and Methods

Angle-resolved photoemission: Single-crystal samples of PdCoO2 were grown via
a flux method in sealed quartz tubes [44]. These were cleaved in situ at the
measurement temperature of $\approx 10$ K. ARPES measurements were performed
at the I05 beamline of Diamond Light Source, using a Scienta R4000
hemispherical electron analyser, and photon energies between 70 and 90 eV. All
measurements used linear-horizontal ($p$-) polarised light. The lateral spot
size of the photon beam on the sample is $\approx 50$ $\mu$m.

STM measurements: The STM and tunneling spectroscopy experiments were
performed using a home-built low temperature STM that operates at a base
temperature down to 1.8 K [45]. Pt/Ir tips were used, and conditioned by field
emission on a gold target. Differential conductance ($dI/dV$) maps and single
point spectra were recorded by means of standard lock-in technique (see also
Supplementary Fig. 7), with the frequency of the bias modulation set at 413
Hz. The STM/S results reported here were obtained at a sample temperature of
4.2 K.

Self-energy analysis: The electron-boson self-energy was extracted through a
self-consistent analysis. The bare band is described by a nearly-parabolic
$\mathbf{k}\cdot\mathbf{p}$ band, whose Fermi wavevectors are fixed to the
experimental value. The parameters that describe the non-interacting band have
been iteratively determined such that the causality connection between the
extracted real and imaginary parts of the self energy enforced by Kramers-
Kronig transformation is preserved. The electron-electron contribution to the
self energy has been subtracted from the self-energies shown in Fig. 2b to aid
comparison of the electron-boson contribution. The full extracted self energy,
including electron-electron contribution, is shown in Supplementary Fig. 1.

The real and imaginary parts of the self-energy have been modeled using a
Migdal-Eliashberg approach. The electron-phonon interaction was described by a
conventional two-phonon Debye-model with characteristic cut-off energies.
Finite temperature was also included in the model, as described in Ref. [46;
47]. For the electron-magnon coupling, the magnon dispersion relation,
$\omega_{mgn}\propto q^{2}$, yields a density of states proportional to the
square root of the energy, which means $D_{mgn}\propto\omega^{\frac{1}{2}}$.
This manifests in a different shape of both real and imaginary parts of the
self energy compared to the one expected for phonons
(($D_{ph}\propto\omega$)). This approach is similar to what has been used in
Ref. [27]. Under the Migdal-Eliashberg approach, the contribution of electron-
boson coupling to the imaginary part of the self-energy is calculated as:

$\displaystyle\Im\Sigma^{electron-
boson}(\omega,T)=\pi\int_{0}^{\omega_{\mathrm{max}}}\alpha^{2}(\omega^{\prime})F(\omega^{\prime})[1+2n(\omega^{\prime})$
$\displaystyle\quad+\textit{f}(\omega+\omega^{\prime})-f(\omega-\omega^{\prime})]d{\omega^{\prime}}$

where $\omega_{\mathrm{max}}$ is the highest energy allowed for the boson
mode, $f(\omega)$ and $n(\omega)$ are the fermion and boson distribution,
respectively, and $T$ is the temperature. For the phonons,
$\alpha^{2}F(\omega)=0$ if $\omega>\omega_{\mathrm{max}}$ and
$\alpha^{2}F(\omega)=\lambda(\omega/\omega_{\mathrm{max}})^{2}$ for
$\omega<\omega_{\mathrm{max}}$. In an analogous way, for magnons,
$\alpha^{2}F(\omega)=0$ if $\omega>\omega_{\mathrm{max}}$ and
$\alpha^{2}F(\omega)=\lambda_{mgn}(\omega/\omega_{mgn})^{\frac{1}{2}}$ for
$\omega<\omega_{\mathrm{max}}$, where $\lambda_{mgn}$ represents the electron-
magnon coupling strength of the system [27].

##  V Acknowledgements

We thank C. Hooley, T. Frederiksen, G. van der Laan, G. Panaccione, H. Rosner
and G. Siemann for useful discussions. We gratefully acknowledge support from
the European Research Council (through the QUESTDO project, 714193), the Royal
Society, the Max-Planck Society, and the UKRI Engineering and Physical
Sciences Research Council (Grant No. EP/S005005/1). We thank Diamond Light
Source for access to Beamline I05 (Proposals SI12469, SI14927, and SI16262),
which contributed to the results presented here. V.S., O.J.C., and L.B.
acknowledge the EPSRC for PhD studentship support through Grants EP/L015110/1,
EP/K503162/1, and EP/G03673X/1, respectively. I.M. and D.C. acknowledge
studentship support from the International Max-Planck Research School for
Chemistry and Physics of Quantum Materials.

## References

  * [1] Burch, K., Mandrus, D. & Park, J. Magnetism in two-dimensional van der Waals materials. Nature. 563, 47-52 (2018)
  * [2] Huang, B., McGuire, M., May, A., Xiao, D., Jarillo-Herrero, P. & Xu, X. Emergent phenomena and proximity effects in two-dimensional magnets and heterostructures. Nature Materials. 19, 1276-1289 (2020)
  * [3]
  * [4] Onsager, L. Crystal Statistics. I. A Two-Dimensional Model with an Order-Disorder Transition. Phys. Rev.. 65, 117-149 (1944)
  * [5] Gibertini, M., Koperski, M., Morpurgo, A. & Novoselov, K. Magnetic 2D materials and heterostructures. Nature Nanotechnology. 14, 408-419 (2019)
  * [6] Gong, C., Li, L., Li, Z., Ji, H., Stern, A., Xia, Y., Cao, T., Bao, W., Wang, C., Wang, Y., Qiu, Z., Cava, R., Louie, S., Xia, J. & Zhang, X. Discovery of intrinsic ferromagnetism in two-dimensional van der Waals crystals. Nature. 546, 265-269 (2017)
  * [7] Huang, B., Clark, G., Navarro-Moratalla, E., Klein, D., Cheng, R., Seyler, K., Zhong, D., Schmidgall, E., McGuire, M., Cobden, D., Yao, W., Xiao, D., Jarillo-Herrero, P. & Xu, X. Layer-dependent ferromagnetism in a van der Waals crystal down to the monolayer limit. Nature. 546, 270-273 (2017)
  * [8] Mazzola, F., Sunko, V., Khim, S., Rosner, H., Kushwaha, P., Clark, O., Bawden, L., Marković, I., Kim, T., Hoesch, M., Mackenzie, A. & King, P. Itinerant ferromagnetism of the Pd-terminated polar surface of PdCoO2. Proceedings Of The National Academy Of Sciences. 115, 12956-12960 (2018)
  * [9] Mackenzie, A. The properties of ultrapure delafossite metals. Reports On Progress In Physics. 80, 032501 (2017)
  * [10] Sunko, V., Rosner, H., Kushwaha, P., Khim, S., Mazzola, F., Bawden, L., Clark, O., Riley, J., Kasinathan, D., Haverkort, M., Kim, T., Hoesch, M., Fujii, J., Vobornik, I., Mackenzie, A. & King, P. Maximal Rashba-like spin splitting via kinetic-energy-coupled inversion-symmetry breaking. Nature. 549, 492-496 (2017)
  * [11] Kim, K., Choi, H. & Min, B. Fermi surface and surface electronic structure of delafossite PdCoO2. Phys. Rev. B. 80, 035116 (2009)
  * [12] Noh, H., Jeong, J., Jeong, J., Cho, E., Kim, S., Kim, K., Min, B. & Kim, H. Anisotropic Electric Conductivity of Delafossite PdCoO2 Studied by Angle-Resolved Photoemission Spectroscopy. Phys. Rev. Lett.. 102, 256404 (2009)
  * [13] Harada, T., Sugawara, K., Fujiwara, K., Ito, S., Nojima, T., Takahashi, T., Sato, T. & Tsukazaki, A. Anomalous Hall effect at the spontaneously electron-doped polar surface of PdCoO2 ultrathin films. ArXiv:1908.08173. (2019)
  * [14] Yim, C., Chakraborti, D., Rhodes, L., Khim, S., Mackenzie, A. & Wahl, P. Quasiparticle interference and quantum confinement in a correlated Rashba spin-split 2D electron liquid. Science Advances. 7 (2021)
  * [15] Sunko, V., McGuinness, P., Chang, C., Zhakina, E., Khim, S., Dreyer, C., Konczykowski, M., Borrmann, H., Moll, P., König, M., Muller, D. & Mackenzie, A. Controlled Introduction of Defects to Delafossite Metals by Electron Irradiation. Phys. Rev. X. 10, 021018 (2020)
  * [16] Such surface defects could in principle also be due to residual H absorption [48], although we note that the STM measurements here were performed in cryogenic vacuum making that scenario less likely.
  * [17] See Supplementary Fig. 1 for the full self energy with the electron-electron contribution included.
  * [18] Cheng, L., Yan, Q. & Hu, M. The role of phonon–phonon and electron–phonon scattering in thermal transport in PdCoO2. Phys. Chem. Chem. Phys.. 19, 21714-21721 (2017)
  * [19] Kumar, S., Gupta, H. & Karandeep First principles study of structural, bonding and vibrational properties of PtCoO2, PdCoO2 and PdRhO2 metallic delafossites. Journal Of Physics And Chemistry Of Solids. 74, 305-310 (2013)
  * [20] Takatsu , H., Yonezawa , S., Mouri , S., Nakatsuji , S., Tanaka , K. & Maeno , Y. Roles of High-Frequency Optical Phonons in the Physical Properties of the Conductive Delafossite PdCoO2. Journal Of The Physical Society Of Japan. 76, 104701 (2007)
  * [21] Halilov, S., Eschrig, H., Perlov, A. & Oppeneer, P. Adiabatic spin dynamics from spin-density-functional theory: Application to Fe, Co, and Ni. Phys. Rev. B. 58, 293-302 (1998)
  * [22] Kubler, J. Theory of Itinerant Electron Magnetism. Oxford University Press. (2009)
  * [23] Hayashi, H., Shimada, K., Jiang, J., Iwasawa, H., Aiura, Y., Oguchi, T., Namatame, H. & Taniguchi, M. High-resolution angle-resolved photoemission study of electronic structure and electron self-energy in palladium. Phys. Rev. B. 87, 035140 (2013)
  * [24] Młyńczak, E., Müller, M., Gospodarič, P., Heider, T., Aguilera, I., Bihlmayer, G., Gehlmann, M., Jugovac, M., Zamborlini, G., Tusche, C., Suga, S., Feyer, V., Plucinski, L., Friedrich, C., Blügel, S. & Schneider, C. Kink far below the Fermi level reveals new electron-magnon scattering channel in Fe. Nature Communications. 10, 505 (2019)
  * [25] Cui, X., Shimada, K., Hoesch, M., Sakisaka, Y., Kato, H., Aiura, Y., Higashiguchi, M., Miura, Y., Namatame, H. & Taniguchi, M. Angle-resolved photoemission spectroscopy study of Fe(110) single crystal: Many-body interactions between quasi-particles at the Fermi level. Surface Science. 601, 4010-4012 (2007)
  * [26] Młyńczak, E., Eschbach, M., Borek, S., Minár, J., Braun, J., Aguilera, I., Bihlmayer, G., Döring, S., Gehlmann, M., Gospodarič, P., Suga, S., Plucinski, L., Blügel, S., Ebert, H. & Schneider, C. Fermi Surface Manipulation by External Magnetic Field Demonstrated for a Prototypical Ferromagnet. Phys. Rev. X. 6, 041048 (2016)
  * [27] Hofmann, A., Cui, X., Schäfer, J., Meyer, S., Höpfner, P., Blumenstein, C., Paul, M., Patthey, L., Rotenberg, E., Bünemann, J., Gebhard, F., Ohm, T., Weber, W. & Claessen, R. Renormalization of Bulk Magnetic Electron States at High Binding Energies. Phys. Rev. Lett.. 102, 187204 (2009)
  * [28] Schäfer, J., Schrupp, D., Rotenberg, E., Rossnagel, K., Koh, H., Blaha, P. & Claessen, R. Electronic Quasiparticle Renormalization on the Spin Wave Energy Scale. Phys. Rev. Lett.. 92, 097205 (2004)
  * [29] Savrasov, S. & Savrasov, D. Electron-phonon interactions and related physical properties of metals from linear-response theory. Phys. Rev. B. 54, 16487-16501 (1996)
  * [30] Bose, S. Electron–phonon coupling and spin fluctuations in 3d and 4d transition metals: implications for superconductivity and its pressure dependence. Journal Of Physics: Condensed Matter. 21, 025602 (2008)
  * [31] Pinski, F. & Butler, W. Calculated electron-phonon contributions to phonon linewidths and to the electronic mass enhancement in Pd. Phys. Rev. B. 19, 6010-6015 (1979)
  * [32] Moser, S., Moreschini, L., Jaćć, J., Barišć, O., Berger, H., Magrez, A., Chang, Y., Kim, K., Bostwick, A., Rotenberg, E., Forro, L. & Grioni, M. Tunable Polaronic Conduction in Anatase TiO2. Phys. Rev. Lett.. 110, 196403 (2013)
  * [33] Lee, J., Schmitt, F., Moore, R., Johnston, S., Cui, Y., Li, W., Yi, M., Liu, Z., Hashimoto, M., Zhang, Y., Lu, D., Devereaux, T., Lee, D. & Shen, Z. Interfacial mode coupling as the origin of the enhancement of Tc in FeSe films on SrTiO3. Nature. 515, 245-248 (2014)
  * [34] Chen, C., Avila, J., Frantzeskakis, E., Levy, A. & Asensio, M. Observation of a two-dimensional liquid of Fröhlich polarons at the bare SrTiO3 surface. Nature Communications. 6, 8585 (2015)
  * [35] Wang, Z., McKeown Walker, S., Tamai, A., Wang, Y., Ristic, Z., Bruno, F., Torre, A., Riccò, S., Plumb, N., Shi, M., Hlawenka, P., Sánchez-Barriga, J., Varykhalov, A., Kim, T., Hoesch, M., King, P., Meevasana, W., Diebold, U., Mesot, J., Moritz, B., Devereaux, T., Radovic, M. & Baumberger, F. Tailoring the nature and strength of electron–phonon interactions in the SrTiO3(001) 2D electron liquid. Nature Materials. 15, 835-839 (2016)
  * [36] Cancellieri, C., Mishchenko, A., Aschauer, U., Filippetti, A., Faber, C., Barišić, O., Rogalev, V., Schmitt, T., Nagaosa, N. & Strocov, V. Polaronic metal state at the LaAlO3/SrTiO3 interface. Nature Communications. 7, 10386 (2016)
  * [37] Riley, J., Caruso, F., Verdi, C., Duffy, L., Watson, M., Bawden, L., Volckaert, K., Laan, G., Hesjedal, T., Hoesch, M., Giustino, F. & King, P. Crossover from lattice to plasmonic polarons of a spin-polarised electron gas in ferromagnetic EuO. Nature Communications. 9, 2305 (2018)
  * [38] Caruso, F., Amsalem, P., Ma, J., Aljarb, A., Schultz, T., Zacharias, M., Tung, V., Koch, N. & Draxl, C. Two-dimensional plasmonic polarons in n-doped monolayer MoS2. Phys. Rev. B. 103, 205152 (2021)
  * [39] Li, F. & Sawatzky, G. Electron Phonon Coupling versus Photoelectron Energy Loss at the Origin of Replica Bands in Photoemission of FeSe on SrTiO3. Phys. Rev. Lett.. 120, 237001 (2018)
  * [40] Harada, T., Fujiwara, K. & Tsukazaki, A. Highly conductive PdCoO2 ultrathin films for transparent electrodes. APL Materials. 6, 046107 (2018)
  * [41] Ok, J., Brahlek, M., Choi, W., Roccapriore, K., Chisholm, M., Kim, S., Sohn, C., Skoropata, E., Yoon, S., Kim, J., Lee, H. Pulsed-laser epitaxy of metallic delafossite PdCrO2 films. APL Materials. 8, 051104 (2020)
  * [42] Sun, J., Barone, M., Chang, C., Holtz, M., Paik, H., Schubert, J., Muller, D., Schlom, D. Growth of PdCoO2 by ozone-assisted molecular-beam epitaxy. APL Materials. 7, 121112 (2019)
  * [43] Brahlek, M., Rimal, G., Ok, J., Mukherjee, D., Mazza, A., Lu, Q., Lee, H., Ward, T., Unocic, R., Eres, G. & Oh, S. Growth of metallic delafossite PdCoO2 by molecular beam epitaxy. Phys. Rev. Materials. 3, 093401 (2019)
  * [44] Tanaka , M., Hasegawa , M. & Takei , H. Growth and Anisotropic Physical Properties of PdCoO 2 Single Crystals. Journal Of The Physical Society Of Japan. 65, 3973-3977 (1996)
  * [45] White, S., Singh, U. & Wahl, P. A stiff scanning tunneling microscopy head for measurement at low temperatures and in high magnetic fields. Review Of Scientific Instruments. 82, 113708 (2011)
  * [46] Mazzola, F., Wells, J., Yakimova, R., Ulstrup, S., Miwa, J., Balog, R., Bianchi, M., Leandersson, M., Adell, J., Hofmann, P. & Balasubramanian, T. Kinks in the $\sigma$ Band of Graphene Induced by Electron-Phonon Coupling. Phys. Rev. Lett.. 111, 216806 (2013)
  * [47] Mazzola, F., Frederiksen, T., Balasubramanian, T., Hofmann, P., Hellsing, B. & Wells, J. Strong electron-phonon coupling in the $\sigma$ band of graphene. Phys. Rev. B. 95, 075430 (2017)
  * [48] Fernández-Torres, L., Sykes, E., Nanayakkara, S. & Weiss, P. Dynamics and Spectroscopy of Hydrogen Atoms on Pd 111. The Journal Of Physical Chemistry B. 110, 7380-7384 (2006)

Figure S1: Self-energy analysis. (a) Electronic structure measurements at $90$
eV photon energy showing the $\gamma$ and $\delta$ bands, reproduced from Fig.
2 of the main text. (b) Full real and imaginary parts of the self energy
extracted from the data. The three-mode model (red line) is also shown,
including a magnon at $245$ meV and two phonons closer to the Fermi level,
i.e. at $130$ meV and $50$ meV as described in the text. Electron-electron and
electron-impurity scattering has also been included. The former manifests as
an increase in the tail of both the real and imaginary parts of the self
energy, as indicated in (b) by the green line. The real and imaginary parts
retain causality through Kramers-Kronig transformations. Figure S2: Modelling
including only phonon modes. Real and imaginary parts of the electron-boson
self energy extracted from the data (gray markers) along with simulated
counterparts with inclusion of only 2 phonon modes, as described in the main
text. The electron-phonon coupling strengths have been varied to attempt to
obtain the best match to the experimental results, however even with arbitrary
variation of the coupling strengths, it is not possible to simultaneously
satisfactorily describe the real and imaginary parts of the self energy and
their energy dependence. Similar lack of agreement was found with further
variation of the mode energies, up to a maximum of $150$ meV which corresponds
to the extreme maximum of the acoustic branch. Thus we conclude that with only
phonon-based models, we cannot describe our experimental observations. Figure
S3: Curvature analysis of ARPES data. (a) ARPES measurements reproduced from
Fig. 3(a) of the main text and (b) corresponding curvature analysis222Zhang,
P. et al., A precise method for visualizing dispersive features in image
plots. Rev. Sci. Instrum. 82, 043712 (2011). of the measured dispersions. The
trends visible in the curvature data support the conclusions from the
quantitative analysis presented in the main text, including: the multi-kink
structure of sample S1; the increase in coupling strength from samples S1 to
S4 leading to a pronounced decrease in Fermi velocity; the disappearance of
the $\delta$-pocket with doping; and the emergence of a replica band feature
split off from the main quasiparticle peak by 230 meV in sample S4.  Figure
S4: Sample-dependent doping variations. Energy distribution curves from
samples S1-S4 shown in the main text (Fig. 3), taken at a momentum $-1.35\pm
0.015$Å-1, cutting through the flat portion of the $\alpha$ band. A clear
shift in the onset of this band towards the Fermi level is observed (indicated
by the arrows), consistent with the p-type doping effect discussed in the main
text. An increased linewidth is also visible, consistent with an increase in
electron-impurity scattering rate due to surface disorder. Figure S5: Tight
Binding Analysis. (a,b) A simple tight-binding parametrization of the Pd-
derived surface states which cross the Fermi level, shown superimposed on the
ARPES data from sample S1 of the main text. Blue and red colors indicate the
majority and minority spin components, respectively. Our tight-binding model
agrees well with the ARPES data both for (a) energy-momentum dispersion and
(b) the surface Fermi surface. (c) Corresponding change in surface carrier
density for these tight-binding bands as a function of a shift of the chemical
potential. Shifts of the chemical potential on the order of $40$ meV, as
observed experimentally between samples S1 to S4, correspond to doping changes
on the order of $0.02$ electrons/unit cell, which would arise from a change in
surface Pd vacancy concentration of 2% between our measured samples; an
entirely reasonable value to expect from cleave-to-cleave variations. We note
that this analysis neglects the changes in quasi-particle velocity due to the
varying many-body interactions which we observe, as well as any potential
feedback of the doping on the resulting exchange splitting of the electronic
states in the system. However, this should make only small changes in the
calculated doping vs. band energy shift, which yields values consistent with
direct estimation of carrier densities from the measured ARPES data. Figure
S6: Doping-dependent self-energy variations. Self energy extraction for (a) S2
and (b) S3 samples of the main text. The self energy components have been
fitted in two ways: The first, indicated by the red model, is obtained by
allowing the strength of all the electron-boson couplings to vary. This model,
regardless of the samples, gives a very good description of the experimental
data and a negligible variation of the electron-phonon coupling strength. The
second fit (blue lines) is obtained by fixing the magnon strength to match the
one extracted for S1 and allowing the electron-phonon coupling strength to
vary. This model, irrespective of the strength of the electron-phonon coupling
does not show a satisfactory model to explain the experimental points. We thus
conclude that the main effects underpinning the changes observed
experimentally are due to changes in the electron-magnon coupling strength, as
described in the main text.  Figure S7: Differential conductance. Point
differential conductance $dI/dV$ spectrum recorded from a defect-free position
on the Pd-terminated surface, measured using the following parameters:
spectroscopy set-point ($V_{SP}$, $I_{SP}$): 850 mV, 520 pA; amplitude and
frequency of bias modulation: 6 mV, 413 Hz.


# STREAMING TARGET SOUND EXTRACTION USING ATTENTION QUERIES

# Causal TARGET SOUND Selection USING ATTENTION QUERIES

# Transformers for causal label-based source extraction

# Real-time TARGET SOUND EXTRACTION USING ATTENTION QUERIES

# Causal Target source extraction using a transformer decoder

# Real-time label-based sound extraction

# Real-time TARGET SOUND EXTRACTION USING Transformers

# Streaming TARGET SOUND EXTRACTION using attention

# Real-time TARGET SOUND EXTRACTION using attention

# Real-time TARGET SOUND EXTRACTION

# Waveformer: Transformer for Real-time TARGET SOUND EXTRACTION

# Real-time TARGET SOUND EXTRACTION

# Real-time TARGET SOUND EXTRACTION

###### Abstract

We present the first neural network model to achieve real-time and streaming
target sound extraction. To accomplish this, we propose Waveformer, an
encoder-decoder architecture with a stack of dilated causal convolution layers
as the encoder, and a transformer decoder layer as the decoder. This hybrid
architecture uses dilated causal convolutions for processing large receptive
fields in a computationally efficient manner, while also leveraging the
generalization performance of transformer-based architectures. Our evaluations
show as much as 2.2–3.3 dB improvement in SI-SNRi compared to the prior models
for this task while having a 1.2–4x smaller model size and a 1.5–2x lower
runtime. We provide code, dataset, and audio samples:
https://waveformer.cs.washington.edu/.

Index Terms—  Sound selection, streaming, attention

##  1 Introduction

Humans are exceptionally adept at attending their auditory focus to specific
sounds even in a noisy environment [1]. Recent works that aim to create a
computational equivalent of this human capability formulate this problem as
target sound extraction [1, 2, 3]. The goal is to extract sound signals of
interest from a mixture of various overlapping sounds, given clues that
provide information about the target sound class such as embeddings of a one-
hot label [1], audio clips [3, 4], and images [5, 6]. Streaming target sound
extraction could enable real-time intelligent acoustic applications for
headphones, hearing aids, and telephony by filtering out undesired sounds from
the environment (e.g., traffic) and presenting only sounds of interest to the
user (e.g., sirens).

Recent works on target sound extraction have shown promising performance even
for mixtures containing a large number of sound classes [1]. However, none of
these prior works demonstrate real-time streaming capabilities. In particular,
the prior works for this task are based on non-streaming models and designed
for offline processing, where the neural network has access to a large block
($\geq 1$ s) of audio samples [1]. In contrast, real-time streaming
applications impose significant algorithmic and computational constraints,
requiring networks to operate on small blocks ($\leq 10$ ms) with a limited
number of lookahead samples for each block. All these factors can
significantly degrade the performance [7].

In this paper, we present the first deep learning method to perform target
sound extraction in a streaming manner. Fig. 1 shows Waveformer, our encoder-
decoder architecture where the encoder is a stack of dilated causal
convolution (DCC) layers and the decoder is a transformer decoder [8].111We
call our network, Waveformer, since it uses a hybrid architecture with the
causal convolution layers, common in WaveNet [9] based architectures, as the
encoder and a transformer as the decoder. Our intuition is that much of the
complexity in prior models comes with processing large receptive fields,
especially at high sampling rates. For example, recent transformer-based
architectures proposed for speech separation [10, 11] implement chunk-based
processing, where each chunk independently attends to all the chunks in the
receptive field. Thus, to achieve a receptive field of length $R$, for each
chunk, these models have an $\mathcal{O}(R)$ computational complexity.
Instead, since DCC layers have a complexity of $\mathcal{O}(\log R)$ for
achieving the same amount of receptive field (§3.1), we use a stack of DCC
layers as the encoder that processes the receptive field. We then use the
decoder layer of the transformer architecture [8] as our model’s decoder. The
decoder generates a mask that can extract the specified target sound to
produce the output signal.

To evaluate our network architecture, we implement a causal version of Conv-
TasNet and a streaming version of ReSepformer [10] for the task of streaming
target sound extraction. Evaluations show that our hybrid network architecture
achieves state-of-the-art performance for this task. Further, the smallest and
largest versions of our model have real-time factors (RTFs) of 0.66 and 0.94,
respectively, on a consumer-grade CPU, demonstrating the real-time target
sound extraction capability, thus outperforming prior models in terms of both
efficiency and signal quality.

(a) End-to-end architecture

(b) DCC Encoder

(c) Transformer Decoder

Fig. 1: Waveformer architecture. Streaming inference demonstrated using an
example input mixture segment of length $4L$ samples, corresponding to a chunk
length $K=4$. The query is a one-hot or a multi-hot label encoding. The
Dilated Causal Convolution (DCC) encoder encodes the input chunk $y_{k}$ using
the context computed from the receptive field. The transformer decoder
computes the target mask by attending to current and previous encoded chunks.

##  2 Related work

Universal sound separation. The task here is to decompose a mixture of
arbitrary sound types into their component sounds, regardless of the number of
sounds in the mixture [12]. This becomes increasingly challenging as the
number of possible sound types in the mixture increases. Several networks have
been proposed for this task including convolutional long short-term memory
networks [12], time-dilated convolution networks [12] based on Conv-TasNet
[7], and transformer networks [13]. Prior work also proposed the use of
embeddings learned by a sound classifier trained on a large sound ontology
[14] for conditioning a separation network.

Target sound extraction. This approach can circumvent the challenge of
universal sound separation struggling to deal with a mixture of a large number
of sounds. The clues may be provided as an embedding of an audio clip [3, 4],
an image [5, 6], natural language text [15, 16], onomatopoeic words [17], or a
one-hot sound label vector [1]. The prior works [18, 19] have also evaluated
the use of a sound event detector to detect the time when the target sound
occurs in a mixture. Although these works are often motivated for practical
usage [16, 1], none of them use streaming models. In contrast to these prior
works, we design the first streaming network for target sound extraction using
attention.

Speech-specific networks. Prior work has also focused on speech enhancement
[20, 21, 22], speech separation [7, 23, 24, 25, 26, 27, 28] and speech
selection using clues provided to the network [29, 30, 31, 32]. For speech
enhancement, neural networks have been proposed [21, 33] to realize real-time
operation. Recently, efficient transformer-based architectures have been
proposed for speech recognition and separation tasks [34, 11, 35]. These
methods either use standard transformer blocks [10] or convolution-augmented
transformer blocks [11, 35]. In contrast, we use DCC layers along with the
transformer decoder. Recent work on ReSepformer [10] proposed a causal mode
for their transformer method, which we use for our baseline comparisons.

##  3 Waveformer Architecture

We process individual audio chunks of duration $\tau$ seconds. For streaming,
we need to operate at the chunk level: an output chunk can depend on the
current and past chunks. Thus, streaming models have an intrinsic latency
equal to the duration of a single chunk. In real-time practical systems, it is
desirable that this latency is on the order of 10 ms [36]. Fig. 1 shows our
proposed time-domain model architecture, which employs an encoder-decoder
222Our use of _encoder_ and _decoder_ is in the context of mask generation. In
contrast, Conv-TasNet [7] uses those terms for input Conv1d and output
ConvTranspose1D, respectively.based mask generation network to generate an
element-wise multiplicative mask in the latent space. Let $x_{k}\in R^{S}$
denote the current input audio chunk, where $S=\tau F_{s}$ is the number of
audio samples included in the current chunk with a sampling rate of $F_{s}$.
In the first step, a 1D-convolution layer with stride $L$ and kernel size $3L$
is applied to the input audio chunk $x_{k}$ to obtain the latent space
representation, $y_{k}\in R^{E\times K}$, where $E$ is the latent space
feature dimensions and $K=\frac{S}{L}$ is the feature sequence length in the
latent space. Setting the kernel size to $3L$ and stride to $L$ requires an
overlap of $L$ samples with the previous and the future chunk, resulting in a
lookahead of $2L$ samples. In our experiments, we set $L=32$ samples at 44.1
kHz. This results in a lookahead of around $1.45$ ms, which is negligible.
Given a one-hot or multi-hot query vector $q\in\\{0,1\\}^{N_{c}}$, where
$N_{c}$ is the total number of classes, streaming target sound extraction is
achieved by computing feature masks $m_{k}\in R^{E\times K}$. With the mask
generation network and element-wise multiplication denoted as $\mathcal{M}$
and $\odot$, respectively, the target sound signal, $\hat{x}_{k}\in R^{S}$, is
computed as:

$\displaystyle y_{k}=$
$\displaystyle\text{Conv1d}(x_{k}),~{}~{}~{}m_{k}=\mathcal{M}(y_{k}\ |\
y_{k-1},..,y_{2},y_{1},q)$ $\displaystyle\hat{x}_{k}=$
$\displaystyle\text{ConvTranspose1d}(y_{k}\odot m_{k}).$

###  3.1 Dilated causal convolution encoder

Our encoder is a stack of dilated causal convolution (DCC) layers [9], and the
decoder is a transformer network [8]. The motivation for such an architecture
is that the encoder computes a _contextful_ representation of the input chunk,
considering the previous chunks up to a certain receptive field, and the
decoder conditions the encoder output with the query vector to estimate the
target mask. While recent transformer models for speech separation [34, 10]
have demonstrated performance gains over convolution-based methods [7], the
latter have generally been more computationally efficient.

We attribute this efficiency gap to the difference in the way existing
transformer models process the receptive field compared to convolution-based
architectures. To achieve a receptive field of length $R$, given the chunk-
based processing in existing transformer architectures, each chunk
individually attends to all previous chunks in the receptive field resulting
in $\mathcal{O}(R)$ complexity. In contrast, convolution based models [9, 7]
using a stack of $M$ DCC layers with kernel size $P$ and exponentially scaling
dilation factors $\\{2^{0},2^{1},2^{2},...,2^{M-1}\\}$ have a receptive field
of $(P-1)\cdot(2^{M}-1)$. Its complexity is $\mathcal{O}(PM)$. With a small
kernel size $P$, the computational complexity of the stacked DCC layers is
$\mathcal{O}(P\cdot\log~{}(1+\frac{R}{P-1}))$ $\sim$ $\mathcal{O}(\log R)$ for
it to have a receptive field of length $R$.

We use 10 DCC layers with a kernel size of 3 and dilation factors
$\\{2^{0},2^{1},2^{2},...,2^{10-1}\\}$ in our encoder, resulting in a
receptive field of $(3-1)\cdot(2^{10}-1)=2046$ samples in the latent space.
With the initial input convolution stride $L$ set to 0.73 ms, our encoder’s
receptive field is $\approx 1.5s$. Fig. 1 (b) shows an encoding of an input
chunk of length 4. For chunk-based streaming inference, the encoder maintains
a context buffer for each DCC layer. This context is initially computed from
the $1s$ receptive field and then updated dynamically after encoding each
subsequent chunk. For encoding a chunk, each DCC layer is fed with the output
chunk of the previous layer, left padded with the context of length twice the
layer’s dilation. After encoding a chunk, the context is updated with the
rightmost elements of the padded input for it to be used in encoding the next
chunk. For each input chunk $y_{k}$, the DCC encoder computes an encoded
representation, $e_{k}\in R^{E\times K}$.

###  3.2 Query-conditioned transformer decoder

To get the mask, the encoded representation computed above must be conditioned
with the query, $q$. To this end, we first compute an embedding, $l\in
R^{E\times 1}$, corresponding to $q$. This is achieved by using an embedding
layer comprising three 512-dimensional feed-forward sub-layers with an
$N_{C}$-dimensional input and an $E$-dimensional output. Our transformer
decoder conditions the encoded chunk $e_{k}$ with the query embedding $l$ and
derives the mask as follows.

Fig. 1 (c) shows our decoder architecture. First, we perform multiplicative
query integration [1, 3] to compute the conditioned representation:
${e_{k}}^{\prime}=e_{k}\odot l$. Since transformers are more computationally
expensive with higher dimensionality, we first project the encoded
representations, ${e_{k}}^{\prime}$ and $e_{k}$, to the decoder dimensions
$D\leq E$ with $1\times 1$ convolution. This results in projected encoded
representations, ${pe}_{k},{{pe}_{k}}^{\prime}\in R^{D\times K}$. The decoded
representations are then computed by passing ${{pe}_{k},{pe}_{k}}^{\prime}$ to
the transformer decoder layer’s self-attention and cross-attention blocks,
respectively, to obtain target mask ${pm}_{k}\in R^{D\times K}$ in the
projected decoder space. It is then projected back to the encoder dimensions
with another $1\times 1$ convolution layer to obtain ${m_{k}}^{\prime}\in
R^{E\times K}$. Since the bottleneck caused by the projection layers might
affect the flow of gradients, as depicted in the diagram, we use a skip
connection immediately after the multiplicative query integration to the
output of the projection layer to compute the final mask:
$m_{k}={m_{k}}^{\prime}+{e_{k}}^{\prime}$.

Within the decoder, we use the chunk-based streaming attention scheme proposed
in [37]. As shown in Fig. 1(c), for decoding the current chunk, $e_{k}$, the
transformer decoder only attends to the samples in the current chunk, $e_{k}$,
and one previous chunk, $e_{k-1}$. This ensures that the input length to the
transformer decoder is fixed at $2K$ (current chunk + one previous chunk) and
prevents the inference time from growing as the input audio length increases.

Table 1: Performance on the single-target extraction task. In our model, $E$ and $D$ correspond to encoder and decoder dimensionalities, respectively. RTF is the real-time factor for a consumer CPU. Model | Model size | RTF | SI-SNRi
---|---|---|---
Conv-TasNet | 4.57M | 1.34 | 6.14
ReSepformer | 13.24M | 1.60 | 7.26
Ours ($E=256;~{}D=128$) | 1.10M | 0.66 | 9.02
Ours ($E=256;~{}D=256$) | 1.69M | 0.75 | 9.40
Ours ($E=512;~{}D=128$) | 3.29M | 0.88 | 9.26
Ours ($E=512;~{}D=256$) | 3.88M | 0.94 | 9.43
Table 2: SI-SNRi comparison in the multi-target extraction task. | # selected classes |
---|---|---
Model | 1 | 2 | 3 | Mean
Conv-TasNet | 7.20 | 3.63 | 0.19 | 3.67
ReSepformer | 7.42 | 3.56 | 0.33 | 3.77
Ours ($E=256;~{}D=128$) | 9.06 | 4.78 | 1.51 | 5.11
Ours ($E=256;~{}D=256$) | 9.12 | 4.76 | 1.31 | 5.06
Ours ($E=512;~{}D=128$) | 9.39 | 4.92 | 1.39 | 5.23
Ours ($E=512;~{}D=256$) | 9.29 | 4.92 | 1.35 | 5.19

##  4 Experiments and results

Dataset. We use a synthetic sound mixture dataset created from the FSD Kaggle
2018 dataset [38]. FSD Kaggle 2018 is a set of sound event and class label
pairs, with 41 different sound classes, which are a subset of the Audioset
ontology [14]. Our synthetic dataset consists of 50k training samples, 5k
validation samples, and 10k test samples. Sound mixtures are created using the
Scaper toolkit [39] with FSD Kaggle 2018 and TAU Urban Acoustic Scenes 2019
[40] as foreground and background sources, respectively. Foreground sound
classes are randomly sampled without replacement so that each sample has 3-5
unique classes. We construct the sound mixtures by sampling 3-5s crops from
each foreground sound and then pasting them on a 6s background sound. The SNRs
of the foreground sounds are randomly chosen between 15 and 25 dB, relative to
the background sound. Our training and validation data are sampled from the
development splits of FSD Kaggle 2018 and TAU Urban Acoustic Scenes 2019,
while our test samples are from the test splits. From each mixture, up to 3
foreground sounds are randomly selected as targets. During training, the
choices of the target foreground sounds in the training set are randomized.
Since we mainly consider human listening applications for streaming target
sound extraction, we run our experiments at a 44.1 kHz sampling rate to cover
the full audible range.

(a) Input mixture

(b) Ground-truth for label ‘Bark’

(c) Ground-truth for label ‘Cough’

(d) Output of the single-target extraction of ‘Bark’

(e) Output of the multi-target extraction of ‘Bark’ and ‘Cough’

Fig. 2: Visualization of time-domain waveforms of single-target and multi-
target extraction (x-axis represents time).

Evaluation setup. Prior works [1, 3] show that Conv-TasNet, originally
proposed for speech separation, can also be used for target sound extraction.
Further, ReSepformer proposes an efficient transformer architecture for speech
separation that allows a streaming inference. Here, we compare the performance
of our architecture with the causal or streaming implementations of Conv-
TasNet and ReSepformer as described in the original papers [7, 10] for the
target sound extraction task.

For all the models, we set the stride of the initial convolution, $L$, to 32,
which is about $0.73$ ms at 44.1 kHz. We train multiple configurations of our
model with different encoder and decoder dimensions. We fix the number of DCC
layers to 10, the number of transformer layers to 1, and the chunk length,
$K$, to 13. This chunk length corresponds to 416 samples in the time domain or
a chunk duration of 9.43 ms. For Conv-TasNet, we follow the configuration used
in [1] except for the number of repeats, which we set to 2. This ensures that
the runtime of the Conv-TasNet baseline is not too large compared with that of
our model’s largest configuration. For the ReSepformer baseline, we set the
model dimensionality to 512, the number of blocks to 2, the number of
transformer layers to 2, and the chunk size to 13 (9.43 ms). We perform label
integration after the first transformer block, as we found it to perform
better than integrating it at the beginning.

Loss function and training hyper-parameters. We use a linear combination of
90% signal-to-noise-ratio (SNR) and 10% scale-invariant-signal-to-noise-ratio
(SI-SNR) [41] as the loss function for training. We set the initial learning
rate to 5e-4 for our models and Conv-TasNet, and to 1.5e-4 for ReSepformer. We
train the models for 100 epochs and choose the model after the epoch that
resulted in the best validation SI-SNRi.

Results. We separately train the models for single-target and multi-target
extraction tasks and evaluate them on our testset. For multi-target
evaluation, we train our model as well as baselines to make predictions with
multi-hot query vectors, as opposed to one-hot queries used in the single-
target evaluation. During the multi-target training, 1-3 foreground sounds are
randomly selected as target sounds. This training method using an arbitrary
number of target sources helps the model learn multi-target embeddings. The
same model configurations are used for both the single-target and multi-target
experiments. The Conv-TasNet and ReSepfromer baselines are also trained in the
same way for the multi-target extraction task.

We also evaluate the real-time factors (RTFs) of the models on an Intel Core
i5 CPU using a single thread. RTF is computed by measuring the runtime
consumed by the models to process a 416 sample audio chunk (9.43 ms at 44.1
kHz), and dividing that by the chunk duration, 9.43 ms. For the RTF
measurement, we include the padding for dilated convolution layers in our
model’s DCC encoder and Conv-TasNet’s Temporal Convolution Network (TCN)
blocks, accounting for the entire receptive field. In the case of the
ReSepformer, using a single chunk for RTF measurement excludes the overhead
caused by causal attention masking in its inter-attention blocks.
Consequently, the RTF value reported for ReSepformer is a lower bound of what
is practically achievable.

Table 1 compares our models with different configurations with the baselines
in terms of both efficiency and performance. We show that our approach results
in 2.2-3.3 dB SI-SNRi improvement compared with the baselines while being
1.5-2x more computationally efficient with 1.2-4x fewer parameters. Table 2
compares the performance of our models with the baselines for the multiple
target extraction task. It shows that our method outperforms the baselines by
1.2-1.4 dB for the 2-target case and 1-1.2 dB for the 3-target case. As with
prior work [1], the SI-SNR improvements are lower in the 3-target selection
task since there is greater similarity between the input mixture and the
target signal, compared to the single-target case, resulting in a larger input
SI-SNR. We obtained $p$-values $<0.05$ for all comparisons except for the
comparison between $(E=512;D=256)$ and $(E=256;D=256)$, for which the
$p$-value was $0.57$.

In Fig. 2, we qualitatively show an example of single-target extraction and
multi-target extraction from a 4-class input mixture, using our multi-target
extraction model. Fig. 2(a) shows the input mixture waveform, and Figs. 2(b)
and 2(c) show the isolated ground-truth sounds. We provide the input mixture
to our multi-target model, with a single-target query followed by a two-target
query. Figs. 2(d) and 2(e) are the output waveforms obtained when the single
target and the two targets are queried, respectively. The waveforms show that
the model successfully recognizes the queried events and extracts the relevant
sounds. It can also be observed that our model preserves the original
amplitudes of the sounds in the input mixture well.

We also implemented a non-causal version of the proposed Waveformer. The
dilated causal convolution (DCC) block was made non-causal by padding on both
sides of each DCC layer’s input sequence, while the causal version only padded
to the left. The non-causal transformer decoder attends to the previous and
next chunks, in addition to the current chunk. Following [1], we trained our
non-causal model with both one-hot/multi-hot based extraction and Permutation
Invariant Training + Oracle Selection (PIT + OS) objectives. Table 3 compares
the performance of the non-causal version of our model with non-causal
baselines. The performance of our causal model is only 1.1 dB less than the
non-causal version, in contrast, to the 3–5 dB observed in prior source
separation works [7, 10]. We achieve this resilience by using layer
normalization throughout our architecture (avoiding the gLN to cLN switch in
[7, 10]) and a small context length for the transformer decoder.

Table 3: Performance comparison with non-causal baselines. Model | SI-SNRi
---|---
Listen to What You Want [1] | 9.91
Ours ($E=512;~{}D=256$; Non-causal) | 10.50
Ours ($E=512;~{}D=256$; Non-causal; PIT+OS) | 11.31

##  5 Conclusions

We demonstrate the first deep learning method for real-time and streaming
target sound extraction. Future work includes the use of more constrained
computing platforms, larger datasets with more classes, and multiple
microphones. Our Waveformer architecture may be applicable to other acoustic
applications like source separation and directional hearing, which deserves
further exploration.

Acknowledgements.  The UW researchers are funded by the Moore Inventor Fellow
award #10617 and the National Science Foundation.

## References

  * [1] T. Ochiai, M. Delcroix, Y. Koizumi, H. Ito, K. Kinoshita, and S. Araki,  “Listen to What You Want: Neural Network-based Universal Sound Selector,”  arXiv e-prints, p. arXiv:2006.05712, 2020.
  * [2] M. Delcroix, J. B. Vázquez, T. Ochiai, K. Kinoshita, and S. Araki,  “Few-shot learning of new sound classes for target sound extraction,”  arXiv preprint arXiv:2106.07144, 2021.
  * [3] M. Delcroix, J. B. Vázquez, T. Ochiai, K. Kinoshita, Y. Ohishi, and S. Araki,  “Soundbeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning,”  arXiv preprint arXiv:2204.03895, 2022.
  * [4] B. Gfeller, D. Roblek, and M. Tagliasacchi,  “One-shot conditional audio filtering of arbitrary sounds,”  in ICASSP. IEEE, 2021.
  * [5] R. Gao and K. Grauman,  “Co-separating sounds of visual objects,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [6] X. Xu, B. Dai, and D. Lin,  “Recursive visual sound separation using minus-plus net,”  in IEEE/CVF ICCV, 2019.
  * [7] Y. Luo and N. Mesgarani,  “Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,”  IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 8, 2019.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention is all you need,” 2017.
  * [9] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu,  “Wavenet: A generative model for raw audio,” 2016.
  * [10] C. Subakan, M. Ravanelli, S. Cornell, F. Lepoutre, and F. Grondin,  “Resource-efficient separation transformer,”  arXiv preprint arXiv:2206.09507, 2022.
  * [11] J. Luo, J. Wang, N. Cheng, E. Xiao, X. Zhang, and J. Xiao,  “Tiny-sepformer: A tiny time-domain transformer network for speech separation,”  arXiv preprint arXiv:2206.13689, 2022.
  * [12] I. Kavalerov, S. Wisdom, H. Erdogan, B. Patton, K. Wilson, J. Le Roux, and J. R. Hershey,  “Universal sound separation,”  in 2019 WASPAA. IEEE, 2019.
  * [13] A. Zadeh, T. Ma, S. Poria, and L.-P. Morency,  “Wildmix dataset and spectro-temporal transformer model for monoaural audio source separation,”  arXiv preprint arXiv:1911.09783, 2019.
  * [14] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in ICASSP. IEEE, 2017.
  * [15] K. Kilgour, B. Gfeller, Q. Huang, A. Jansen, S. Wisdom, and M. Tagliasacchi,  “Text-driven separation of arbitrary sounds,”  arXiv preprint arXiv:2204.05738, 2022.
  * [16] X. Liu, H. Liu, Q. Kong, X. Mei, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang,  “Separate what you describe: Language-queried audio source separation,”  arXiv preprint arXiv:2203.15147, 2022.
  * [17] Y. Okamoto, S. Horiguchi, M. Yamamoto, K. Imoto, and Y. Kawaguchi,  “Environmental sound extraction using onomatopoeic words,”  in ICASSP. IEEE, 2022.
  * [18] Q. Kong, Y. Wang, X. Song, Y. Cao, W. Wang, and M. D. Plumbley,  “Source separation with weakly labelled data: An approach to computational auditory scene analysis,”  in ICASSP. IEEE, 2020.
  * [19] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov,  “Zero-shot audio source separation through query-based learning from weakly-labeled data,”  in AAAI, 2022, vol. 36.
  * [20] S. Wisdom, J. R. Hershey, K. Wilson, J. Thorpe, M. Chinen, B. Patton, and R. A. Saurous,  “Differentiable consistency constraints for improved deep speech enhancement,”  in ICASSP. IEEE, 2019.
  * [21] S. E. Eskimez, T. Yoshioka, H. Wang, X. Wang, Z. Chen, and X. Huang,  “Personalized speech enhancement: New models and comprehensive evaluation,”  in ICASSP. IEEE, 2022.
  * [22] I. Chatterjee, M. Kim, V. Jayaram, S. Gollakota, I. Kemelmacher, S. Patel, and S. M. Seitz,  “Clearbuds: wireless binaural earbuds for learning-based speech enhancement,”  in MobiSys, 2022.
  * [23] Y. Luo, Z. Chen, and T. Yoshioka,  “Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation,”  in ICASSP. IEEE, 2020.
  * [24] P.-S. Huang, M. Kim, M. Hasegawa-Johnson, and P. Smaragdis,  “Deep learning for monaural speech separation,”  in ICASSP. IEEE, 2014.
  * [25] J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe,  “Deep clustering: Discriminative embeddings for segmentation and separation,”  in ICASSP. IEEE, 2016.
  * [26] Y. Isik, J. L. Roux, Z. Chen, S. Watanabe, and J. R. Hershey,  “Single-channel multi-speaker separation using deep clustering,”  arXiv preprint arXiv:1607.02173, 2016.
  * [27] D. Yu, M. Kolbæk, Z.-H. Tan, and J. Jensen,  “Permutation invariant training of deep models for speaker-independent multi-talker speech separation,”  in ICASSP. IEEE, 2017.
  * [28] Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno,  “Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking,”  arXiv preprint arXiv:1810.04826, 2018.
  * [29] E. Tzinis, G. Wichern, A. Subramanian, P. Smaragdis, and J. L. Roux,  “Heterogeneous target speech separation,”  arXiv preprint arXiv:2204.03594, 2022.
  * [30] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein,  “Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation,”  arXiv preprint arXiv:1804.03619, 2018.
  * [31] A. Wang, M. Kim, H. Zhang, and S. Gollakota,  “Hybrid neural networks for on-device directional hearing,”  AAAI, vol. 36, no. 10, 2022.
  * [32] Z. Wang, R. Giri, S. Venkataramani, U. Isik, J.-M. Valin, P. Smaragdis, M. Goodwin, and A. Krishnaswamy,  “Semi-supervised time domain target speaker extraction with attention,”  arXiv preprint arXiv:2206.09072, 2022.
  * [33] R. Giri, S. Venkataramani, J.-M. Valin, U. Isik, and A. Krishnaswamy,  “Personalized percepnet: Real-time, low-complexity target voice separation and enhancement,”  arXiv preprint arXiv:2106.04129, 2021.
  * [34] C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong,  “Attention is all you need in speech separation,”  in ICASSP. IEEE, 2021.
  * [35] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang,  “Conformer: Convolution-augmented transformer for speech recognition,” 2020.
  * [36] M. Sunohara, C. Haruta, and N. Ono,  “Low-latency real-time blind source separation for hearing aids based on time-domain implementation of online independent vector analysis with truncation of non-causal components,”  in ICASSP, 2017.
  * [37] X. Chen, Y. Wu, Z. Wang, S. Liu, and J. Li,  “Developing real-time streaming transformer transducer for speech recognition on large-scale dataset,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 5904–5908.
  * [38] E. Fonseca, M. Plakal, F. Font, D. P. W. Ellis, X. Favory, J. Pons, and X. Serra,  “General-purpose tagging of freesound audio with audioset labels: Task description, dataset, and baseline,”  in DCASE2018, November 2018.
  * [39] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello,  “Scaper: A library for soundscape synthesis and augmentation,”  in WASPAA, 2017.
  * [40] A. Mesaros, T. Heittola, and T. Virtanen,  “A multi-device dataset for urban acoustic scene classification,”  in DCASE2018, November 2018.
  * [41] J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey,  “SDR - half-baked or well done?,” 2018.