# End-To-End Deep Learning-based Adaptation Control for Linear Acoustic Echo
Cancellation

Thomas Haubner, Andreas Brendel,
and Walter Kellermann  This article has been submitted to IEEE/ACM
Transactions on Audio, Speech, and Language Processing.
Thomas Haubner and Walter Kellermann are with the chair of Multimedia
Communications and Signal Processing (LMS), Friedrich-Alexander-Universität
Erlangen-Nürnberg, D-91058 Erlangen, Germany. Andreas Brendel contributed to
the work while he was at LMS. He is now with the Fraunhofer Institute for
Integrated Circuits (IIS), D-91058 Erlangen, Germany. e-mail:
<EMAIL_ADDRESS>
Digital Object Identifier

###### Abstract

The attenuation of acoustic loudspeaker echoes remains to be one of the open
challenges to achieve pleasant full-duplex hands free speech communication. In
many modern signal enhancement interfaces, this problem is addressed by a
linear acoustic echo canceler which subtracts a loudspeaker echo estimate from
the recorded microphone signal. To obtain precise echo estimates, the
parameters of the echo canceler, i.e., the filter coefficients, need to be
estimated quickly and precisely from the observed loudspeaker and microphone
signals. For this a sophisticated adaptation control is required to deal with
high-power double-talk and rapidly track time-varying acoustic environments
which are often faced with portable devices. In this paper, we address this
problem by end-to-end deep learning. In particular, we suggest to infer the
step-size for a least mean squares frequency-domain adaptive filter update by
a Deep Neural Network (DNN). Two different step-size inference approaches are
investigated. On the one hand broadband approaches, which use a single DNN to
jointly infer step-sizes for all frequency bands, and on the other hand
narrowband methods, which exploit individual DNNs per frequency band. The
discussion of benefits and disadvantages of both approaches leads to a novel
hybrid approach which shows improved echo cancellation while requiring only
small DNN architectures. Furthermore, we investigate the effect of different
loss functions, signal feature vectors, and DNN output layer architectures on
the echo cancellation performance from which we obtain valuable insights into
the general design and functionality of DNN-based adaptation control
algorithms.

###### Index Terms:

Acoustic echo cancellation, system identification, adaptation control, step-
size control, double-talk detection, DNN

##  I Introduction

The attenuation of loudspeaker echoes is a crucial component of any full-
duplex hands-free speech communication interface [1]. In general, there are
two main algorithmic approaches to achieve this goal: Acoustic Echo
Cancellation (AEC) and Acoustic Echo Suppression (AES) [1]. While echo
cancelers attenuate the unwanted signal components by subtracting a
loudspeaker echo estimate from the microphone signal, echo suppressors
directly filter the microphone signal with a postfilter [1]. In many modern
speech enhancement algorithms, both approaches are combined by applying a
postfilter to the echo-reduced error signal of the echo canceler [1, 2].
Despite recent progress by deep learning-based postfilters, the benefit of
additionally using an echo canceler is significant [3, 4, 5]. Yet, the full
potential of an echo canceler is only obtained, if its model parameters are
precisely inferred from the observed loudspeaker and microphone signals. For
this, the continuous minimization of the microphone error signal power by
gradient descent-based parameter updates has proven to be a powerful tool [6].
Yet, a sophisticated adaptation, i.e., step-size, control is required to
rapidly track time-varying acoustic scenes and deal with high-power double-
talk and noise [7].

Adaptation control has evolved in the last decades from simple binary stall-
or-adapt methods [8, 9], which mainly considered double-talk, to advanced
continuous step-size estimators which employ complex probabilistic models [10,
11, 12, 13, 14, 15, 16, 17]. In particular, the inference of the model
parameters by Kalman filtering [11, 12] or semi-supervised blind source
separation [15, 16, 17] can be considered as state of the art according to
recent challenge results [3, 4, 5]. Yet, the performance of these traditional
step-size estimators relies on the precise estimation of statistics of
unobserved signals, e.g., the near-end speech and noise Power Spectral
Densities (PSDs), which are difficult to obtain in practice [18]. Traditional
estimators of these quantities often suffer from sensitive hyperparameter
choices which do not generalize well to different acoustic scenes [18]. To
remedy this limitation, it has recently been proposed to use machine learning
models to support traditional step-size estimators [19, 20]. In particular,
the approximation of the interference PSD of the Kalman filter model,
capturing near-end speech and noise, by non-negative dictionaries [19] and
Deep Neural Networks (DNNs) [20] has shown significant performance
improvements relative to traditional, i.e., non-trainable, PSD estimators.
Besides the support of traditional step-size estimators by trainable PSD
models, machine learning has also been used to directly approximate optimum
step-sizes of a time-domain Normalized Least-Mean-Squares (NLMS) algorithm
[21] or Short-Time Fourier Transform (STFT)-domain recursive least squares
algorithm [22]. Yet, despite significant performance improvements, it remains
unclear whether machine learning-supported approximation of target step-sizes,
e.g., a Kalman filter step-size with oracle statistics, is optimum w.r.t. echo
cancellation performance. Furthermore, the loss function design for training
the machine learning models is challenging, as the effect of PSD or step-size
estimation errors on the AEC performance is complicated. Therefore, it has
recently been proposed to optimize DNN-based step-size estimators directly
w.r.t. the echo estimation quality which is typically termed end-to-end
training [23, 24, 25, 26, 27, 28, 29]. The various approaches differ in the
echo estimation model, feature design, DNN integration and loss function
design. While [23, 25, 26, 27] employ a time-domain Finite Impulse Response
(FIR) echo estimation model which is optimized in the frequency domain, [24,
26, 28, 29] consider a Convolutive Transfer Function (CTF) model (cf. e.g.
[30, 31, 32]) which operates in the STFT domain. Furthermore, while [23, 25,
26, 27] investigate linear echo estimation models, [24, 28, 29] consider
additionally non-linear pre-processors to deal with hardware imperfections,
e.g., loudspeaker distortions. Another key difference is given by the
integration of the DNN into the filter update. In [26, 27, 28], it is proposed
to directly estimate the gradient, or parts of it, by complex-valued DNNs. In
contrast, [23, 24, 25, 29] propose to infer only parameters of traditional
step-size estimators while keeping the gradient direction of the respective
filter update unchanged. Furthermore, various step-size inference
architectures have been considered ranging from broadband approaches, which
jointly estimate a frequency-selective step-size vector with a single DNN,
[23, 24, 25], to narrowband approaches, which use individual DNNs per
frequency band [26, 28, 29]. In addition, joint control of frequency groups
has been investigated [27]. Finally, there are also echo cancellation
algorithms which exploit no physical knowledge for designing the echo
estimation model but instead use a completely data-driven DNN-based estimator
[33, 34, 35].

In this paper, we provide a general view on deep learning-based adaptation
control for linear AEC and the differences between broadband and narrowband
step-size inference architectures. By investigating their benefits and
disadvantages, we introduce a new hybrid control strategy which combines the
low DNN parameter count of narrowband step-size estimators with the capability
of broadband approaches to exploit information from the entire frequency
spectrum. All three generic adaptation control approaches, i.e., narrowband,
broadband, and hybrid, are compared for a variety of challenging AEC
scenarios. Furthermore, we analyze the effect of different features, loss
functions and DNN output layer designs on the echo cancellation performance.
As a specific parameterization of our adaptation control algorithm is
equivalent to a basic DNN-based control of the NLMS algorithm, the conclusions
are broadly applicable. Additionally, to gain deeper insights into the DNN-
based step-size estimator, we analyze its internal states and step-size
estimates by comparing them to the spectrograms of the individual microphone
signal components. Finally, we provide the code for training the DNNs
111https://github.com/ThomasHaubner/e2e_dnn_ad_control_for_lin_aec to simplify
future comparison of different DNN-based adaptation control algorithms and
fuel the research on further developments.

The remainder of the paper is structured as follows: In Sec. II, we define a
signal model which describes a generic hands-free speech communication
interface. Subsequently, the proposed deep learning-based adaptation control
algorithms are introduced in Sec. III and their relation to traditional
approaches is discussed. An experimental evaluation follows in Sec. IV and
conclusions are presented in Sec. V.

In the following, we use $f$ and $\tau$ as frequency and frame index of STFT-
domain quantities, respectively. The corresponding time-domain signals are
denoted by an underline and $\kappa$ is used as the sample index. The real and
imaginary components of a complex-valued number $z\in\mathbb{C}$ are denoted
by $\mathcal{R}\\{z\\}$ and $\mathcal{I}\\{z\\}$, respectively, and its
complex conjugate by $z^{*}$. The Euclidean norm of a vector is denoted by
$\left|\left|\cdot\right|\right|$ and its transpose by $(\cdot)^{\text{T}}$.
Additionally, we introduce the all-one vector $\boldsymbol{1}_{F}$ of
dimension $F$ and the expectation operator $\mathbb{E}[\cdot]$. Finally, the
$l$th diagonal element of a matrix is written as $\left[\cdot\right]_{ll}$.

##  II Linear Acoustic Echo Cancellation

This section introduces the signal model and the filter update which are
commonly used for linear acoustic echo cancelers operating in the frequency
domain as shown in Fig. 1.

###  II-A Signal Model

${{u}}_{f,\tau}$${{y}}_{f,\tau}$$+$${{e}}_{f,\tau}$${{\boldsymbol{h}}}_{f,\tau}$$\hat{{{d}}}_{f,\tau}$$-$${{d}}_{f,\tau}$${{n}}_{f,\tau}$${{s}}_{f,\tau}$Noise
Figure 1: Block diagram of a linear acoustic echo canceler operating in the
frequency domain.

We model the STFT-domain microphone signal $y_{f,\tau}$ as a linear
superposition of an echo component $d_{f,\tau}$, a near-end speech component
$s_{f,\tau}$ and a background noise component $n_{f,\tau}$ as follows:

$y_{f,\tau}=d_{f,\tau}+s_{f,\tau}+n_{f,\tau}\in\mathbb{C}.$ (1)

The goal of AEC is to obtain a precise echo estimate $\hat{d}_{f,\tau}$ which
is subtracted from the microphone signal $y_{f,\tau}$ to obtain an ideally
echo-free error signal

$e_{f,\tau}=y_{f,\tau}-\hat{d}_{f,\tau}$ (2)

as shown in Fig. 1. In the following, we assume the popular CTF echo
estimation model [30, 31, 32] which suggests a linear FIR filter model in each
frequency band

$\hat{d}_{f,\tau}={\boldsymbol{h}}_{f,\tau-1}^{\text{T}}\boldsymbol{u}_{f,\tau}$
(3)

with the length-$L$ filter coefficient vector

$\boldsymbol{h}_{f,\tau-1}^{\text{T}}=\begin{pmatrix}{h}_{0,f,\tau-1}&\dots&{h}_{L-1,f,\tau-1}\end{pmatrix}$
(4)

and the STFT-domain loudspeaker signal vector

$\boldsymbol{u}_{f,\tau}^{\text{T}}=\begin{pmatrix}{u}_{f,\tau}&\dots&{u}_{f,\tau-L+1}\end{pmatrix}.$
(5)

###  II-B Adaptive Filter Estimation

The filter coefficient vector $\boldsymbol{h}_{f,\tau-1}$ should ideally be
updated such that the residual echo power
$\mathbb{E}[|d_{f,\tau}-\hat{d}_{f,\tau}|^{2}]$ is minimized. Yet, as the echo
statistics are not known in practice, filter coefficient updates are typically
derived by iteratively minimizing the microphone error signal power222Note
that the independent minimization of the frequency band-wise subband error
powers requires sufficient suppression of cross-band aliasing to approximate
the minimization of the time-domain error power.

${\psi}^{\text{EE}}_{f,\tau}=\mathbb{E}\left[\left|e_{f,\tau}\right|^{2}\right]=\mathbb{E}\left[\left|d_{f,\tau}-\hat{d}_{f,\tau}+z_{f,\tau}\right|^{2}\right]$
(6)

with $z_{f,\tau}=s_{f,\tau}+n_{f,\tau}$ denoting the interference component
w.r.t. the filter estimation task. Note that the interference $z_{f,\tau}$
includes also all echo components which cannot be described by the linear CTF
model (3), e.g., echoes resulting from late reverberation or loudspeaker non-
linearities. The most popular filter optimization method is given by the LMS
update [6]

${h}_{l,f,\tau}={h}_{l,f,\tau-1}+{\mu}_{l,f,\tau}\left({u}_{f,\tau-l}^{*}e_{f,\tau}\right)$
(7)

with ${\mu}_{l,f,\tau}\geq 0$ being a tap-, frequency- and time-dependent
step-size. Yet, by assuming the residual echo signal
$d_{f,\tau}-\hat{d}_{f,\tau}$ and the interference component $z_{f,\tau}$ to
be orthogonal, we easily conclude from (6) that the error signal power
$\mathbb{E}[|e_{f,\tau}|^{2}]$ is only a good approximation of the residual
echo power $\mathbb{E}[|d_{f,\tau}-\hat{d}_{f,\tau}|^{2}]$ for time-frequency
bins with a high Echo-to-Interference Power Ratio (EIR)

$\text{EIR}_{f,\tau}=10\log_{10}\frac{\mathbb{E}\left[\left|d_{f,\tau}\right|^{2}\right]}{\mathbb{E}\left[\left|z_{f,\tau}\right|^{2}\right]}.$
(8)

To remedy this limitation the step-size $\mu_{l,f,\tau}$ in the LMS update (7)
must be adapted to the respective signal statistics as will be discussed in
the following.

##  III Adaptation Control

In this section, we introduce and motivate the general concept of deep
learning-based step-size estimation and discuss different generic variants.

###  III-A Traditional Approaches

We first introduce traditional model-based methods which will serve as a basis
for the deep learning-based approaches described in the sequel. The majority
of these methods suggests a mapping of certain signal expectations, e.g., the
loudspeaker and interference powers
${\psi}^{\text{{UU}}}_{f,\tau}=\mathbb{E}[||\boldsymbol{u}_{f,\tau}||^{2}]$
and ${\psi}^{\text{ZZ}}_{f,\tau}=\mathbb{E}[|{z}_{f,\tau}|^{2}]$,
respectively, to the optimum step-size

$\mu_{l,f,\tau}\leftarrow
g^{\text{MB}}\left({\psi}^{\text{UU}}_{f,\tau},{\psi}^{\text{ZZ}}_{f,\tau},\dots\right)$
(9)

with $g^{\text{MB}}\left(\cdot\right)$ denoting a deterministic function.
Among the most widely used traditional step-size selection strategies are

a) the stall-or-adapt NLMS step-size, e.g., [8, 9],

$\mu_{l,f,\tau}\leftarrow\frac{m^{\text{DT}}_{\tau}}{{\psi}^{\text{UU}}_{f,\tau}}\qquad\forall
l\in\\{0,\dots,L-1\\}$ (10)

with the binary parameter $m^{\text{DT}}_{\tau}$ being controlled by a double-
talk detector,

b) the error power-aware (EA) NLMS step-size [36, 37]

$\mu_{l,f,\tau}\leftarrow\frac{m^{\text{EA-
NLMS}}}{{\psi}^{\text{UU}}_{f,\tau}+{\psi}^{\text{EE}}_{f,\tau}}\qquad\forall
l\in\\{0,\dots,L-1\\}$ (11)

with the static hyperparameter $m^{\text{EA-NLMS}}>0$,

c) the minimum system-distance NLMS step-size [7]

$\mu_{l,f,\tau}\leftarrow\frac{\mathbb{E}\left[\left|\left(\boldsymbol{h}_{f,\tau}-\boldsymbol{h}_{f,\tau}^{\text{MMSE}}\right)^{\text{T}}\boldsymbol{u}_{f,\tau}\right|^{2}\right]}{\mathbb{E}\left[\left|\left(\boldsymbol{h}_{f,\tau}-\boldsymbol{h}_{f,\tau}^{\text{MMSE}}\right)^{\text{T}}\boldsymbol{u}_{f,\tau}\right|^{2}\right]+\psi^{\text{ZZ}}_{f,\tau}}~{}\frac{1}{{\psi}^{\text{UU}}_{f,\tau}},$
(12)

for all $l\in\\{0,\dots,L-1\\}$ with the Minimum Mean Square Error (MMSE)
filter estimate $\boldsymbol{h}_{f,\tau}^{\text{MMSE}}$, and

d) the Kalman filter step-size [11, 12, 38]

$\mu_{l,f,\tau}\leftarrow\frac{{\Psi}^{\text{HH}}_{l,f,\tau}}{\sum_{l=0}^{L-1}{\Psi}^{\text{HH}}_{l,f,\tau}\left|{u}_{f,\tau-l}\right|^{2}+\psi^{\text{ZZ}}_{f,\tau}}$
(13)

with the filter estimation variances
${\Psi}^{\text{HH}}_{0,f,\tau},\dots,{\Psi}^{\text{HH}}_{L-1,f,\tau}$. Note
that except for the Kalman update (13), all considered adaptation control
methods assume equal step-sizes for updating the filter coefficients within a
time-frequency bin. Furthermore, while (10) represents a binary broadband
stall-or-adapt step-size, (12) and (13) yield continuous-valued frequency-
selective step-sizes. All approaches have in common that they rely on a
precise estimation of typically unknown expectations which often depend on
non-observable signals, e.g., the interference power
$\psi^{\text{ZZ}}_{f,\tau}$ or the MMSE filter estimate
$\boldsymbol{h}_{f,\tau}^{\text{MMSE}}$. Furthermore, the claimed optimality
of the respective step-sizes holds only under strong model assumptions, e.g.,
a linear Gaussian model for the Kalman filter update (13), whose validity is
often questionable in practice. To remedy these limitations, we resort now to
the concept of deep learning-based step-size estimation.

###  III-B Deep Learning-based Adaptation Control

The general idea of deep learning-based step-size estimation is to replace the
mapping $g^{\text{MB}}(\cdot)$ in (9) by a DNN [23, 25, 24]

$\boldsymbol{\mu}_{\tau}=\begin{pmatrix}\mu_{1,\tau}&\dots&\mu_{F,\tau}\end{pmatrix}^{\text{T}}\leftarrow
g^{\text{BB-
DNN}}\left(\boldsymbol{\chi}_{\tau},\boldsymbol{\zeta}_{\tau};\boldsymbol{\theta}^{\text{BB}}\right)$
(14)

with parameter vector $\boldsymbol{\theta}^{\text{BB}}$ which infers the
frame-dependent broadband step-size vector $\boldsymbol{\mu}_{\tau}$ from an
observable feature vector $\boldsymbol{\chi}_{\tau}$ and internal DNN state
vector $\boldsymbol{\zeta}_{\tau}$. Note that we assume in Eq. (14) equal
step-sizes within a time-frequency bin, i.e., $\forall
l\in\\{0,\dots,L-1\\}:\mu_{f,\tau}=\mu_{f,l,\tau}$, which is common to many
adaptation control approaches (cf. Eqs. (10) - (12)).

Spectro-Temporal Feature Maps$f$$\tau$$g^{\text{BB-
DNN}}(\cdot)$$\boldsymbol{\mu}_{\tau}$$\boldsymbol{\chi}_{1,\tau}$$\boldsymbol{\chi}_{2,\tau}$$\boldsymbol{\chi}_{F,\tau}$$\boldsymbol{\vdots}$
(a) Broadband

Spectro-Temporal Feature Maps$f$$\tau$$g^{\text{NB-
DNN}}(\cdot)$$\boldsymbol{\chi}_{f,\tau}$$\mu_{f,\tau}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$
(b) Narrowband

Spectro-Temporal Feature Maps$f$$\tau$$g^{\text{HB-
DNN}}(\cdot)$$\boldsymbol{\chi}_{f,\tau}$${\mu}_{f,\tau}$$g^{\text{BB-
FEAT}}(\cdot)$$\boldsymbol{\xi}_{\tau}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$$\boldsymbol{\vdots}$
(c) Hybrid

Figure 2: Narrowband, broadband and hybrid DNN-based adaptation control. While
in broadband control a single DNN is used to simultaneously infer a frequency-
selective broadband step-size vector $\boldsymbol{\mu}_{\tau}$, narrowband and
hybrid approaches use independent DNNs to estimate scalar step-sizes
$\mu_{f,\tau}$ per frequency band. While narrowband approaches exploit only
information from the respective frequency band, hybrid methods use an
additional low-dimensional feature vector $\boldsymbol{\xi}_{\tau}$ which is
representative of the entire spectrum. The dashed arrows represent the
dependency of the vector $\boldsymbol{\xi}_{\tau}$ on the observations of all
frequency bands.

We now describe three different generic adaptation control strategies which
differ in the structure of the DNN-based mapping $g^{\text{BB-DNN}}(\cdot)$.
For this we first introduce a decomposition of the broadband feature vector
$\boldsymbol{\chi}_{\tau}$ into $F$ narrowband feature vectors
$\boldsymbol{\chi}_{f,\tau}$ with $f=1,\dots,F$, which contain only
information about the respective frequency bands as follows:

$\boldsymbol{\chi}_{\tau}=\begin{pmatrix}\boldsymbol{\chi}_{1,\tau}^{\text{T}}&\dots&\boldsymbol{\chi}_{F,\tau}^{\text{T}}\end{pmatrix}^{\text{T}}.$
(15)

This forms the input to the following three DNN-based step-size inference
approaches:

a) Broadband DNN. The first and simplest method is to jointly infer all $F$
narrowband step-sizes $\mu_{1,\tau},\dots,\mu_{F,\tau}$, contained in the
broadband vector $\boldsymbol{\mu}_{\tau}$, by a single DNN which exploits
features from the entire frequency spectrum. This approach is
straightforwardly related to the mapping (14) and visualized in Fig. 2(a).

b) Narrowband DNN. In many traditional approaches (cf. Sec. III-A) the
narrowband step-size $\mu_{f,\tau}$ is computed only from statistics of the
respective frequency band, i.e., without exploiting any inter-frequency
dependencies. This motivates a decomposition of the general mapping
$g^{\text{BB-DNN}}(\cdot)$ in (14) into $F$ narrowband mappings [26, 28, 29]

$\mu_{f,\tau}\leftarrow g^{\text{NB-
DNN}}\left(\boldsymbol{\chi}_{f,\tau},\boldsymbol{\zeta}_{f,\tau};\boldsymbol{\theta}^{\text{NB}}\right)$
(16)

as shown in Fig. 2(b). Eq. (16) suggests an inference of the narrowband step-
size $\mu_{f,\tau}$ from a narrowband feature vector
$\boldsymbol{\chi}_{f,\tau}$ with $\boldsymbol{\zeta}_{f,\tau}$ and
${\boldsymbol{\theta}}^{\text{NB}}$ denoting the internal state and parameter
vectors of the DNN, respectively. Note that we assume the DNN parameter vector
${\boldsymbol{\theta}}^{\text{NB}}$ to be shared among different frequency
bands which significantly reduces the number of trainable DNN parameters. It
furthermore offers the possibility to use a single trained DNN for
applications with different frequency resolutions which increases the
flexibility.

c) Hybrid DNN. A possible disadvantage of the narrowband relative to the
broadband method is that it cannot exploit statistical inter-frequency
dependencies for adaptation control. To remedy this limitation, while still
benefiting from the low DNN parameter count of narrowband approaches, we
introduce a hybrid of both methods which is shown in Fig. 2(c). The key idea
is to employ an individual mapping per frequency band, as suggested in the
narrowband approach, yet, extend the narrowband feature vector
$\boldsymbol{\chi}_{f,\tau}$ by a small number of additional hybrid features
which are representative for the entire frequency spectrum. The respective
hybrid step-size mapping is given by

$\mu_{f,\tau}\leftarrow g^{\text{HB-
DNN}}\left(\begin{pmatrix}\boldsymbol{\chi}_{f,\tau}^{\text{T}}&\boldsymbol{\xi}_{\tau}^{\text{T}}\end{pmatrix}^{\text{T}},\boldsymbol{\zeta}_{f,\tau};{\boldsymbol{\theta}}^{\text{HB}}\right)$
(17)

with the narrowband feature vector $\boldsymbol{\chi}_{f,\tau}$, the hybrid
feature vector $\boldsymbol{\xi}_{\tau}$, the internal DNN state vector
$\boldsymbol{\zeta}_{f,\tau}$ and the parameter vector
$\boldsymbol{\theta}^{\text{HB}}$. In Fig. 2(c) the computation of the hybrid
feature vector $\boldsymbol{\xi}_{\tau}$ is indicated by the function
$g^{\text{BB-FEAT}}(\cdot)$ which does not exhibit any trainable parameters.
The benefit of the additional features critically depends on their design
which is discussed in Sec. III-C.

Finally, we observe that all three approaches suggest a frequency-selective
adaptation control scheme. An alternative is to enforce equivalent step-sizes
for all frequency bands which is common to many time-domain LMS-based filter
updates. This approach can be directly incorporated into the broadband method
(14) by assuming

$\boldsymbol{\mu}_{\tau}\leftarrow\mu_{\tau}\boldsymbol{1}_{F}$ (18)

with the scalar step-size $\mu_{\tau}$ being provided by the DNN.

###  III-C Feature Vectors

We will now discuss various choices for the narrowband and hybrid feature
vectors $\boldsymbol{\chi}_{f,\tau}$ and $\boldsymbol{\xi}_{\tau}$,
respectively, which are used to infer the step-sizes (cf. Eqs. (14) - (17)).
The feature vectors can be computed from any subset of the following
observable signals: The loudspeaker signal $u_{f,\tau}$, the microphone signal
$y_{f,\tau}$, the error signal $e_{f,\tau}$ and the echo estimate
$\hat{d}_{f,\tau}$. As we consider in this paper only real-valued DNNs, each
complex-valued signal must be transformed to the field of real numbers. A
straightforward mapping is given by stacking the real and imaginary parts of a
subset of the complex-valued signals, e.g., the loudspeaker and microphone
signals $u_{f,\tau}$ and $y_{f,\tau}$, respectively,

$\boldsymbol{\chi}_{f,\tau}^{\text{T}}\leftarrow\begin{pmatrix}\mathcal{R}\\{u_{f,\tau}\\}&\mathcal{I}\\{u_{f,\tau}\\}&\mathcal{R}\\{y_{f,\tau}\\}&\mathcal{I}\\{y_{f,\tau}\\}\end{pmatrix}.$
(19)

Many traditional adaptation control approaches (cf. Sec. III-A) mainly rely on
the signal magnitude, i.e., discard the phase information. This motivates a
non-linear mapping by the magnitude operator and an additional logarithmic
transformation with exemplary narrowband feature vectors being given by

$\displaystyle\boldsymbol{\chi}_{f,\tau}^{\text{T}}$
$\displaystyle\leftarrow\begin{pmatrix}\left|u_{f,\tau}\right|&\left|y_{f,\tau}\right|\end{pmatrix},$
(20) $\displaystyle\boldsymbol{\chi}_{f,\tau}^{\text{T}}$
$\displaystyle\leftarrow\begin{pmatrix}\log_{10}\left(\left|u_{f,\tau}\right|+\delta^{\text{FEAT}}\right)&\log_{10}\left(\left|y_{f,\tau}\right|+\delta^{\text{FEAT}}\right)\end{pmatrix},$
(21)

respectively, with $\delta^{\text{FEAT}}>0$ being a small constant to avoid
numerical instabilities. Note that in principle any combination of signals and
transformations can be used to define a narrowband feature vector
$\boldsymbol{\chi}_{f,\tau}$. Yet, some combinations might be less promising,
considering for example the linear relationship of the microphone, error and
echo estimates.

We will now discuss the design of the hybrid feature vector
$\boldsymbol{\xi}_{\tau}$ which should be representative for the entire
frequency spectrum at time frame $\tau$. Similar to the narrowband feature
vector $\boldsymbol{\chi}_{f,\tau}$, it can be computed from a subset of the
complex-valued signals $u_{f,\tau}$, $y_{f,\tau}$, $e_{f,\tau}$ and
$\hat{d}_{f,\tau}$, yet, now considering all frequency bands. To condense the
respective broadband vectors to low-dimensional real-valued scalars, we
consider the arithmetic magnitude averages [39]

$\bar{y}_{\tau}=\frac{1}{F}\sum_{f=1}^{F}\left|y_{f,\tau}\right|$ (22)

with $\bar{e}_{\tau}$ and $\bar{\hat{d}}_{\tau}$ being defined accordingly. By
stacking a subset of the magnitude averages $\bar{y}_{\tau}$, $\bar{e}_{\tau}$
and $\bar{\hat{d}}_{\tau}$ into a vector, we obtain an exemplary hybrid
feature vector $\boldsymbol{\xi}_{\tau}$. As the feature (22) is independent
of the frequency resolution, the hybrid method keeps the flexibility benefit,
i.e., application to different frequency resolutions, of the narrowband
approach. Finally, we note that all feature vectors are normalized to zero
mean and unit variance with the respective statistics being estimated during
training.

###  III-D Output Layer Design of the DNNs

In Sec. III-B, we discussed the key idea of estimating the step-size
${\mu}_{f,\tau}$, which controls the CTF filter adaptation (cf. Eq. (7)), by a
DNN. Yet, due to the non-whiteness and non-stationarity that many acoustic
signals exhibit [40], a direct estimation of the LMS step-size $\mu_{f,\tau}$
requires the DNN to cover a large numerical range in its last layer. To remedy
this limitation we enforce a specific structure in the output layer of the
DNN. To motivate this structure we first consider the optimum NLMS step-size
(12). We observe that the second part of the product, i.e., the loudspeaker
power normalization, is easily estimated by recursive averaging

$\hat{\psi}_{f,\tau}^{\text{UU}}=\lambda_{U}\hat{\psi}_{f,\tau-1}^{\text{UU}}+(1-\lambda_{U})\left|\left|\boldsymbol{u}_{f,\tau}\right|\right|^{2}$
(23)

whereas the first part is difficult to estimate, yet, limited to the range
$[0,1]$. Thus, to address the numerical range difficulty, a straightforward
idea is to use a sigmoid activation in the output layer of the DNN and
subsequently normalize the outcome by the loudspeaker power estimate (23) [23,
24]. Yet, as the basic NLMS update is not interference-robust, the DNN
training might be complicated as no guidance is available. In contrast, the
traditional step-size estimators (11) - (13) have already been derived with
the goal of interference-robustness. To exploit this additional domain
knowledge, and therefore potentially simplifying the DNN training, we enforce
the following output layer, and thus step-size, structure [23]

$\mu_{f,\tau}\leftarrow\frac{m^{\text{DNN-$\mu$}}_{f,\tau}}{\hat{\psi}^{\text{UU}}_{f,\tau}+\left|m^{\text{DNN-e}}_{f,\tau}e_{f,\tau}\right|^{2}+\delta^{\text{VSS}}}$
(24)

with the DNN-provided masks $m^{\text{DNN-$\mu$}}_{f,\tau}$ and
$m^{\text{DNN-e}}_{f,\tau}$ being limited to the range from $0$ to $1$ and the
small regularization constant $\delta^{\text{VSS}}>0$. Note that the output
layer structure (24) is closely related to the error power-aware NLMS step-
size (11) with the DNN providing a frequency- and frame-selective
hyperparameter $m^{\text{EA-NLMS}}$ and controlling the error power
normalization by $m^{\text{DNN-e}}_{f,\tau}$. By choosing
$m^{\text{DNN-e}}_{f,\tau}$ $=$ $0$ we obtain a DNN-controlled NLMS update
similar to [24]. The overall approach, including a narrowband DNN, is shown in
Fig. 3.

DNN$\frac{m^{\text{DNN-$\mu$}}_{f,\tau}}{\hat{\psi}^{\text{UU}}_{f,\tau}+\left|m^{\text{DNN-e}}_{f,\tau}e_{f,\tau}\right|^{2}}$${\boldsymbol{\chi}}_{f,\tau}$${\mu}_{f,\tau}$${m}_{f,\tau}^{\text{DNN-}\mu}$${m}_{f,\tau}^{\text{DNN-e}}$$\boldsymbol{\zeta}_{f,\tau}$
Figure 3: Block diagram of the proposed DNN-controlled narrowband step-size
estimation approach with the DNN-provided masks
${m}_{f,\tau}^{\text{DNN-}\mu}$ and ${m}_{f,\tau}^{\text{DNN-e}}$, the feature
vector ${\boldsymbol{\chi}}_{f,\tau}$ and the internal DNN state vector
$\boldsymbol{\zeta}_{f,\tau}$.

###  III-E Loss Functions

We now discuss different loss functions for optimizing the broadband,
narrowband and hybrid DNN parameter vectors $\boldsymbol{\theta}^{\text{BB}}$,
${\boldsymbol{\theta}}^{\text{NB}}$, and ${\boldsymbol{\theta}}^{\text{HB}}$,
respectively. Note that we use $\boldsymbol{\theta}$ as representative for all
three parameter vectors in this section. The relation of the DNN parameter
vector $\boldsymbol{\theta}$ to the CTF filter estimates
$\boldsymbol{h}_{f,\tau}$, STFT-domain echo estimates $\hat{d}_{f,\tau}$ and
time-domain echo estimates $\hat{\underline{d}}_{\kappa}$ with the frequency
index $f\in\\{1,\dots,F\\}$, the frame index $\tau\in\\{1,\dots,T\\}$ and the
time-domain sample index $\kappa\in\\{1,\dots,K\\}$, respectively, is shown in
Fig. 4.

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\boldsymbol{h}}}_{f,0}}$${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\boldsymbol{h}}}_{f,1}}$${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\boldsymbol{h}}}_{f,2}}$$\dots$${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\boldsymbol{h}}}_{f,T}}$${\mu}_{f,1}$${\mu}_{f,2}$$\dots$${\mu}_{f,T}$$\dots$${{\hat{d}}}_{f,1}$${{\hat{d}}}_{f,2}$${{\hat{d}}}_{f,T}$$\boldsymbol{\theta}$Inverse
STFT$\dots$${{\hat{\underline{d}}}}_{1}$${{\hat{\underline{d}}}}_{2}$${{\hat{\underline{d}}}}_{K}$
Figure 4: Relation of the DNN parameter vector $\boldsymbol{\theta}$ to the
CTF filter estimates $\boldsymbol{h}_{f,\tau}$, STFT-domain echo estimates
$\hat{d}_{f,\tau}$ and time-domain echo estimates
$\hat{\underline{d}}_{\kappa}$.

In general, there are two types of loss functions which differ in their
optimization targets: 1) Approximation of oracle target step-sizes, 2)
Approximation of the desired echo signal. Possible choices for the desired
target step-sizes could be the minimum system distance NLMS or Kalman step-
sizes given in Eqs. (12) and (13). Yet, this approach is challenged by the
difficulty to design a proper distance measure as the effect of estimation
errors on the echo cancellation performance is complicated. Furthermore, as
optimum model-based step-sizes are typically derived by certain signal
assumptions, e.g., a linear Gaussian observation model within the Kalman
filter framework, their optimality cannot be guaranteed for realistic data.
Therefore, we resort in this paper to the second approach which represents an
end-to-end optimization strategy [23, 24, 25, 26, 27, 28, 29] as it directly
quantifies the effect of the step-size estimate on the echo cancellation
performance. Considering the different signal domains, i.e., time or frequency
domain, we always investigate pairs of loss functions in the following. A
first naive choice is given by the Mean Square Error (MSE) losses

$\displaystyle\mathcal{J}^{\text{FD-MSE}}(\boldsymbol{\theta})$
$\displaystyle=\frac{1}{TF}\sum_{\tau,f=1}^{T,F}\left|{d}_{f,\tau}-\hat{{d}}_{f,\tau}\right|^{2}$
(25) $\displaystyle\mathcal{J}^{\text{TD-MSE}}(\boldsymbol{\theta})$
$\displaystyle=\frac{1}{K}\sum_{\kappa=1}^{K}\left|\underline{d}_{\kappa}-\hat{\underline{d}}_{\kappa}\right|^{2}$
(26)

with $\underline{d}_{\kappa}$ and $\hat{\underline{d}}_{\kappa}$ denoting the
time-domain echo and estimated echo signals at sample index $\kappa$,
respectively, and $K,T$ denoting the number of time-domain samples and frames,
respectively. It should be noted that while the frequency-domain loss (25)
suggests an independent residual echo power minimization per frequency band,
its time-domain counterpart (26) couples the frequency-wise echo estimates by
the inverse STFT. In addition, we examine the negated logarithmic Echo Return
Loss Enhancement (ERLE)-type losses

$\displaystyle\mathcal{J}^{\text{FD-ERLE}}(\boldsymbol{\theta})$
$\displaystyle=-\log_{10}\frac{\delta^{\text{LOSS}}+\frac{1}{TF}\sum_{\tau,f=1}^{T,F}\left|{d}_{f,\tau}\right|^{2}}{\delta^{\text{LOSS}}+\frac{1}{TF}\sum_{\tau,f=1}^{T,F}\left|{d}_{f,\tau}-\hat{d}_{f,\tau}\right|^{2}}$
(27) $\displaystyle\mathcal{J}^{\text{TD-ERLE}}(\boldsymbol{\theta})$
$\displaystyle=-\log_{10}\frac{\delta^{\text{LOSS}}+\frac{1}{K}\sum_{\kappa=1}^{K}\left|\underline{d}_{\kappa}\right|^{2}}{\delta^{\text{LOSS}}+\frac{1}{K}\sum_{\kappa=1}^{K}\left|\underline{d}_{\kappa}-\hat{\underline{d}}_{\kappa}\right|^{2}}$
(28)

with $\delta^{\text{LOSS}}>0$ being a regularization factor to avoid numerical
instabilities. Note that the echo powers in the numerators of (27) and (28) do
not depend on the DNN parameter vector $\boldsymbol{\theta}$ and could
therefore be discarded. Nevertheless we keep it here for its analogy with the
commonly-used ERLE. Note that the respective MSE and ERLE losses essentially
differ in the additional transformation by a logarithm.

##  IV Experiments

We will now compare the various adaptation control approaches (cf. Sec.
III-B), signal feature vector choices (cf. Sec. III-C), output layer designs
(cf. Sec. III-D) and loss functions (cf. Sec. III-E) w.r.t. each other and
relative to various well-known traditional adaptation control algorithms (cf.
III-A) for a variety of challenging acoustic scenarios.

###  IV-A Experimental Setup

In this section, we describe the experimental setup which was used to train
the DNNs and compare the different adaptation control algorithms. As
introduced in Sec. II-A, we consider the microphone signal to be composed of a
loudspeaker echo, near-end speech and background noise component (cf. Eq.
(1)). The near-end speech signals for the training and test sets were sampled
randomly from the real ($12$ h) and blind test ($1.2$ h) data sets,
respectively, of the Microsoft AEC Challenges [3, 4, 5], which contains
recordings from a variety of acoustic environments and devices. The background
noise signals were sampled randomly from the training ($150$ h) and test sets
($17$ h) of the DNS [41] and the CHiME3 [42] challenges, respectively. The
echo signals were computed by a linear convolution of randomly-selected
Loudspeaker-Enclosure-Microphone (LEM) Impulse Responses (IRs) with randomly-
selected music or speech loudspeaker signals. The LEM IRs for computing the
echo signals in the training data were sampled randomly from the Microsoft AEC
Challenges [3, 4, 5] and MIT Survey [43] data sets and comprised a total of
$1134$ measured IRs from a variety of acoustic scenes and devices. Note that
[3, 4, 5] provide only sweep measurements and the respective IRs must be
computed in a pre-processing step. The LEM IRs for generating the test echo
data were taken from a disjoint subset of [3, 4, 5] and represented in total
$300$ different acoustic scenes and devices. The loudspeaker speech and music
signals for training were taken from the training subsets of [44] ($11.5$ h
and $919$ speakers) and [45, 46] ($5.5$ h from different genres),
respectively. The test speech signals were taken from [44, 47] ($2.6$ h and
$209$ speakers) and the music test signals were taken from the test subset of
[45, 46] ($2.4$ h). Note that we refrained from directly taking the recorded
echo signals from [3, 4, 5] as only speech far-end signals are available and
many recordings contain high-level fan noise which cannot be attenuated by an
echo canceler and thus complicate a proper evaluation. The respective fan
noise is typically dealt with by a post filter [1] which we do not consider in
this paper, yet, could easily be appended to the discussed echo
cancelers333Note that fan noise is still considered in the background noise
signals from [41] and thus its effect on the adaptation control performance
can be assessed.. To investigate the tracking behaviour of the algorithms,
which is of particular interest for mobile devices, we also considered time-
varying acoustic scenes. For this, we switched the LEM IRs in $90$% of the
signals after randomly-selected scene durations in the range from $3$ s to $6$
s. To include both, hard and smooth transitions, a linear fading with a
randomly-selected duration between $0$ s to $1$ s has been considered.
Furthermore, two thirds of the loudspeaker and near-end speech signals were
pre-processed by a temporal masking with random onsets and offsets to mimic
conversations with non- and partially overlapping activity. Finally, the
signals were scaled to obtain random near-end-to-echo and near-end-to-echo-
and-noise power ratios in the ranges $[-10,10]$ dB and $[20,40]$ dB,
respectively, and unit loudspeaker/microphone variances. Note that the
respective power estimates were computed only over segments with considerable
signal activity to limit the effect of pauses. All signals were sampled at a
sampling frequency of $16$ kHz and limited to a duration of $8$ s. In total,
we used $6$ h of training data, $1.5$ h of validation data and $1$ h of
testing data. Note that the LEM IRs, speech, music and noise signals of the
test data are disjoint from the training/validation data sets.

###  IV-B Algorithmic Settings

We now discuss the parameter settings of the various algorithms. The frame
shift and DFT length of the STFT were chosen to $128$ and $512$, respectively,
and the Hamming window has been applied prior to transformation444Note that a
quarter-frameshift is chosen to attain sufficient cross-band attenuation.. The
CTF filter length $L$ was set to $8$ which covers approximately $88$ ms of the
corresponding LEM IRs.

####  IV-B1 Deep Learning-based Adaptation Control

The DNN architecture was composed of a fully connected layer with a leaky RELU
activation, two stacked Gated Recurrent Unit (GRU) layers and two parallel
fully connected layers with sigmoid activations which map from the GRU states
to the different adaptation control masks $m^{\text{DNN-$\mu$}}_{f,\tau}$ and
$m^{\text{DNN-e}}_{f,\tau}$, respectively that define the step-size (cf. Eq.
(24)). This commonly used architecture was chosen to encode important
information about the near-end speech/background noise activity and filter
convergence state from the feature sequence within the GRU states. Note that
the proposed method is not limited to this architecture and other
architectures can straightforwardly be embedded. The recursive loudspeaker
power estimation factor and the various regularization constants are chosen to
$\lambda_{U}=0.9$, $\delta^{\text{FEAT}}=\delta^{\text{LOSS}}=10^{-12}$ and
$\delta^{\text{VSS}}=10^{-3}$, respectively. The dimension of the GRU states
was set to $128$ for the broadband step-size estimator (cf. Eq. (14)) and to
$64$ for the narrowband and hybrid estimators (cf. Eqs. (16) and (17)),
respectively. Note that the GRU dimensions have been chosen differently to
account for the respective task difficulties, i.e., estimation of a step-size
vector $\boldsymbol{\mu}_{\tau}$ in comparison to a scalar step-size
$\mu_{f,\tau}$. The DNNs were trained for $60$ epochs with the ADAM optimizer
[48], a step-size of $10^{-3}$ and an early stopping after $20$ epochs with no
performance improvement on the validation data. Furthermore, the ADAM step-
size was decreased by one-half after five consecutive epochs with no
performance improvement, and Euclidean norm-based gradient clipping with a
threshold of $0.5$ was applied. The batch size was chosen as $4$ and $32$ for
the narrowband/hybrid and broadband methods, respectively, to account for
hardware memory requirements. Finally, the optimum model was selected based on
the lowest training loss on the validation data.

####  IV-B2 Traditional and Oracle Adaptation Control

To assess the performance of the proposed deep learning-based adaptation
control algorithms, we consider various widely-used traditional step-size
estimation algorithms. In particular, we consider the error power-aware NLMS
algorithm [36, 37], denoted by EA-NLMS (cf. Eq. (11)), and the narrowband CTF
Kalman filter update [11, 12, 38], denoted by KF (cf. Eq. (13)). The
interference power $\psi_{f,\tau}^{ZZ}$ and the error power
$\psi_{f,\tau}^{EE}$ which are required for the respective updates are both
estimated by recursive averaging of the magnitude-squared error
$|e_{f,\tau}|^{2}$ with a smoothing factor of $0.5$ [18]. Furthermore, the raw
step-size $m^{\text{EA-NLMS}}$ and loudspeaker smoothing parameter
$\lambda_{U}$ of the EA-NLMS algorithm were chosen to $0.2$ and $0.9$,
respectively, and the state transition factor and initial filter estimation
variances of the Kalman filter update are chosen to $0.99$ and $1$,
respectively. The filter process noise power for the Kalman filter was chosen
proportionally to a temporally-smoothed average of the magnitude-squared
filter coefficients as described in [12]. In addition, an additive
regularization of $10^{-3}$ was added to the numerators of the EA-NLMS and KF
updates (cf. Eqs. (11) and (13)) and a minimum filter process noise power of
$10^{-3}$ was chosen for the Kalman filter which drastically improved their
robustness. Note that the hyperparameters have been optimized by a grid search
on the test data set which favors the baseline results.

Besides the traditional step-size estimators, we consider also two oracle
baselines which have knowledge about the individual signal components. On the
one hand, we consider an oracle NLMS algorithm, denoted by Oracle-Grad-NLMS,
which uses the true echo component $d_{f,\tau}$ instead of the microphone
signal $y_{f,\tau}$ to compute the CTF filter gradient (cf. Eqs. (2) and (7)).
On the other hand, an interference power (IP)-informed Kalman filter approach,
denoted by Oracle-IP-KF, is investigated which estimates the interference
power $\psi_{f,\tau}^{ZZ}$ by a recursive average of the magnitude-squared
interfering signal $|z_{f,\tau}|^{2}=|s_{f,\tau}+n_{f,\tau}|^{2}$ with a
smoothing parameter of $0.5$. It is important to note that while Oracle-Grad-
NLMS uses the true echo signal for computing the CTF filter update, and
therefore adapts continuously, Oracle-IP-KF exploits only oracle knowledge
about the interference power for computing the step-size, and therefore does
not adapt continuously. The step-size of Oracle-Grad-NLMS was chosen to $0.2$
and the loudspeaker power was estimated by recursive averaging with a
smoothing parameter of $\lambda_{U}=0.9$. Furthermore, a regularization
constant of $10^{-3}$ was added to the numerator of the update. The state
transition factor, initial filter estimation variances, minimum process noise
power and additive regularization of Oracle-IP-KF were chosen to $0.999$,
$0.1$, $10^{-4}$ and $1$, respectively. Finally, note that the results of both
oracle algorithms do not pose an upper performance bound for the linear CTF
echo estimation model (3) as the hyperparameters are frequency independent and
have been chosen equivalently for all acoustic scenes of the test data set,
based on the best average performance.

###  IV-C Performance Measures

We now introduce various measures to compare the performance of the different
adaptation control algorithms. One of the most popular measures is given by
the ERLE [2]

$\text{ERLE}=10\log_{10}\frac{\sum_{\kappa=1}^{K}\underline{d}_{\kappa}^{2}}{\sum_{\kappa=1}^{K}(\underline{d}_{\kappa}-\hat{\underline{d}}_{\kappa})^{2}}$
(29)

which is computed by the logarithmic ratio of the time-domain echo power
relative to the residual echo power after the echo canceler. It is important
to note that the ERLE definition (29) considers the true echo signal
$\underline{d}_{\kappa}$ and not the microphone signal
$\underline{y}_{\kappa}$ and thus can also be used to assess the echo power
attenuation during double-talk. Furthermore, it is, apart from regularization
constants, equivalent to the negated loss function (28). Due to its simple
interpretability, the ERLE is widely used in practice. Yet, it is does not
consider the characteristics of the possibly distorted desired near-end speech
signal. Thus, we additionally introduce the Perceptual Evaluation of Speech
Quality (PESQ) [49]

$\text{PESQ}=f^{\text{PESQ}}(\underline{s}_{\kappa},\underline{s}_{\kappa}+\underline{d}_{\kappa}-\hat{\underline{d}}_{\kappa})$
(30)

with the residual echo $\underline{d}_{\kappa}-\hat{\underline{d}}_{\kappa}$
representing an additive distortion. Note that we refrained from considering
the background noise component $\underline{n}_{\kappa}$ in (30) as it cannot
be canceled by the echo canceler and thus only dilutes the comparability.

###  IV-D Evaluation

In the following, the different adaptation control algorithms are compared
w.r.t. their performance on the $450$ acoustic scenes of the test data set
(cf. Sec. IV-A). We always provide mean and standard deviation in the format
mean/std with the best measure of each algorithmic class being typeset in bold
font. For ease of readability, we abbreviate the DNN-based adaptation control
algorithms in the results tables by Broadband (BB), Narrowband (NB) and Hybrid
(HB), respectively.

####  IV-D1 Number of DNN Parameters and Runtime

We start by comparing the algorithms in terms of DNN parameter count, runtime
$t_{\text{proc}}$ to process a single STFT block of length $32$ ms and Real
Time Factor (RTF). Tab. I shows the respective measures for the considered
traditional and deep learning-based adaptation control algorithms.

TABLE I: Approximate number of DNN parameters, block processing runtime $t_{\text{proc}}$ and RTF for a typical configuration of the different adaptation control approaches. | EA-NLMS | KF | BB-DNN | NB-DNN | HB-DNN
---|---|---|---|---|---
#Parameters | — | — | $330\cdot 10^{3}$ | $\boldsymbol{50\cdot 10^{3}}$ | $50\cdot 10^{3}$
$t_{\text{proc}}$ in ms | $\boldsymbol{0.11}~{}\hskip 1.84949pt$ | ${0.15}~{}\hskip 1.84949pt$ | $0.59~{}$ | $0.79~{}~{}$ | $0.83$
RTF | $\boldsymbol{0.014}$ | $0.018$ | $0.074$ | $0.098$ | $0.10$

The block processing runtime $t_{\text{proc}}$ and RTF have been measured on
an Intel Xeon<EMAIL_ADDRESS>CPU. Note that all presented values depend on the
specific algorithmic configuration, i.e., dimensions of the signal feature
vectors and GRU states. Thus, the values can only be considered as typical
representatives of the respective algorithmic class. Obviously the broadband
deep learning-based adaptation control (BB-DNN) exhibits significantly more
parameters than the narrowband (NB-DNN) and hybrid (HB-DNN) versions which is
straightforwardly explained by the larger feature, masking and GRU dimensions.
Furthermore, the narrowband and hybrid DNNs exhibit a similar number of DNN
parameters as they differ only by an additional fully connected layer, with at
most $192$ parameters, which maps from the hybrid feature vector
$\boldsymbol{\xi}_{\tau}$ to the GRU states (cf. Fig. 2). Considering the
runtime, the DNN-based algorithms are approximately four to seven times slower
than their traditional counterparts with the hybrid method exhibiting the
highest runtime. Yet, it should be noted that the runtime of all algorithms
could easily be reduced by subsampling the step-size inference, i.e., keeping
it constant for several successive frames. This is in particular attractive
for an STFT with a quarter-frameshift as considered in this paper. Finally, it
might seem counterintuitive that the broadband method has a lower processing
runtime than the narrowband/hybrid methods, despite exhibiting more
parameters. Yet, this is straightforwardly explained by considering that the
broadband approach requires only a single DNN inference to jointly estimate
the step-sizes for all frequency bands while the narrowband and hybrid methods
execute individual DNN inferences per frequency band.

####  IV-D2 Loss Functions

We now investigate the different loss functions for DNN training that have
been introduced in Sec. III-E. Tab. II shows the performance evaluation for
the broadband, narrowband and hybrid DNN-based adaptation control algorithms
after being trained with the different loss functions (25) - (28).

TABLE II: Loss function comparison for broadband (BB), narrowband (NB) and hybrid (HB) DNN-based adaptation control algorithms with the absolute STFT-domain loudspeaker and microphone signals as feature vectors. Loss (Eq.) | Structure | ERLE | PESQ
---|---|---|---
FD-MSE (25) | BB-DNN | $12.79/3.1$ | $1.93/0.8$
NB-DNN | $14.54/3.3$ | $\textbf{2.11}/0.8$
HB-DNN | $\textbf{14.66}/3.3$ | $\textbf{2.11}/0.8$
TD-MSE (26) | BB-DNN | $12.84/3.2$ | $1.92/0.8$
NB-DNN | $14.48/3.3$ | $2.10/0.8$
HB-DNN | $\textbf{14.61}/3.3$ | $\textbf{2.11}/0.8$
FD-ERLE (27) | BB-DNN | $12.78/3.5$ | $1.94/0.8$
NB-DNN | $14.51/3.4$ | $2.10/0.8$
| HB-DNN | $\textbf{14.70}/3.4$ | $\textbf{2.12}/0.8$
TD-ERLE (28) | BB-DNN | $12.99/3.3$ | $1.94/0.8$
NB-DNN | ${14.71}/3.4$ | $\textbf{2.12}/0.8$
HB-DNN | $\textbf{14.85}/3.4$ | $\textbf{2.12}/0.8$
EA-NLMS | $10.16/3.3$ | $1.81/0.7$
KF | $\textbf{11.99}\hskip 0.28436pt/3.4$ | $\textbf{1.96}/0.7$
Oracle-IP-KF | $14.90/3.6$ | $2.16/0.8$
Oracle-Grad-NLMS | $\textbf{15.17}/3.7$ | $\textbf{2.20}/0.9$

As feature vectors we considered the absolute STFT-domain loudspeaker and
microphone signals and the DNN output layer was chosen according to (24). We
conclude from Tab. II that all deep learning-based step-size estimators (cf.
rows $1-4$) outperform the traditional error power-aware NLMS and Kalman
filter baselines (cf. EA-NLMS and KF) in terms of ERLE. Among the DNN-based
step-size estimators, the narrowband and hybrid methods outperform their
broadband counterparts by approximately $1.7$ dB ERLE and $0.17$ PESQ on
average. It might seem counterintuitive that the narrowband and hybrid models
perform better than the broadband approach despite having much fewer DNN
parameters (cf. Tab. I). Yet, it should be noted again that both the
narrowband and the hybrid algorithms use individual DNNs, with different
internal states, per frequency band. Considering the various loss functions,
we observe only minor differences with small benefits for the TD-ERLE cost
function (28). Thus, we conclude that the different deep learning-based
adaptation control algorithms are highly robust w.r.t. the cost function
choice and choose the TD-ERLE cost function (28) for the following
experiments.

####  IV-D3 DNN Feature Vectors

In this section, the influence of the feature vector choice (cf. Sec. III-C)
on the echo cancellation performance is investigated. We first examine the
different transformations for mapping from the complex-valued signals to real-
valued features. Tab. III shows the average performance measures of the
narrowband and broadband DNN-based step-size estimators for the real- and
imaginary (cf. Eq. (19)), magnitude (cf. Eq. (20)) and logarithmic magnitude
(cf. Eq. (21)) transformations of the complex-valued loudspeaker and
microphone signals.

TABLE III: Comparison of different feature transformations (cf. Eqs. (19) - (21)) for broadband (BB) and narrowband (NB) deep learning-based adaptation control. Structure | Feature Mapping | ERLE | PESQ
---|---|---|---
BB-DNN | $\left(\mathcal{R}\\{\cdot\\},\mathcal{I}\\{\cdot\\}\right)$ | $11.59/3.2$ | $1.85/0.7$
$\left|\cdot\right|$ | $\textbf{12.99}/3.3$ | $\textbf{1.94}/0.8$
$\log_{10}\left(\left|\cdot\right|+\delta^{\text{FEAT}}\right)$ | $12.88/3.4$ | $\textbf{1.94}/0.8$
NB-DNN | $\left(\mathcal{R}\\{\cdot\\},\mathcal{I}\\{\cdot\\}\right)$ | $14.67/3.4$ | $\textbf{2.12}/0.8$
$\left|\cdot\right|$ | $\textbf{14.71}/3.4$ | $\textbf{2.12}/0.8$
$\log_{10}\left(\left|\cdot\right|+\delta^{\text{FEAT}}\right)$ | $14.66/3.4$ | $\textbf{2.12}/0.8$

We conclude that all feature transformations perform similar for the
narrowband method. In contrast, a clear benefit for the magnitude-based
features in comparison to the real and imaginary features can be observed for
the broadband method. This might be surprising as the latter contain the most
information and the magnitude could be computed from the real and imaginary
components. Yet, this might require more powerful DNNs architectures. It
indicates also that the phase information is less relevant for adaptation
control which is also suggested by many traditional step-size estimators (cf.
Eqs. (10) - (12)). In addition, it should be considered that the feature
dimension of the real and imaginary feature vector is twice the dimension of
the magnitude-based feature vectors. As the simple magnitude transformation
tends to perform best, it is chosen as basis for the subsequent
investigations.

TABLE IV: Comparison of different magnitude-based signal features for broadband (BB), narrowband (NB) and hybrid (HB) DNN-based adaptation control algorithms. Structure | Features | ERLE | PESQ
---|---|---|---
$u_{f}$ | $y_{f}$ | $e_{f}$ | $\hat{d}_{f}$ | $\bar{y}$ | $\bar{e}$ | $\bar{\hat{d}}$
BB-DNN | ✓ | ✓ |  |  |  |  |  | $12.99/3.3$ | $1.94/0.8$
✓ |  | ✓ |  |  |  |  | $13.49/3.5$ | $1.99/0.8$
| ✓ |  | ✓ |  |  |  | $13.35/3.5$ | $1.98/0.8$
✓ | ✓ | ✓ |  |  |  |  | $13.71/3.5$ | $2.00/0.8$
✓ | ✓ | ✓ | ✓ |  |  |  | $\textbf{14.04}/3.4$ | $\textbf{2.03}/0.8$
NB-DNN | ✓ | ✓ |  |  |  |  |  | $14.71/3.4$ | $2.12/0.8$
✓ |  | ✓ |  |  |  |  | $14.60/3.5$ | $2.13/0.8$
| ✓ |  | ✓ |  |  |  | $14.71/3.4$ | $2.12/0.8$
✓ | ✓ | ✓ |  |  |  |  | $15.10/3.5$ | $2.15/0.8$
✓ | ✓ | ✓ | ✓ |  |  |  | $\textbf{15.17}/3.5$ | $\textbf{2.16}/0.8$
HB-DNN | ✓ | ✓ |  |  | ✓ |  |  | $14.85/3.4$ | $2.12/0.8$
✓ |  | ✓ |  |  | ✓ |  | $14.80/3.6$ | $2.13/0.8$
✓ | ✓ | ✓ |  | ✓ | ✓ |  | ${15.35}/3.5$ | ${2.16}/0.8$
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | $\textbf{15.60}/3.5$ | $\textbf{2.18}/0.8$
EA-NLMS | $10.16/3.3$ | $1.81/0.7$
KF | $\textbf{11.99}\hskip 0.28436pt/3.4$ | $\textbf{1.96}/0.7$
Oracle-IP-KF | $14.90/3.6$ | $2.16/0.8$
Oracle-Grad-NLMS | $\textbf{15.17}/3.7$ | $\textbf{2.20}/0.9$

We now examine the effect of different signal choices for computing the
magnitude-based features. The average performance measures of the accordingly
trained broadband, narrowband and hybrid DNN-based adaptation control
algorithms is shown in Tab. IV. We observe that in general increasing the
feature set improves the adaptation control performance. In particular, the
joint usage of the microphone and the error signal seems to be beneficial.
This might be explained by the importance of the convergence state information
of the echo canceler that is difficult to extract from the respective
individual signals. We finally observe that the best-performing DNN-based
step-size estimator outperforms the oracle NLMS method in terms of ERLE and
almost attains its PESQ values. Yet, it should be noted again that both oracle
algorithms assume equivalent (frequency independent) hyperparameters for all
acoustic scenes of the test data set.

####  IV-D4 Output Layer Design

Finally, we investigate different output layer designs of the DNN (Sec.
III-D). In particular, we examine the effect of omitting the adaptive error
power normalization in Eq. (24) by setting $m^{\text{DNN-e}}_{f,\tau}=0$ or a
non-adaptive selection by setting $m^{\text{DNN-e}}_{f,\tau}=1$. Furthermore,
we evaluate a frequency-independent selection of the step-size masks
$m^{\text{DNN-$\mu$}}_{f,\tau}$ and $m^{\text{DNN-e}}_{f,\tau}$, respectively,
for the broadband method, i.e., an identical choice for all frequency bands.
As feature vectors we use the magnitude loudspeaker, microphone and error
signals which showed good performance according to Tab. IV. The respective
ERLE and PESQ results are shown in Tab. V. We conclude that frequency-
selective step-size masks significantly outperform a frequency-independent
choice (cf. BB-DNN) which can be explained by the non-whiteness which is
typical for speech and audio signals, both as desired signals and as
interference. Furthermore, a general error power normalization within the
output layer of the broadband approach, either controlled by a DNN (cf.
$m^{\text{DNN-e}}_{f,\tau}$$=$selective) or a static choice (cf.
$m^{\text{DNN-e}}_{f,\tau}$$=$$1$), is beneficial in comparison to a simple
DNN-controlled NLMS (cf. $m^{\text{DNN-e}}_{f,\tau}$$=$$0$). Yet, the DNN
seems to infer values close to one as $m^{\text{DNN-e}}_{f,\tau}$$=$selective
and $m^{\text{DNN-e}}_{f,\tau}$$=$$1$ show a similar performance. In contrast,
no clear benefit of the adaptive error-power normalization within the
narrowband and hybrid approaches can be observed. This might be a result of
the step-size inference by individual DNNs per frequency band which can handle
for the considered architectures more challenging situations in comparison to
the broadband approach. Thus, a more conservative, i.e., smaller, step-size
choice by error power normalization is not required.

TABLE V: Comparison of different output layer designs (cf. Eq. (24)) for DNN-based step-size estimation. While selective approaches estimate a different mask for each frequency band, non-selective methods estimate a frequency-independent mask which is equivalently applied to all frequency bands. Structure | Output layer | ERLE | PESQ
---|---|---|---
$m^{\text{DNN-$\mu$}}_{f,\tau}$ | $m^{\text{DNN-e}}_{f,\tau}$
BB-DNN | Selective | Selective | $\textbf{13.71}/3.5$ | ${2.00}/0.8$
Selective | $0$ | $13.03/3.5$ | $1.95/0.8$
Selective | $1$ | $13.70/3.4$ | $\textbf{2.01}/0.8$
Non-Selective | Non-Selective | $12.97/3.4$ | $1.90/0.7$
Non-Selective | $0$ | $11.45/3.8$ | $1.76/0.7$
Non-Selective | $1$ | $12.87/3.3$ | $1.89/0.7$
NB-DNN | Selective | Selective | $\textbf{15.10}/3.5$ | $\textbf{2.15}/0.8$
Selective | $0$ | $15.07/3.5$ | $\textbf{2.15}/0.8$
Selective | $1$ | $14.92/3.4$ | $2.14/0.8$
HB-DNN | Selective | Selective | $\textbf{15.35}/3.5$ | $\textbf{2.16}/0.8$
Selective | $0$ | $15.30/3.5$ | $\textbf{2.16}/0.8$
Selective | $1$ | $15.13/3.5$ | $\textbf{2.16}/0.8$

###  IV-E Visual Interpretation

After comparing the various approaches numerically, we will now visually
interpret the DNN-based step-size estimates. For this we show in Fig. 5 the
logarithmic interference, i.e., summed near-end speech and background noise,
echo and residual echo signal power spectrograms relative to the DNN-provided
step-size mask $m^{\text{DNN-$\mu$}}_{f,\tau}$ for a specific sequence. Note
that around $3.1$ s an echo path change occurs which is indicated by the
vertical red lines. A broadband approach without any error-power
normalization, i.e., $m^{\text{DNN-e}}_{f,\tau}$ $=$ $0$, and only a single
GRU layer with $128$ internal states was used to create the results. The
stacked magnitude loudspeaker, microphone and error signals were used as
feature vectors. To gain more insights into what the DNN actually learns, we
additionally try to find similar GRU state vectors across temporally different
frames [50]. For this we cluster the respective GRU state vectors of the
sequence by agglomerative clustering with the city block metric into two
different clusters [51]. Subfig. 5(e) shows the frame-based assignment of the
internal GRU state vectors to the two different classes. We observe from the
cluster assignment results in Subfig. 5(e) that the first class indicates
error signal frames which are dominated by interference, i.e., near-end speech
or background noise (or no echo activity), whereas the second cluster
indicates error signal frames which are dominated by residual echo. Thus, we
conclude that the GRU states encode important information about the
interference activity which points to the interpretation that the DNN includes
a refined double-talk detector. The step-size vectors (columns in Subfig.
5(d)) which are computed from the GRU states of the second cluster exhibit a
clear low-pass structure, i.e., high-values in the low frequency range, which
is reasonable considering the speech loudspeaker signal. Of particular
interest is the step-size selection after the echo path change: While there is
no adaptation during the initial double-talk period from
$3.1~{}\text{s}-3.5~{}\text{s}$, the subsequent decreasing interference power
results in high step-sizes and thus rapid filter updates.

$0$$2$$4$$6$$8$Frequency [kHz]$-60$$-40$$-20$$0$ (a)
$10\log_{10}|s_{f,\tau}+n_{f,\tau}|^{2}$

$0$$2$$4$$6$$8$Frequency [kHz]$-60$$-40$$-20$$0$ (b)
$10\log_{10}|d_{f,\tau}|^{2}$

$0$$2$$4$$6$$8$Frequency [kHz]$-60$$-40$$-20$$0$ (c)
$10\log_{10}|d_{f,\tau}-\hat{d}_{f,\tau}|^{2}$

$0$$2$$4$$6$$8$Frequency [kHz]$0$$0.2$$0.4$$0.6$$0.8$ (d) DNN-provided step-
size mask $m^{\text{DNN-$\mu$}}_{f,\tau}$

$0$$1$$2$$3$$4$$5$$1$$2$Time in secClass Index (e) Cluster assignment of GRU
states.

Figure 5: Logarithmic interference, echo and residual echo power spectrograms
in comparison to the DNN-provided step-size mask
$m^{\text{DNN-$\mu$}}_{f,\tau}$ with the red line indicating an echo path
change. In Subfig. 5(e), the frame-based assignment of the internal GRU states
of the DNN to two different classes/clusters is shown.

##  V Conclusion

In this paper, we investigated various DNN-based adaptation control methods
for frequency domain acoustic echo cancellation. In particular, we compared
broadband and narrowband step-size inference approaches and proposed a hybrid
method which combines the best out of both worlds. Our results show
significant performance improvements of deep learning-based step-size
estimation relative to traditional approaches, in particular for dynamic
acoustic scenes. Furthermore, we provide a general view on different signal
feature vectors, cost functions and DNN output layers which yields novel
insights and guidelines for the design of other deep learning-based adaptation
control algorithms. As future work, our generic step-size adaptation approach
can be applied to unsupervised system identification applications, e.g.,
relative transfer function estimation, and to the joint control of various
algorithmic components of a multichannel hands-free speech interface, e.g.,
echo canceler, beamformer and spectral postfilter, by a single DNN as outlined
in [25].

## References

  * [1] E. Hänsler and G. Schmidt, _Acoustic Echo and Noise Control: A practical Approach_. NJ, USA: Wiley-Interscience, 2004.
  * [2] G. Enzner, H. Buchner, A. Favrot, and F. Kuech, “Acoustic Echo Control,” in _Academic Press Library in Signal Processing_. Florida, USA: Elsevier, 2014, vol. 4, pp. 807–877.
  * [3] K. Sridhar, R. Cutler, A. Saabas, T. Parnamaa, M. Loide, H. Gamper, S. Braun, R. Aichner, and S. Srinivasan, “ICASSP 2021 Acoustic Echo Cancellation Challenge: Datasets, testing framework, and results,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Toronto, Canada, Jun. 2021, pp. 151–155.
  * [4] R. Cutler, A. Saabas, T. Parnamaa, M. Loide, S. Sootla, M. Purin, H. Gamper, S. Braun, K. Sorensen, R. Aichner, and S. Srinivasan, “Interspeech 2021 Acoustic Echo Cancellation Challenge,” in _Interspeech_ , Brno, Czech Republic, Aug. 2021.
  * [5] R. Cutler, A. Saabas, T. Parnamaa, M. Purin, H. Gamper, S. Braun, K. Sørensen, and R. Aichner, “ICASSP 2022 Acoustic Echo Cancellation Challenge,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Singapore, Singapore, May 2022, pp. 9107–9111.
  * [6] S. Haykin, _Adaptive filter theory_ , 4th ed. Upper Saddle River, New Jersey: Prentice Hall, 2002.
  * [7] A. Mader, H. Puder, and G. U. Schmidt, “Step-size control for acoustic echo cancellation filters – An overview,” _Signal Processing_ , vol. 80, no. 9, pp. 1697–1719, 2000.
  * [8] T. Gansler, M. Hansson, C.-J. Ivarsson, and G. Salomonsson, “A double-talk detector based on coherence,” _IEEE Transactions on Communications_ , vol. 44, no. 11, pp. 1421–1427, 1996.
  * [9] J. Benesty, D. Morgan, and J. Cho, “A new class of doubletalk detectors based on cross-correlation,” _IEEE Transactions on Speech and Audio Processing_ , vol. 8, no. 2, pp. 168–172, 2000.
  * [10] B. H. Nitsch, “A frequency-selective stepfactor control for an adaptive filter algorithm working in the frequency domain,” _Signal Processing_ , vol. 80, no. 9, pp. 1733––1745, Sep. 2000.
  * [11] G. Enzner and P. Vary, “Frequency-domain adaptive Kalman filter for acoustic echo control in hands-free telephones,” _Signal Processing_ , vol. 86, no. 6, pp. 1140–1156, 2006.
  * [12] F. Kuech, E. Mabande, and G. Enzner, “State-space architecture of the partitioned-block-based acoustic echo controller,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Florence, Italy, May 2014, pp. 1295–1299.
  * [13] J. Benesty, H. Rey, L. Vega, and S. Tressens, “A Nonparametric VSS NLMS Algorithm,” _IEEE Signal Processing Letters_ , vol. 13, no. 10, pp. 581–584, Oct. 2006.
  * [14] J.-M. Valin, “On Adjusting the Learning Rate in Frequency Domain Echo Cancellation With Double-Talk,” _IEEE Transactions on Audio, Speech and Language Processing_ , vol. 15, no. 3, pp. 1030–1034, Mar. 2007.
  * [15] F. Nesta, T. S. Wada, and B. Juang, “Batch-online semi-blind source separation applied to multi-channel acoustic echo cancellation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 3, pp. 583–599, 2011.
  * [16] J. Gunther, “Learning Echo Paths During Continuous Double-Talk Using Semi-Blind Source Separation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 2, pp. 646–660, 2012.
  * [17] G. Cheng, L. Liao, H. Chen, and J. Lu, “Semi-Blind Source Separation for Nonlinear Acoustic Echo Cancellation,” _IEEE Signal Processing Letters_ , vol. 28, pp. 474–478, 2021.
  * [18] F. Yang, G. Enzner, and J. Yang, “Frequency-Domain Adaptive Kalman Filter With Fast Recovery of Abrupt Echo-Path Changes,” _IEEE Signal Processing Letters_ , vol. 24, no. 12, pp. 1778–1782, Dec. 2017.
  * [19] T. Haubner, A. Brendel, M. Elminshawi, and W. Kellermann, “Noise-robust adaptation control for supervised acoustic system identification exploiting a noise dictionary,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Toronto, Canada, June 2021, pp. 945–949.
  * [20] T. Haubner, M. M. Halimeh, A. Brendel, and W. Kellermann, “A synergistic kalman- and deep postfiltering approach to acoustic echo cancellation,” in _European Signal Processing Conference (EUSIPCO)_ , Dublin, Ireland, August 2021, pp. 990–994.
  * [21] A. Ivry, I. Cohen, and B. Berdugo, “Deep adaptation control for acoustic echo cancellation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Singapore, Singapore, May 2022.
  * [22] O. Schwartz and A. Schwartz, “RNN-based step-size estimation for the RLS algorithm with application to acoustic echo cancellation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Rhodes Island, Greece, June 2023.
  * [23] T. Haubner, A. Brendel, and W. Kellerman, “End-To-End Deep Learning-Based Adaptation Control for Frequency-Domain Adaptive System Identification,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Singapore, Singapore, May 2022, pp. 766–770.
  * [24] H. Zhang, S. Kandadai, H. Rao, M. Kim, T. Pruthi, and T. Kristjansson, “Deep Adaptive AEC: Hybrid of Deep Learning and Adaptive Acoustic Echo Cancellation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Singapore, Singapore, May 2022, pp. 756–760.
  * [25] T. Haubner and W. Kellermann, “Deep Learning-Based Joint Control of Acoustic Echo Cancellation, Beamforming and Postfiltering,” in _European Signal Processing Conference (EUSIPCO)_ , Belgrade, Serbia, August 2022.
  * [26] J. Casebeer, N. J. Bryan, and P. Smaragdis, “Meta-AF: Meta-Learning for Adaptive Filters,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 355–370, February 2023.
  * [27] J. Wu, J. Casebeer, N. J. Bryan, and P. Smaragdis, “Meta-Learning for Adaptive Filters with higher-order Frequency Dependencies,” in _International Workshop on Acoustic Signal Enhancement (IWAENC)_ , Bamberg, Germany, September 2022.
  * [28] D. Yang, F. Jiang, W. Wu, X. Fang, and M. Cao, “Low-complexity acoustic echo cancellation with neural kalman filtering,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Rhodes Island, Greece, June 2023.
  * [29] Y. Zhang, M. Yu, H. Zhang, D. Yu, and D. Wang, “NeuralKalman: A Learnable Kalman Filter for Acoustic Echo Cancellation,” Feb. 2023, arXiv:2301.12363.
  * [30] W. Kellermann, “Analysis and design of multirate systems for cancellation of acoustical echoes,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , New York, USA, Apr. 1988, pp. 2570–2573.
  * [31] P. S. Diniz, _Adaptive Filtering: Algorithms and Practical Implementation_ , 4th ed. Springer US, 2012.
  * [32] Y. Avargel and I. Cohen, “System Identification in the Short-Time Fourier Transform Domain With Crossband Filtering,” _IEEE Transactions on Audio, Speech and Language Processing_ , vol. 15, no. 4, pp. 1305–1319, May 2007.
  * [33] J. Franzen, E. Seidel, and T. Fingscheidt, “AEC in a netshell: On target and topology choices for FCRN acoustic echo cancellation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Toronto, Canada, June 2021, pp. 156–160.
  * [34] E. Seidel, J. Franzen, M. Strake, and T. Fingscheidt, “Y2-net FCRN for acoustic echo and noise suppressio,” in _Interspeech_ , Brno, Czechia, August 2021.
  * [35] S. Braun and M. L. Valero, “Task splitting for dnn-based acoustic echo and noise removal,” in _2022 International Workshop on Acoustic Signal Enhancement (IWAENC)_ , Bamberg, Germany, Sep. 2022.
  * [36] J. E. Greenberg, P. M. Zurek, and M. Brantley, “Evaluation of feedback-reduction algorithms for hearing aids,” _The Journal of the Acoustical Society of America_ , vol. 108, no. 5, pp. 2366–2376, Nov. 2000.
  * [37] A. Spriet, S. Doclo, M. Moonen, and J. Wouters, “Feedback control in hearing aids,” _Springer Handbook of Speech Processing_ , pp. 979–1000, 2008.
  * [38] M. L. Valero, “Acoustic echo reduction for multiple loudspeakers and microphones: Complexity reduction and convergence enhancement,” doctoral thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 2019\.
  * [39] A. Schwarz, C. Hofmann, and W. Kellermann, “Spectral feature-based nonlinear residual echo suppression,” in _IEEE Workshop on Applications of Signal Processing to Audio and Acoustics_ , New Paltz, NY, USA, Oct. 2013.
  * [40] P. Vary and R. Martin, _Digital Speech Transmission_. John Wiley & Sons, 2006.
  * [41] H. Dubey, A. Aazami, V. Gopal, B. Naderi, S. Braun, R. Cutler, H. Gamper, M. Golestaneh, and R. Aichner, “Deep speech enhancement challenge at ICASSP 2023,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Rhodes Island, Greece, June 2023\.
  * [42] J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third ‘chime’ speech separation and recognition challenge: Dataset, task and baselines,” in _IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , Scottsdale, Arizona, USA, Dec. 2015, pp. 504–511.
  * [43] J. Traer and J. H. McDermott, “Statistics of natural reverberation enable perceptual separation of sound and space,” _Proceedings of the National Academy of Sciences_ , vol. 113, no. 48, 2016.
  * [44] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Brisbane, Australia, Apr. 2015, pp. 5206–5210.
  * [45] G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,” _IEEE Transactions on Speech and Audio Processing_ , vol. 10, no. 5, pp. 293–302, 2002.
  * [46] B. L. Sturm, “An analysis of the GTZAN music genre dataset,” in _International ACM Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies_ , Nara, Japan, Oct. 2012.
  * [47] “LibriVox: Free public domain audiobooks,” https://librivox.org, accessed: 2023-03-27.
  * [48] D. Kingma and J. Ba, “ADAM: A method for stochastic optimization,” 2014, arXiv:1412.6980.
  * [49] ITU-T Recommendation P.862.2, “Wideband extension to recommendation P.862 for the assessment of wideband telephone networks and speech codecs,” ITU, Recommendation, Nov. 2007.
  * [50] A. Briegleb, T. Haubner, V. Belagiannis, and W. Kellermann, “Localizing spatial information in neural spatiospectral filters,” 2023, arXiv:2303.08052.
  * [51] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.