# Speech Dereverberation with Frequency Domain Autoregressive Modeling

Anurenjan Purushothaman, , Debottam Dutta, Rohit Kumar, and Sriram Ganapathy
This paper was partly funded by grants from the Samsung Research India,
Bangalore.A. Purushothaman, and S. Ganapathy are with the Learning and
Extraction of Acoustic Patterns (LEAP) lab, Department of Electrical
Engineering, Indian Institute of Science, Bangalore, India, 560012. D. dutta
is with University of Illinois Urbana-Champaign. R. Kumar is associated with
Johns Hopkins University. e-mail: {anurenjanp<EMAIL_ADDRESS>
###### Abstract

Speech applications in far-field real world settings often deal with signals
that are corrupted by reverberation. The task of dereverberation constitutes
an important step to improve the audible quality and to reduce the error rates
in applications like automatic speech recognition (ASR). We propose a unified
framework of speech dereverberation for improving the speech quality and the
ASR performance using the approach of envelope-carrier decomposition provided
by an autoregressive (AR) model. The AR model is applied in the frequency
domain of the sub-band speech signals to separate the envelope and carrier
parts. A novel neural architecture based on dual path long short term memory
(DPLSTM) model is proposed, which jointly enhances the sub-band envelope and
carrier components. The dereverberated envelope-carrier signals are modulated
and the sub-band signals are synthesized to reconstruct the audio signal back.
The DPLSTM model for dereverberation of envelope and carrier components also
allows the joint learning of the network weights for the down stream ASR task.
In the ASR tasks on the REVERB challenge dataset as well as on the VOiCES
dataset, we illustrate that the joint learning of speech dereverberation
network and the E2E ASR model yields significant performance improvements over
the baseline ASR system trained on log-mel spectrogram as well as other
benchmarks for dereverberation (average relative improvements of $10$-$24$%
over the baseline system). The speech quality improvements, evaluated using
subjective listening tests, further highlight the improved quality of the
reconstructed audio.

###### Index Terms:

Frequency domain auto-regressive modeling, Dereverberation, end-to-end ASR,
Joint modeling.

##  I Introduction

The wide spread adoption of voice technologies like meeting assistants, smart
speakers, in-car entertainment systems, and virtual assistants imply that the
audio signal at the input of these system is impacted by reverberation and
noise artifacts [1]. The performance of the downstream applications like,
automatic speech recognition, speaker/language recognition, emotion
recognition or voice activity detection, is shown to degrade significantly in
reverberant conditions [2, 3, 4, 5, 6]. The performance deterioration is
primarily attributed to the smearing of the temporal envelopes caused by
reverberation [7]. The temporal smearing is caused by the emplacement of the
direct path signal on reflected signals, resulting in a weighted summation of
delayed components [8].

One of the approaches to deal with the adverse far-field conditions is to
develop a front-end which performs signal enhancement. Several techniques for
dereverberation like signal processing based (for example, weighted prediction
error (WPE) [9]), mask estimation based (for example, time-frequency mask
estimation [10]) and multi-channel beamforming based (for example, time-delay
estimation [11], generalized eigen-value [12, 13]) have been explored to
improve the signal quality. On the other hand, another effective approach for
system development in reverberant conditions is that of multi-condition
training [14]. However, even with these pre-processing and multi-condition
training methods, the beamformed signal contains significant amount of
temporal smoothing which adversely impacts the ASR performance [15].

In the traditional setting, the first step in the analysis of a signal is the
short-term Fourier transform (STFT). The key assumptions about the convolution
model of reverberation artifacts, is applicable for a long-analysis window in
the time domain, or using convolutional transfer function with cross-band
filters in the STFT domain [16, 17]. In our case, we use the former approach
of long analysis window and explore dereverberation in the sub-band envelope
domain. As the reverberation is a long-term convolution effect, we highlight
that room impulse response (typically with a T$60>400ms$) can be absorbed as a
multiplication in the frequency domain, as well as a convolution in the sub-
band envelope domain.

In this paper, we investigate the effect of reverberation on the long-term
sub-band signals of speech using an envelope-carrier decomposition. The
extraction of the sub-band envelope is achieved using the autoregressive (AR)
modeling approach in the spectral domain, termed as frequency domain linear
prediction (FDLP). Our previous work showed that a feature level enhancement
with the FDLP envelope improves speech recognition performance [18, 19].
However, the prior works did not allow the reconstruction of the audio signal
for quality improvement. Further, the enhancement of the carrier signal was
not addressed in the previous work due to the challenges in the handling the
impulsive nature of the carrier signal.

In this paper, we propose a novel approach to the joint dereverberation of the
envelope and carrier signals using a neural modeling framework. While using
the sub-band signals directly, the sample level de-convolution with a suitable
loss function can be a difficult design choice to learn using neural models.
Hence, we propose using an envelope-carrier decomposition of the sub-band
signals. Our rationale for the envelope-carrier decomposition based setup is
the fact the envelope information is alone used in the ASR experiments. Thus,
the ASR loss has to impact only the envelope dereverberation branch. However,
the carrier and the envelope components are part of the signal reconstruction
branch.

We develop a dual path long short term memory (DPLSTM) architecture for the
dereverberation of the temporal envelope and carrier signals. In our case, the
goal of the neural model is to perform a dereverberation of the envelope and
the carrier components of the sub-band signal. These signals have a time
profile, with varying dynamic range and properties. Further, merging all the
sub-band signals in the decomposition also brings in a frequency profile.
Thus, the design choice of the neural model, for enhancing the sub-band
envelope-carrier signals, has to learn the sequence level patterns in both the
time and frequency domains. The DPLSTM [20] is a suitable choice, as the model
is able to integrate information effectively in both the time and frequency
domains.

Following the dereverberation step, the sub-band modulation and synthesis step
generates the reconstructed audio signal. The neural enhancement and sub-band
synthesis can also be implemented as a part of the larger neural pipeline for
downstream tasks like ASR, thereby enabling the joint learning of the ASR and
dereverberation model parameters. We refer to the proposed approach as Dual
path dereverberation using Frequency domain Auto-Regressive modeling (DFAR)
and the joint end-to-end model as E2E-DFAR.

Various ASR experiments are performed on the REVERB challenge dataset [21] as
well as the VOiCES dataset [22, 23]. The key contributions from this work,
over the prior work [18], can be summarized as follows,

  * •

Proposing an analysis for dereverberation with a sub-band decomposition and
envelope-carrier demodulation.

  * •

Proposing a dual-path long short time memory model named, DPLSTM for the
dereverberation of sub-band envelope and carrier signals. This approach is
termed as DFAR.

  * •

Developing a joint learning scheme, where the ASR model and the DFAR model are
optimized in a single end-to-end framework. This model is referred to as the
E2E-DFAR.

  * •

Evaluating the proposed approaches on speech quality improvement tasks as well
as on ASR tasks on two benchmark datasets - REVERB challenge dataset and the
VOiCES dataset.

##  II Related prior work

###  II-A Enhancement and dereverberation

For speech enhancement, Xu et. al. [24] devised a mapping from noisy speech to
clean speech using a supervised neural network. In a similar manner, ideal
ratio mask based neural mappings [25] have been explored for speech separation
tasks. On the dereverberation front, Zhao et. al. proposed an LSTM model for
late reflection prediction in the spectrogram domain [26]. Han et. al [27]
developed a spectral mapping approach using the log-magnitude inputs and
Williamson et. al [10] proposed a mask-based approach for dereverberation on
the complex short-term Fourier transform. In a different line of work, speech
enhancement in the time domain was pursued by Pandey et. al [28].

The application of speech dereverberation as a pre-processing step for
downstream applications like ASR have been explored in several works (for
example, [29, 30, 31]). The recent years have seen the use of recurrent neural
network architectures for dereverberation. For example, Maas et. al [32],
utilized a recurrent neural network (RNN) to establish mapping between noise-
corrupted input features and their corresponding clean targets. Also, the use
of a context-aware recurrent neural network-based convolutional encoder-
decoder architecture was investigated by Santos et. al. [33].

Figure 1: Illustration of a 4-channel uniform QMF decomposition using a
2-stage binary QMF tree. In our work, we use $64$-channel decomposition, using
a 6-way binary tree.

###  II-B Robust multi-channel ASR

In the design of robust ASR, Generalized sidelobe canceller (GSC) [34, 35] is
a common approach. It was introduced by Li et. al in [36], where the authors
proposed a neural network-based generalized side-lobe canceller. To combine
spectral and spatial information from multiple channels using attention
layers, an end-to-end multi-channel transformer was investigated in [37]. In
another attention modelling approach, the streaming ASR model based on
monotonic chunk-wise attention was proposed by Kim et. al in [38]. Ganapathy
et. al. [4] proposed a 3-D CNN model for far-field ASR.

###  II-C Joint modeling of enhancement and ASR

The attempt proposed by Wang et. al. [39] incorporates a DNN based speech
separation model coupled with a DNN based acoustic model. The work reported by
Wu et. al. [40] explored a unification of separately trained speech
enhancement neural model and the acoustic model, where the joint model is
fine-tuned to improve the ASR performance. Here, the DNN based dereverberation
front end leverages the knowledge about reverberation time. While traditional
GSC is optimized for signal level criteria, the neural network-based GSC,
proposed by Li et. al [36], was optimized for ASR cost function.

##  III Proposed DFAR approach

###  III-A Quadrature Mirror Filter (QMF)

For the sub-band decomposition, we had the following design considerations

  * •

The decomposition approach should allow the long-term artifacts of
reverberation to be captured in the sub-band domain as a convolution,

  * •

The analysis method should allow a perfect reconstruction back to the audio
using the synthesis part, and

  * •

The sub-band components should be critically sampled for efficient computation
of the dereverberated components in a deep neural model.

The quadrature mirror filter (QMF) met all the above requirements and hence,
this work has used the QMF analysis and synthesis for speech dereverberation
task.

A quadrature mirror filter (QMF) is a filter whose magnitude response is a
mirror reflection at quadrature frequency ($\frac{\pi}{2}$) of another filter
[41]. In signal processing, the QMF filter-pairs are used for the design of
perfect reconstruction filter banks. Let $H_{0}(e^{j\Omega})$ and
$H_{1}(e^{j\Omega})$ denote low-pass and high-pass filter’s frequency domain
function, where $\Omega$ is the digital frequency. In addition to the
quadrature property ($H_{1}(e^{j\Omega})=H_{0}(e^{j(\Omega-\pi)})$), the
filters used in QMF filter-banks also satisfy the complimentary property,

$\displaystyle|H_{0}(e^{j\Omega})|^{2}+|H_{1}(e^{j\Omega})|^{2}=1.$ (1)

The design of sub-band decomposition scheme with QMF involves a series of
filtering and down-sampling operations for the analysis [42]. The synthesis is
achieved by up-sampling and filtering operations. A tree-like structure can be
formed using a recursive decomposition operation. The down-sampling process
enables a critical rate of processing, where the sum of the number of samples
in each sub-band equals the number of the samples in the full-band signal.

In this work, we use an uniform $64$-band Quadrature Mirror Filter bank (QMF)
for decomposing the input signal into $64$ uniformly spaced frequency bands.
Inspired by the audio decomposition scheme outlined in Motlicek et. al. [43],
we use a 6-level binary tree structure. The schematic of the sub-band
decomposition is shown in Fig. 1. For the implementation in a neural pipeline,
the down-sampling operation is equivalent to a stride, while the up-sampling
operation is that of un-pooling.

###  III-B Autoregressive modeling of temporal envelopes

The application of linear prediction model in the frequency domain, an
approach called frequency domain linear prediction (FDLP), enables the
modeling of the temporal envelopes of a signal with an autoregressive (AR)
model [44, 8]. The sub-band signal is transformed to the spectral domain using
a discrete cosine transform (DCT) [8], where a linear prediction model is
applied.

Let the sub-band signal be denoted as $x_{q}[n]$, where $q=1,..,Q$ denotes the
sub-band index. The analytic signal, in signal processing theory, is a complex
valued function, whose real value is the original signal while the imaginary
value is the Hilbert transform of the signal. It finds application in single
side-band amplitude modulation and quadrature filtering. Let the analytic
version of sub-band signal, $x_{q}[n]$ be denoted as, $x_{q}^{a}[n]$. The
corresponding analytic signal in the frequency domain, $X_{q}^{a}[k]$ can be
shown to be the one-sided discrete Fourier transform (DFT) [8] of the even
symmetric version of $x_{q}[n]$.

We apply linear prediction (LP) on the frequency domain signal,
$X_{q}^{a}[k]$. The corresponding LP coefficients are denoted by
$\\{b_{p}\\}_{p=0}^{m}$, where $m$ is the order of the LP. The temporal
envelope estimate of $x_{q}^{a}[n]$, is given by,

$\displaystyle e_{q}[n]=\frac{\alpha}{|\sum_{p=0}^{m}b_{p}e^{-2\pi ipn}|^{2}}$
(2)

where $\alpha$ denotes the LP gain. The envelope represents the autoregressive
model of the Hilbert envelope. In this paper, we use the Burg method [45] for
estimating the AR envelope.

The corresponding carrier (remaining residual signal), $c_{q}[n]$ is found as,

$\displaystyle c_{q}[n]=\frac{x_{q}[n]}{\sqrt{e_{q}[n]}}$ (3)

The division operation in the expression above is well defined as the envelope
given in Eq. (2) is always positive. Further, the modeling of the temporal
envelopes using the AR model ensures that the peaks of the sub-band signal in
the time-domain are well represented [46, 47].

###  III-C Effect of reverberation on envelope and carrier signals

The effect of reverberation on the time-domain speech signal can be expressed
in the form of a convolution operation,

$\displaystyle y[n]=x[n]*r[n],$ (4)

where $x[n]$ denotes the clean speech signal, $r[n]$ is the impulse response
of the room and $y[n]$, is the reverberant speech signal. The room response
function can be further split into two parts, $r[n]=r_{e}[n]+r_{l}[n]$, where
$r_{e}[n]$ and $r_{l}[n]$ are the early and late reflection components,
respectively.

Let $x_{q}[n]$, $r_{q}[n]$ and $y_{q}[n]$ denote the sub-band versions of the
clean speech, room-response function and the reverberant speech signal
respectively. Assuming an ideal band-pass filtering, it can be shown that the
analytic signal, $x_{q}^{a}[n]$, is given by [48, 8],

$\displaystyle y_{q}^{a}[n]=\frac{1}{2}[x_{q}^{a}[n]*r_{q}^{a}[n]],$ (5)

For band-pass filters with narrow band-width, the envelopes of the reverberant
speech can be approximated as [18],

$\displaystyle e_{yq}[n]\simeq\frac{1}{2}e_{xq}[n]*e_{rq}[n],$ (6)

where $e_{yq}[n]$, $e_{xq}[n]$, $e_{rq}[n]$ denote the sub-band envelopes of
reverberant speech, clean speech and room response respectively. Prior efforts
in envelope normalization focus on suppressing the linear effects of
reverberation by setting the gain of the reconstructed envelopes to unity
[49]. However, in this work, we develop neural models that can remove the non-
linear effects of reverberation. The reverberant sub-band envelope can also be
viewed an additive model [18, 50].

$\displaystyle{\color[rgb]{0,0,0}e_{yq}[n]=e_{yqe}[n]+e_{yql}[n]},$ (7)

where, $e_{yqe}[n]$ is the early reflection component (which includes the
direct path and the early reflections), while $e_{yql}[n]$ is the late
reflection part of the sub-band envelope $e_{yq}[n]$.

The key assumptions about the reverberation model of Eq. (4-6), is a long-
analysis window in the time domain. As the reverberation is a long-term
convolution effect, we highlight that the room impulse response (typically
with a T$60>400ms$) can be absorbed as a multiplication in the frequency
domain, as well as a convolution in the sub-band envelope domain, only in the
case of a long analysis window. The widely used short-time Fourier transform
(STFT) does not capture the room impulse response function directly, and hence
does not allow a convolutive modeling of the artifacts. Further, the phase
effects in STFT domain are somewhat cumbersome to model. The above mentioned
issues of STFT are also verified experimentally in Sec. V.

Figure 2: Block schematic of speech dereverberation model, the feature
extraction module and the E2E ASR model. The red arrows denote the envelopes,
$e[n]$, and the green arrows represent the carrier, $c[n]$. The entire model
can be constructed as an end-to-end neural framework.

Envelope enhancement: A neural model can be used to learn late reflection
component $e_{xql}[n]$ from the sub-band temporal envelope $e_{xq}[n]$. The
predicted late reflection component can be subtracted from the sub-band
envelope to suppress the artifacts of reverberation.

We pose the problem in the log domain to reduce the dynamic range of the
envelope magnitude. The neural model is trained with reverberant sub-band
envelopes ($log~{}(e_{xq}[n])$) as input. The model outputs the gain (in the
log domain, i.e., $log~{}\frac{e_{sq}[n]}{e_{xq}[n]}$). This gain is added in
the log-domain to generate dereverberated signal envelope
($log~{}(\hat{e}_{sq}[n]$).

Figure 3: The dual path LSTM model architecture for envelope-carrier
dereverberation. The top LSTM path models the recurrence along the time
dimension while the one on the bottom models the recurrence along the
frequency dimension.

Envelope-carrier dereverberation model: In a similar manner, the non-linear
mapping between the reverberant carrier, $c_{xq}[n]$ and clean carrier,
$c_{xq}[n]$, can be learned using a neural network. A neural model is trained
with reverberant sub-band carrier ($c_{xq}[n]$) as input and model outputs the
residual (an estimate of the late reflection component, $c_{xql}[n]$), which
when added with the reverberant carrier generates the estimate of source
signal carrier ($\hat{c}_{sq}[n]$). Instead of independent operations of
dereverberation of the envelope and the carrier, we propose to learn the
mapping between clean and reverberant versions of both the envelope and the
carrier in a joint model. The input to the neural model is the sub-band
reverberant envelope spliced with the corresponding carrier signal. The
network is trained to output the late reflection components of both the
envelope and carrier. With this approach, the model also learns the non-linear
relationships between the envelope and carrier signals for the dereverberation
task. From the model output, the estimate of the clean sub-band signal
$\hat{s}_{q}[n]$ is generated. In our implementation, the audio signal is
divided into non overlapping segments of $1$ sec. length and passed through
the envelope-carrier dereverberation model. The model is outlined in Fig. 2.

###  III-D DFAR model architecture using DPLSTM

We propose the dual path long short term model (DPLSTM) for the
dereverberation of the envelope-carrier components of the sub-band signal. Our
proposed model is inspired by dual path RNN proposed by Luo et. al [20]. The
block schematic of the DPLSTM model architecture is shown in Fig. 3. For $1$
sec. of audio sampled at $16$ kHz, the envelope
($\mathrm{\boldsymbol{E}}^{y}$) and carrier ($\mathrm{\boldsymbol{C}}^{y}$)
components of the critically sampled sub-band signals ($64$ channel QMF
decomposition) are of length $250$. The envelope/carrier signals of all the
sub-bands, for the reverberant signal ($\mathrm{\boldsymbol{Y}}$), is of size
$64\times 250$. The combined envelope-carrier input is therefore of size
$128\times 250$, which forms the input to the DPLSTM model. The DPLSTM model
outputs are also of the same size of the input, and the model is trained using
the mean squared error (MSE) loss.

The proposed DPLSTM has two paths, one LSTM path models the recurrence along
the time dimension, while the other models the recurrence along the frequency
dimension. We use two separate $3$-layer LSTM architectures for these paths.
The output dimensions are kept the same as the input dimension for each of
these paths. The frequency recurrence LSTM output is transposed and these are
concatenated in the frequency dimension. This combined output is fed to a
multi layer bi-directional LSTM, which performs recurrence over time. The
final output is split into sub-band specific envelope and carrier components.
The modulation of the envelope with the respective carrier components
generates the sub-band signals, which are passed through the QMF synthesis to
generate the full-band dereverberated signal.

###  III-E Joint learning of dereverberation model for ASR

The joint learning of the envelope-carrier dereverberation module with the E2E
ASR architecture is achieved by combining the two separate models to train a
single joint neural model. This is shown in Fig. 2. We initialize the modules
with weights obtained from the independent training of each component.
Specifically, the envelope-carrier dereverberation model is trained using MSE
loss, which is followed by a sub-band synthesis (right side half of Fig. 1).
The QMF synthesis is implemented using a 1-D CNN layer to generate the
dereverberated speech signal. Further, the E2E ASR architecture is separately
trained on the log-mel filter bank features, obtained from the dereverberated
speech. The mel-filter bank feature generation can also implemented using a
neural framework. Thus, the final model, composed of neural components from
the envelope-carrier dereverberation, sub-band synthesis, feature extraction
and ASR, can now be jointly optimized using the E2E ASR loss function. This
model is refered to as E2E-DFAR model111The implementation of the work can be
found in https://github.com/anurenjan/DFAR. The trainable components are the
DPLSTM model and the ASR model parameters, while the sub-band synthesis and
feature extraction parameters are not learnable.

##  IV Experimental setup

###  IV-A Datasets

####  IV-A1 REVERB Challenge ASR

The audio samples in REVERB challenge dataset [51] are $8$ channel recordings
with both real and simulated reverberant conditions. The real samples are
utterances from MC-WSJ-AV corpus [52], spoken by human speakers in a noisy
reverberant room. The simulated samples of the dataset are generated by
convolving six different room impulse responses with the clean WSJCAM0
recordings followed by the addition of noise at the signal-to-noise ratio
(SNR) of $20$ dB. The training data consists of $7861$ ( $\sim$ $17.5$ hours)
utterances which are obtained by convolving WSJCAM0 train data with $24$
measured RIRs. The reverberation time of the measured impulse responses range
from $0.2$ to $0.8$ sec. The training, development and evaluation data sets
consist of $92$, $15$ and $38$ speakers respectively. The development data
consists of $1663$ ($3.3$ hours) utterances and the evaluation data consists
of $2548$ ($5.4$ hours) utterances.

####  IV-A2 VOiCES Dataset

The VOiCES training set is a subset ($80$ hours) of the LibriSpeech dataset.
This set has utterances from $427$ speakers recorded in clean environments
with close-talking microphones. The development and evaluation sets are far-
field microphone recordings from diverse room dimensions, environments and
noise conditions containing $19$ and $20$ hours of speech, respectively. The
three sets namely training, development and evaluation, do not have any
overlap in terms of the speakers. The robustness of the developed models is
challenged by the mismatch that exists between the training and
development/evaluation sets. We artificially added reverberation and noise on
the $80$ hours training set, which served as the training set for all the E2E
ASR experiments on the VOiCES dataset. The development set contains $20$ hours
of distant recordings from the $200$ speakers. The evaluation data of $19$
hours consists of recordings $100$ speakers. The training set has $22741$
utterances, development set has $4318$ utterances and evaluation set has
$4600$ utterances.

TABLE I: WER (%) on the REVERB dataset for envelope/carrier, envelope-carrier
dereverberation (DFAR) and the joint E2E-DFAR model. The relative improvements
(%) pertain to the comparison of the various configurations w.r.t. the BF-
FBANK baseline system.

Model Config. | Dev | Eval
---|---|---
Real | Sim. | Avg. [Rel. Imp.] | Real | Sim | Avg. [Rel. Imp.]
BF-FBANK (baseline) | 12.8 | 8.7 | 10.8 [- - -] | 11.9 | 7.9 | 9.9 [- - -]
DCCRN[53] \+ BF-FBANK | 17.4 | 10.4 | 13.9 [-28.7] | 15.3 | 8.8 | 12.1 [-22.2]
Fullsubnet + BF-FBANK [54] | 11.8 | 7.9 | 9.9 [+8.3] | 10.5 | 7.4 | 9.0 [+9.1]
Deep non-linear filter[55] \+ BF-FBANK | 12.4 | 8.1 | 10.3 [+4.6] | 10.5 | 7.2 | 8.9 [+10.1]
Reverb. time shortening [56]\+ BF-FBANK | 11.5 | 7.6 | 9.6 [+11.1] | 10.1 | 7.6 | 8.9 [+10.1]
STFT Deverb. + BF-FBANK | 12.0 | 7.8 | 9.9 [+8.3] | 10.8 | 7.3 | 9.1 [+8.1]
Sub-band sig. Dereverb. + BF-FBANK | 13.3 | 9.8 | 11.6 [-7.4] | 12.8 | 8.6 | 10.7 [-8.1]
FDLP Env. Derevb. + BF-FBANK | 12.7 | 8.5 | 10.6 [+1.9] | 10.1 | 7.8 | 9.0 [+9.1]
FDLP Carr. Dereverb. + BF-FBANK | 11.2 | 8.3 | 9.8 [+9.3] | 10.8 | 7.6 | 9.2 [+7.1]
DFAR + BF-FBANK | 10.6 | 7.6 | 9.1 [+15.7] | 9.1 | 6.9 | 8.0 [+19.2]
E2E-DFAR | 9.4 | 6.4 | 7.9 [+26.9] | 7.3 | 5.7 | 6.5 [+34.3]

###  IV-B E2E ASR baseline system

For all the ASR experiments, we use the weighted prediction error based pre-
processing [9] and unsupervised generalized eigenvalue (GEV) beamforming [13].
The baseline features are $36$-dimensional log-mel filter bank features with
frequency range from $200$ Hz to $6500$ Hz. The ESPnet toolkit [57] is used to
perform all the end-to-end ASR experiments, with a Pytorch backend [58]. The
model architecture uses 12 conformer encoder layers with $2048$ units in the
projection layer. The $6$-layer transformer architecture with $2048$ units in
the projection layer serves as the decoder. Both connectionist temporal cost
(CTC) loss and attention based cross entropy (CE) loss are used in the
training, with CTC-weight set at $0.3$ [59]. A single layer of $1000$ LSTM
cell recurrent neural network is used for language modeling (RNN-LM). For
training the model, we use stochastic gradient descent (SGD) optimizer with a
batch size of $32$. For language model training, data is augmented from Wall
Street Journal (WSJ) corpus.

###  IV-C Performance metrics

####  IV-C1 ASR performance metrics

  * •

WER/CER (Word/Character Error Rate): The word/character error rate is given by
the ratio of number of word/character insertions, deletions and substitutions
in the system output to the total number of words/characters in the reference.

####  IV-C2 Speech quality metrics

  * •

SRMR: Speech to reverberation modulation ratio (SRMR) is a non intrusive
measure. Here, a representation is obtained using an auditory-inspired filter
bank analysis of critical band temporal envelopes of the signal. The
modulation spectral information is used to get an adaptive measure termed as
speech to reverberation modulation energy ratio[60, 61]. A higher value
indicates an improved quality of the given speech signal.

  * •

MOS (Mean Opinion Score): To evaluate the performance of dereverberation
algorithms, subjective quality and intelligibility measurement methods are
needed. The most widely used subjective method is the ITU-T standard [62],
where a panel of listeners are asked to rate the quality/intelligibility of
the audio.

##  V Experiments and results

The baseline features are the beamformed log-mel filter-bank energy features
(denoted as BF-FBANK).

###  V-A REVERB Challenge ASR

TABLE II: Comparison of the results with other works reported on the REVERB
challenge dataset.

System | Eval-sim. | Eval-real | Avg.
---|---|---|---
Subramanian et. al. [63] | 6.6 | 10.6 | 8.6
Heymann et. al. [64] | - | 10.8 | -
Fujita et. al. [65] | 4.9 | 9.8 | 7.4
Purushothaman et. al. [18] | 7.1 | 12.1 | 9.6
Zhang et. al. [66] | - | 10.0 | -
This work | 5.7 | 7.3 | 6.5

The word error rates (WER) for the dereverberation experiments are shown in
Table I. Note that, all the experiments use the same input features (log-mel
filter bank features) along with the same E2E ASR architecture (conformer
encoder and transformer decoder). The only difference between the various
rows, reported in Table I, is the dereverberation pre-processing applied on
the raw audio waveform. All the dereverberation experiments use the DPLSTM
architecture described in Sec. III.

TABLE III: WER (%) in REVERB dataset for different architectures for the
dereverberation model.

Model Config. | Dev | Eval
---|---|---
Real | Sim | Avg | Real | Sim | Avg
Baseline | 12.8 | 8.7 | 10.8 | 11.9 | 7.9 | 9.9
CLSTM | 14.5 | 9.7 | 12.1 | 12.4 | 9.1 | 10.8
4-layer LSTM | 12.5 | 8.0 | 10.3 | 10.1 | 7.1 | 8.9
DPLSTM | 10.6 | 7.6 | 9.1 | 9.1 | 6.9 | 8.0

####  V-A1 Various dereverberation configurations

In Table I, the first row is the baseline result with the beamformed audio
(unsupervised GEV beamforming [13] and weighted prediction error (WPE)
processing [9].

The next set of rows compare several prior works.

  * •

Fullsubnet - A full-band and sub-band fusion model for speech enhancement
[54].

  * •

DCCRN - Deep complex convolution recurrent neural network model for speech
enhancement [53].

  * •

Deep non-linear filter for multi-channel audio [55]

  * •

Reverberation time shortening [56]

The prior works are trained on the same data settings as used in the DFAR
framework. All the prior works, except DCCRN (which is not designed for ASR),
improve the baseline system in range of $8$-$11$% in terms of relative WER.
However, the proposed DFAR/E2E-DFAR approach is observed to provide the best
WER, with relative improvement in WER of $19$/$34$% on the evaluation data.

In Table I, we have also performed two ASR experiments - i) using STFT inputs
(log magnitude), and ii) using the sub-band signal directly without the
envelope-carrier decomposition. Both these experiments, use the DPLSTM
dereverberation model proposed in this work. As seen in Table I, the
dereverberation on the STFT magnitude component improves the ASR systems
significantly over the baseline, while the dereverberation on the sub-band
signal directly is not effective. However, the STFT approach is also seen to
be inferior to the DFAR approach where the envelope-carrier dereverberation is
performed.

The fourth set of rows corresponds to the WER results with envelope/carrier
based dereverberation alone. The relative improvements of $2-9$% are seen here
compared to the baseline BF-FBANK. Separately, with dereverberation based on
the carrier signal alone, a similar improvement is achieved. Further, the
dereverberation of the temporal envelope and carrier components in a combined
fashion using the DPLSTM model improves the ASR results over the separate
dereverberation of envelope/carrier components. Here, average relative
improvements of $16$% and $19$% are seen in the development set and evaluation
set respectively, over the BF-FBANK baseline system for the DFAR approach.

The final row in Table I reports the results using the joint learning of the
dereverberation network and the E2E ASR model. The E2E-DFAR is initialized
using the dereverberation model and the E2E model trained separately. The
proposed E2E-DFAR model yields average relative improvements of $27$% and
$34$% on the development set and evaluation set respectively over the baseline
system. The joint training is also shown to improve over the set up of having
separate networks for dereverberation and E2E ASR. While the DFAR model is
trained only on simulated reverberation conditions, the WER improvement in
real condition is seen to be more pronounced than those observed in the
simulated data. This indicates that the model can generalize well to unseen
reverberation conditions in the real-world.

TABLE IV: WER (%) in REVERB dataset for Hyper parameter $\lambda$, in
$MSE~{}~{}loss=\lambda\times env.~{}loss+(1-\lambda)\times carr.~{}loss$.

Parameter $\lambda$ | Dev | Eval
---|---|---
Real | Simu | Avg | Real | Simu | Avg
0 | 12 | 8.2 | 10.1 | 10.4 | 7.5 | 9.0
0.2 | 11.9 | 8.6 | 10.3 | 10.7 | 7.7 | 9.2
0.4 | 11.6 | 8.2 | 9.9 | 10.1 | 7.2 | 8.7
0.5 | 11.3 | 7.2 | 9.3 | 9.7 | 6.5 | 8.1
0.6 | 10.6 | 7.6 | 9.1 | 9.1 | 6.9 | 8.0
0.8 | 13.1 | 8.7 | 10.9 | 10.9 | 7.9 | 9.4
1 | 13.5 | 8.0 | 10.8 | 10.4 | 6.9 | 8.7

TABLE V: Performance (WER %) on the VOiCES dataset.

Model Config. | Dev | Eval
---|---|---
FBANK (baseline) | 40.3 | 50.8
\+ Env. derevb. | 38.4 | 48.6
\+ Env.-carr. derevb. (DFAR) | 37.1 | 45.4
\+ E2E-DFAR | 36.4 | 44.7

TABLE VI: SRMR values on the REVERB dataset for various signal enhancement
strategies.

Signal | SRMR |
---|---|---
| Dev. (Real) | Dev. (Sim.) | Eval. (Real) | Eval. (Sim.) | REVB. (Train) |
| Unsupervised GEV beamforming [13]
---
5.18 | 4.1 | 4.58 | 4.67 | 4.23 |
|  \+ WPE [9]
---
5.35 | 4.2 | 4.61 | 4.75 | 4.48 |
|  \+ DCCRN[53]
---
5.43 | 4.37 | 4.63 | 4.94 | 4.67 |
|  + Fullsubnet[54]
---
5.36 | 4.32 | 4.64 | 4.97 | 4.63 |
|  \+ Deep Non-Linear Filters[55]
---
5.51 | 4.22 | 4.64 | 5.02 | 4.61 |
|  \+ Reverberation Time Shortening Target[56]
---
5.49 | 4.57 | 4.62 | 5.2 | 4.58 |
|  \+ STFT Mag. + DPLSTM
---
5.44 | 4.33 | 4.64 | 4.94 | 4.6 |
|  \+ Sub-band signal + DPLSTM
---
5.45 | 4.28 | 4.61 | 4.87 | 4.63 |
|  \+ env. derevb. (this work)
---
4.62 | 3.83 | 4.12 | 4.25 | 4.11 |
|  \+ crr. derevb. (this work)
---
5.52 | 4.46 | 4.69 | 5.27 | 4.77 |
|  \+ env. & crr. derevb. [DFAR] (this work)
---
5.52 | 4.47 | 4.69 | 5.27 | 4.77 |

TABLE VII: MOS values in REVERB dataset for envelope and carrier based
enhancements.

| ET Real - near | ET Real - far | ET Simu - near | ET Simu - far
---|---|---|---|---
Baseline - GEV [13] \+ WPE [9] | 3.78 | 3.65 | 3.74 | 4.12
\+ env.-carr. derevb. [DFAR] (this work) | 3.98 | 3.67 | 4.01 | 4.40

####  V-A2 Comparison with prior works

The comparison of the results from prior works reported on the REVERB
challenge dataset is given in Table II. The Table includes results from end-
to-end ASR systems [63, 66, 65] as well as the joint enhancement and ASR
modeling work reported in [64]. We also compare with our prior work reported
in [18].  Specifically, many of the prior works compared in Table II are based
on STFT based enhancement. The work reported in Subramanian et. al. [63], used
a neural beamforming approach in the STFT domain, while the efforts described
in Heymann et. al. [64], used a long-short term memory network for mask
estimation in power spectral domain (PSD). The dynamic convolution method
proposed in Fujita et. al. [65] used deconvolution of log-mel spectrogram
features. Similar to the proposed work, all these efforts have also used the
E2E ASR model training. As seen in Table II, the proposed work improves over
these prior works considered here, further highlighting the benefits of the
dereverberation in the sub-band time domain using long-term envelope-carrier
based DPLSTM models.

####  V-A3 Dereverberation model architecture

The ASR experiments on the REVERB challenge dataset, pertaining to the choice
of different model architectures used in the dereverberation model, are listed
in Table III. We have experimented with convolutional LSTM (CLSTM) [50] and
time-domain LSTM (4-layer LSTM) architecture [67] in addition to the DPLSTM
approach. As seen here, the Dual-path recurrence based DPLSTM gives the best
word error rate in comparison with the other LSTM neural architectures
considered. This may be attributed to the joint time-frequency recurrence
performed to the other approaches which perform only time domain recurrence.

####  V-A4 Dereverberation loss function

The MSE loss function used in the DPLSTM model training consists of a
combination of loss values from the envelope and the carrier components. We
experimented with the hyper parameter, $\lambda$, which controls the
proportion of envelope based loss and carrier based loss in the total loss
($Total~{}loss=\lambda\times env.~{}loss+(1-\lambda)\times carr.~{}loss$). The
ASR results for the various choices of the hyper parameter $\lambda$ are shown
in Table IV. Empirically, the value of $\lambda=0.6$ gives the best WER on the
REVERB challenge dataset. Further, the choice of $\lambda=1$ or $\lambda=0$,
corresponding to envelope/carrier only dereverberation, are inferior to other
choices of $\lambda$, indicating that the joint dereverberation of the
envelope and carrier components is beneficial.

###  V-B VOiCES ASR

The ASR setup used in the VOiCES dataset followed the ESPnet recipe with the
conformer encoder and a transformer decoder. The rest of the model parameters
and hyper-parameters are kept similar to the ones in the REVERB challenge
dataset. The WER results on the VOiCES dataset are given in Table V. The
dereverberation of the envelope alone provides an absolute improvement of
$1.9$% and $2.2$% on the development and evaluation data respectively,
compared to the FBANK baseline system. The dereverberation based on envelope-
carrier modeling further improves the results. An absolute improvement of
$3.3$%/$5.4$% on the development/evaluation data is achieved, compared to the
FBANK baseline. Further, the joint training on envelope-carrier
dereverberation network with the ASR model improves the WER results. We
observe relative improvements of $10$% and $12$% on the development and
evaluation data respectively .

###  V-C Speech quality evaluation

A comparison of the SRMR values for different dereverberation approaches is
reported in Table VI. Here, we compare the baseline unsupervised GEV
beamforming [13] and weighted prediction error (WPE) [9] with various
strategies for beamforming. The deep complex convolutional recurrent network
(DCCRN) based speech enhancement [53] is also implemented on the REVERB
dataset, and these results are reported in Table VI. While the envelope based
dereverberation did not improve the SRMR values, the carrier based
dereverberation is shown to improve the SRMR results. Further, the DFAR model
also achieves similar improvements in SRMR for all the conditions over the
baseline approach (GEV+WPE) and the DCCRN approach.

We conducted a subjective evaluation to further assess the performance of the
dereverberation method. The subjects were asked to rate the quality of the
audio on a scale of $1$ to $5$, $1$ being poor and $5$ being excellent. The
subjects listened to the audio in a relatively quiet room with a high quality
Sennheiser headset. We perform the A-B listening test, where the two versions
of the same audio file were played, the first one with GEV + WPE
dereverberation and the second one with the proposed dereverberation approach.
We chose $20$ audio samples, from four different conditions (real and
simulated data and from near and far rooms) for this evaluation and recruited
$20$ subjects.

The subjective results are shown in Table VII. As seen, the proposed speech
dereverberation scheme shows improvement in subjective MOS scores for all the
conditions considered. The subjective results validate the signal quality
improvements observed in the SRMR values (Table VI).

##  VI Conclusion

In this paper, we propose a speech dereverberation model using frequency
domain linear prediction based sub-band envelope-carrier decomposition. The
sub-band envelope and carrier components are processed through a
dereverberation network. A novel neural architecture, based on dual path
recurrence, is proposed for dereverberation. Using the joint learning of the
neural speech dereverberation module and the E2E ASR model, we perform several
speech recognition experiments on the REVERB challenge dataset as well as on
the VOiCES dataset. These results show that the proposed approach improves
over the state of art E2E ASR systems based on mel filterbank features.

The dereverberation approach proposed in this paper also reconstructs the
audio signal, which makes it useful for audio quality improvement applications
as well as other speech processing systems in addition to the ASR system. We
have further evaluated the reconstruction quality subjectively and objectively
on the REVERB challenge dataset. The quality measurements show that the
proposed speech dereverberation method improves speech quality over the
baseline framework of weighted prediction error. The ablation studies on
various architecture choices provides justification for the choice of the
DPLSTM network architecture. Given that the proposed model allows the
reconstruction of the audio signal, it can be used in conjunction with self-
supervised neural approaches for representation learning of speech as well.
This will form part of our future investigation.

## References

  * [1] R. Haeb-Umbach, J. Heymann, L. Drude, S. Watanabe, M. Delcroix, and T. Nakatani, “Far-field automatic speech recognition,” _Proceedings of the IEEE_ , vol. 109, no. 2, pp. 124–148, 2020.
  * [2] T. Hain, L. Burget, J. Dines, P. N. Garner, F. Grézl, A. El Hannani, M. Huijbregts, M. Karafiat, M. Lincoln, and V. Wan, “Transcribing meetings with the AMIDA systems,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 2, pp. 486–498, 2012.
  * [3] V. Peddinti, Y. Wang, D. Povey, and S. Khudanpur, “Low latency acoustic modeling using temporal convolution and LSTMs,” _IEEE Signal Processing Letters_ , vol. 25, no. 3, pp. 373–377, 2018.
  * [4] S. Ganapathy and V. Peddinti, “3-d CNN models for far-field multi-channel speech recognition,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 5499–5503.
  * [5] A. Gusev, V. Volokhov, T. Andzhukaev, S. Novoselov, G. Lavrentyeva, M. Volkova, A. Gazizullina, A. Shulipa, A. Gorlanov, A. Avdeeva _et al._ , “Deep speaker embeddings for far-field speaker recognition on short utterances,” _arXiv preprint arXiv:2002.06033_ , 2020.
  * [6] Q. Jin, T. Schultz, and A. Waibel, “Far-field speaker recognition,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, no. 7, pp. 2023–2032, 2007.
  * [7] T. Yoshioka _et al._ , “Making machines understand us in reverberant rooms: Robustness against reverberation for automatic speech recognition,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 114–126, 2012.
  * [8] S. Ganapathy, “Signal analysis using autoregressive models of amplitude modulation,” Ph.D. dissertation, Johns Hopkins University, 2012.
  * [9] T. Nakatani, T. Yoshioka, K. Kinoshita, M. Miyoshi, and B.-H. Juang, “Speech dereverberation based on variance-normalized delayed linear prediction,” _IEEE TASLP_ , vol. 18, no. 7, pp. 1717–1731, 2010.
  * [10] D. S. Williamson and D. Wang, “Time-frequency masking in the complex domain for speech dereverberation and denoising,” _IEEE/ACM transactions on Audio, Speech, and Language processing_ , vol. 25, no. 7, pp. 1492–1501, 2017.
  * [11] X. Anguera, C. Wooters, and J. Hernando, “Acoustic beamforming for speaker diarization of meetings,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, no. 7, pp. 2011–2022, 2007.
  * [12] E. Warsitz and R. Haeb-Umbach, “Blind acoustic beamforming based on generalized eigenvalue decomposition,” _IEEE Transactions on audio, speech, and language processing_ , vol. 15, no. 5, pp. 1529–1539, 2007.
  * [13] R. Kumar, A. Sreeram, A. Purushothaman, and S. Ganapathy, “Unsupervised neural mask estimator for generalized eigen-value beamforming based ASR,” in _IEEE ICASSP_ , 2020, pp. 7494–7498.
  * [14] M. L. Seltzer, D. Yu, and Y. Wang, “An investigation of deep neural networks for noise robust speech recognition,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2013, pp. 7398–7402.
  * [15] V. Peddinti, Y. Wang, D. Povey, and S. Khudanpur, “Low latency acoustic modeling using temporal convolution and LSTMs,” _IEEE Signal Processing Letters, vol. 25, issue 3, pp. 373-377_ , 2017.
  * [16] R. Talmon, I. Cohen, and S. Gannot, “Relative transfer function identification using convolutive transfer function approximation,” _IEEE Transactions on audio, speech, and language processing_ , vol. 17, no. 4, pp. 546–555, 2009.
  * [17] Y. Avargel and I. Cohen, “System identification in the short-time fourier transform domain with crossband filtering,” _IEEE transactions on Audio, Speech, and Language processing_ , vol. 15, no. 4, pp. 1305–1319, 2007.
  * [18] A. Purushothaman, A. Sreeram, R. Kumar, and S. Ganapathy, “Dereverberation of autoregressive envelopes for far-field speech recognition,” _Computer Speech & Language_, vol. 72, p. 101277, 2022.
  * [19] A. Purushothaman, A. Sreeram, and S. Ganapathy, “3-D acoustic modeling for far-field multi-channel speech recognition,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6964–6968.
  * [20] Y. Luo, Z. Chen, and T. Yoshioka, “Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 46–50.
  * [21] K. Kinoshita _et al._ , “The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech,” in _IEEE WASPAA_ , 2013, pp. 1–4.
  * [22] C. Richey, M. Barrios _et al._ , “VOiCES obscured in complex environmental settings (voices) corpus,” _arXiv preprint arXiv:1804.05053_ , 2018.
  * [23] M. Nandwana, J. Van Hout, M. McLaren, C. Richey, A. Lawson, and M. Barrios, “The voices from a distance challenge 2019 evaluation plan,” _arXiv preprint arXiv:1902.10828_ , 2019.
  * [24] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “A regression approach to speech enhancement based on deep neural networks,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 1, pp. 7–19, 2014.
  * [25] D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 10, pp. 1702–1726, 2018.
  * [26] Y. Zhao, D. Wang, B. Xu, and T. Zhang, “Late reverberation suppression using recurrent neural networks with long short-term memory,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5434–5438.
  * [27] K. Han, Y. Wang, and D. Wang, “Learning spectral mapping for speech dereverberation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2014, pp. 4628–4632.
  * [28] A. Pandey and D. Wang, “A new framework for CNN-based speech enhancement in the time domain,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 7, pp. 1179–1188, 2019.
  * [29] M. Wöllmer, Z. Zhang, F. Weninger, B. Schuller, and G. Rigoll, “Feature enhancement by bidirectional LSTM networks for conversational speech recognition in highly non-stationary noise,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2013, pp. 6822–6826.
  * [30] Z. Chen, S. Watanabe, H. Erdogan, and J. R. Hershey, “Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [31] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in _International Conference on Latent Variable Analysis and Signal Separation_. Springer, 2015, pp. 91–99.
  * [32] A. L. Maas, T. M. O’Neil, A. Y. Hannun, and A. Y. Ng, “Recurrent neural network feature enhancement: The 2nd chime challenge,” in _Proceedings The 2nd CHiME Workshop on Machine Listening in Multisource Environments held in conjunction with ICASSP_ , 2013, pp. 79–80.
  * [33] J. F. Santos and T. H. Falk, “Speech dereverberation with context-aware recurrent neural networks,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 7, pp. 1236–1246, 2018.
  * [34] L. Griffiths and C. Jim, “An alternative approach to linearly constrained adaptive beamforming,” _IEEE Transactions on antennas and propagation_ , vol. 30, no. 1, pp. 27–34, 1982.
  * [35] S. Gannot, D. Burshtein, and E. Weinstein, “Signal enhancement using beamforming and nonstationarity with applications to speech,” _IEEE Transactions on Signal Processing_ , vol. 49, no. 8, pp. 1614–1626, 2001.
  * [36] G. Li, S. Liang, S. Nie, W. Liu, and Z. Yang, “Deep neural network-based generalized sidelobe canceller for dual-channel far-field speech recognition,” _Neural Networks_ , vol. 141, pp. 225–237, 2021.
  * [37] F.-J. Chang, M. Radfar, A. Mouchtaris, B. King, and S. Kunzmann, “End-to-end multi-channel transformer for speech recognition,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 5884–5888.
  * [38] C. Kim, A. Garg, D. Gowda, S. Mun, and C. Han, “Streaming end-to-end speech recognition with jointly trained neural feature enhancement,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6773–6777.
  * [39] Z.-Q. Wang and D. Wang, “A joint training framework for robust automatic speech recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 4, pp. 796–806, 2016.
  * [40] B. Wu, K. Li, F. Ge, Z. Huang, M. Yang, S. M. Siniscalchi, and C. Lee, “An end-to-end deep learning approach to simultaneous speech dereverberation and acoustic modeling for robust speech recognition,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 8, pp. 1289–1300, 2017.
  * [41] P. Vaidyanathan, “Quadrature mirror filter banks, m-band extensions and perfect-reconstruction techniques,” _IEEE Assp Magazine_ , vol. 4, no. 3, pp. 4–20, 1987.
  * [42] P. P. Vaidyanathan, _Multirate systems and filter banks_. Pearson Education India, 2006.
  * [43] P. Motlicek, S. Ganapathy, H. Hermansky, and H. Garudadri, “Scalable wide-band audio codec based on frequency domain linear prediction,” IDIAP, Tech. Rep., 2007.
  * [44] M. Athineos and D. P. Ellis, “Frequency-domain linear prediction for temporal features,” 2003.
  * [45] J. P. Burg, _Maximum entropy spectral analysis._ Stanford University, 1975.
  * [46] S. Ganapathy, P. Motlicek, and H. Hermansky, “Autoregressive models of amplitude modulations in audio compression,” _IEEE transactions on audio, speech, and language processing_ , vol. 18, no. 6, pp. 1624–1631, 2009.
  * [47] S. Ganapathy, S. H. Mallidi, and H. Hermansky, “Robust feature extraction using modulation filtering of autoregressive models,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 22, no. 8, pp. 1285–1295, 2014.
  * [48] S. Thomas, S. Ganapathy, and H. Hermansky, “Recognition of reverberant speech using frequency domain linear prediction,” _IEEE Signal Processing Letters_ , vol. 15, pp. 681–684, 2008.
  * [49] S. Ganapathy, J. Pelecanos, and M. K. Omar, “Feature normalization for speaker verification in room reverberation,” in _2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2011, pp. 4836–4839.
  * [50] R. Kumar, A. Purushothaman, A. Sreeram, and S. Ganapathy, “End-to-end speech recognition with joint dereverberation of sub-band autoregressive envelopes,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 6057–6061.
  * [51] K. Kinoshita, M. Delcroix, S. Gannot, E. A. Habets, R. Haeb-Umbach, W. Kellermann, V. Leutnant, R. Maas, T. Nakatani, B. Raj _et al._ , “A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2016, no. 1, p. 7, 2016.
  * [52] M. Lincoln, I. McCowan, J. Vepa, and H. K. Maganti, “The multi-channel wall street journal audio visual corpus (MC-WSJ-AV): Specification and initial experiments,” in _IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2005, pp. 357–362.
  * [53] Y. Hu, Y. Liu, S. Lv, M. Xing, S. Zhang, Y. Fu, J. Wu, B. Zhang, and L. Xie, “DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement,” _Proceedings of Interspeech_ , 2020.
  * [54] X. Hao, X. Su, R. Horaud, and X. Li, “Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6633–6637.
  * [55] K. Tesch and T. Gerkmann, “Insights into deep non-linear filters for improved multi-channel speech enhancement,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 563–575, 2022.
  * [56] R. Zhou, W. Zhu, and X. Li, “Speech dereverberation with a reverberation time shortening target,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [57] S. Watanabe, T. Hori _et al._ , “ESPNet: End-to-end speech processing toolkit,” _arXiv preprint arXiv:1804.00015_ , 2018.
  * [58] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in _NIPS-W_ , 2017.
  * [59] S. Karita, N. Chen _et al._ , “A comparative study on transformer vs RNN in speech applications,” in _IEEE ASRU_ , 2019, pp. 449–456.
  * [60] T. H. Falk, C. Zheng, and W.-Y. Chan, “A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 18, no. 7, pp. 1766–1774, 2010.
  * [61] J. F. Santos, M. Senoussaoui, and T. H. Falk, “An improved non-intrusive intelligibility metric for noisy and reverberant speech,” in _2014 14th International Workshop on Acoustic Signal Enhancement (IWAENC)_ , 2014, pp. 55–59.
  * [62] P. RECOMMENDATION, “Itu-tp. 808,” 2018.
  * [63] A. Subramanian, X. Wang, S. Watanabe, T. Taniguchi, D. Tran, and Y. Fujita, “An investigation of end-to-end multichannel speech recognition for reverberant and mismatch conditions,” _arXiv preprint arXiv:1904.09049_ , 2019.
  * [64] J. Heymann, L. Drude, R. Haeb-Umbach, K. Kinoshita, and T. Nakatani, “Joint optimization of neural network-based wpe dereverberation and acoustic model for robust online ASR.” IEEE ICASSP, 2019, pp. 6655–6659.
  * [65] Y. Fujita, A. Subramanian, M. Omachi, and S. Watanabe, “Attention-based asr with lightweight and dynamic convolutions,” in _ICASSP_. IEEE, 2020, pp. 7034–7038.
  * [66] W. Zhang, A. Subramanian, X. Chang, S. Watanabe, and Y. Qian, “End-to-end far-field speech recognition with unified dereverberation and beamforming,” _arXiv preprint arXiv:2005.10479_ , 2020.
  * [67] M. Mimura, S. Sakai, and T. Kawahara, “Speech dereverberation using long short-term memory,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.

|  Anurenjan Purushothaman is associated with the Learning and Extraction of
Acoustic Patterns (LEAP) lab, Department of Electrical Engineering, Indian
Institute of Science, Bangalore, India, 56001, where he is pursuing his PhD
under the guidance of Prof. Sriram Ganapathy. He is also associated with
College of Engineering, Trivandrum and Government Engineering College, Idukki,
where he is an Assistant Professor in the Department of Electronics &
Communication. He received the Bachelor’s degree in Electronics &
Communication Engineering from Govt. Engineering College, Barton Hill,
Trivandrum in 2006, the Master’s degree in Signal Processing from College of
Engineering, Trivandrum in 2008. In March 2008 he joined the Department of
ECE, College of Engineering, Trivandrum as Assistant Professor.
---|---
|  Debottam Dutta is currently a PhD student at the Signals & Inference
Research Group (SiNRG) at University of Illinois at Urbana-Champaign (UIUC).
Prior to joining UIUC, he worked as a Senior Research Fellow for a year at the
Learning and Extraction of Acoustic Patterns (LEAP) lab, Indian Institute of
Science, Bangalore. He obtained his Master of Technology degree in Signal
Processing from Indian Institute of Science, Bangalore in 2021 and Bachelor’s
degree in Electronics and Communication Engineering from National Institute of
Technology, Silchar in 2018.
---|---
|  Rohit Kumar is currently associated with the Laboratory for Computational
Audio Perception (LCAP) at Johns Hopkins University, Baltimore, Maryland. He
is pursuing his PhD under the guidance of Prof. Mounya Elhilali. Rohit earned
his Bachelor’s degree in Electronics & Communication Engineering from Delhi
Technological University (formerly Delhi College of Engineering) in 2017 and
his Master’s degree in Signal Processing from the Indian Institute of Science,
Bangalore, India, in 2020, under the supervision of Prof. Sriram Ganapathy.
Prior to his doctoral studies, he worked as a research fellow in the Learning
and Extraction of Acoustic Patterns (LEAP) lab at the Indian Institute of
Science, Bangalore, India.
---|---
|  Sriram Ganapathy is an Associate Professor at the Electrical Engineering,
Indian Institute of Science, Bangalore, where he heads the activities of the
learning and extraction of acoustic patterns (LEAP) lab. He is also associated
with the Google Research India, Bangalore. Prior to joining the Indian
Institute of Science, he was a research staff member at the IBM Watson
Research Center, Yorktown Heights. He received his Doctor of Philosophy from
the Center for Language and Speech Processing, Johns Hopkins University. He
obtained his Bachelor of Technology from College of Engineering, Trivandrum,
India and Master of Engineering from the Indian Institute of Science,
Bangalore. He has also worked as a Research Assistant in Idiap Research
Institute, Switzerland from 2006 to 2008. At the LEAP lab, his research
interests include signal processing, machine learning methodologies for speech
and speaker recognition and auditory neuro-science. He is a subject editor for
the Speech Communications journal and a senior member of the IEEE.
---|---