# Separate and Reconstruct:
Asymmetric Encoder-Decoder for Speech Separation

Ui-Hyeop Shin Sangyoun Lee Taehan Kim Hyung-Min Park
Department of Electronic Engineering, Sogang University, Seoul, Republic of
Korea
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Since the recent success of a time-domain speech separation, further
improvements have been made by expanding the length and channel of a feature
sequence to increase the amount of computation. When temporally expanded to a
long sequence, the feature is segmented into chunks as a dual-path model in
most studies of speech separation. In particular, it is common for the process
of separating features corresponding to each speaker to be located in the
final stage of the network. However, it is more advantageous and intuitive to
proactively expand the feature sequence to include the number of speakers as
an extra dimension. In this paper, we present an asymmetric strategy in which
the encoder and decoder are partitioned to perform distinct processing in
separation tasks. The encoder analyzes features, and the output of the encoder
is split into the number of speakers to be separated. The separated sequences
are then reconstructed by the weight-shared decoder, as Siamese network, in
addition to cross-speaker processing. By using the Siamese network in the
decoder, without using speaker information, the network directly learns to
discriminate the features using a separation objective. With a common split
layer, intermediate encoder features for skip connections are also split for
the reconstruction decoder based on the U-Net structure. In addition, instead
of segmenting the feature sequence into chunks and processing as dual-path, we
design global and local Transformer blocks to directly process long sequences.
The experimental results demonstrated that this separation-and-reconstruction
framework is effective and that the combination of proposed global and local
Transformer can sufficiently replace the role of inter- and intra-chunk
processing in dual-path structure. Finally, the presented model including both
of these achieved state-of-the-art performance with much less computation than
before in various benchmark datasets.

##  1 Introduction

As a solution to the well-known cocktail party problem [12, 3], single channel
speech separation [26] has been improved since the introduction of time-domain
audio separation network (TasNet) [43, 44], which applies separation masks in
the latent space instead of the short-time Fourier transform (STFT) domain.
Meanwhile, some recent works [55, 37] have shown that direct estimation of an
output representation is more effective than mask estimation, as shown in
Figure 1(a). In particular, experimental results have shown that TasNet
employing a convolution-based audio encoder/decoder performs better when the
kernel length of the audio encoder is shortened [44, 42]. Therefore, the
problem of TasNet has come down to how to model a long sequence effectively.
In fact, minimizing the suppression of the sequence could be beneficial
because the separation process requires generating more output than the input
received.

As a solution to modeling long sequences, DPRNN [42] was proposed using a dual
path model, in which it segments long sequences into chunks of several frames
and processes the chunks independently followed by aggregating the information
across the chunks to model the local and global contexts. As a result, despite
an increase in computational loads, due to promising performances in modeling
long sequences, many TasNet-based approaches have adopted the dual-path model
and repeatedly achieved state-of-the-art performances in monaural speech
separation [9, 62, 34, 35, 86, 53, 56, 47]. Meanwhile, some studies have
tackled the high computational complexity of long sequences in the time domain
approach and proposed using multi-scaled sequence models based on the
recurrent or stacked U-Net structure [66, 28, 39]. They reduced the
computations to some extent, however, they still could not show competitive
performance compared to the dual-path method.

Typically, in most studies for speech separation, the process of separating a
feature sequence for the number of speakers is placed in the final stage of
the network, as illustrated in Figure 1(b). This separation method is
inherently inefficient because a single feature sequence has to include the
features of all the speakers to be separated. Therefore, to improve the
separation performance with the late split method, the feature sequence needs
to be expanded in either the channel, temporal, or both dimensions to avoid
information bottlenecks. Instead, it is beneficial and intuitive to expand the
feature sequence to a dimension corresponding to the number of speakers in
advance, considering them collectively. Therefore, we propose dividing the
features in the middle of the separator, as illustrated in Figure 1(c), and
adopting an asymmetric strategy where the encoder and decoder are partitioned
to perform distinct processing in the separation tasks. The encoder analyzes a
single feature sequence before the split layer, similar to the conventional
separator. Once the feature sequence is separated into multiple sequences, the
decoder focuses on capturing discriminative characteristics between features
using weight-sharing blocks, as siamese structures [4, 32, 10]. This early
split with shared decoder (ESSD) structure can effectively learn the distinct
characteristics of the different speakers in the speech mixture. Additionally,
the cross-speaker block can be introduced in the decoder for the information
interaction between the sequences, as in [13, 37]. The similar concept of two
stages is also commonly utilized in multi-channel speech processing, where
coarse separation is achieved by spatial beamforming [27, 79, 21], and
additional post-enhancement is commonly applied [74, 8, 75, 38].

(a)

(b)

(c)

Figure 1: Block diagrams of (a) TasNet and separator designs of the (b)
conventional and (c) proposed networks. The proposed network consists of
separation encoder and reconstruction decoder based on weight sharing. After
an encoder, separated features are independently processed by a decoder
network.

Consequently, we present the Separation-and-Reconstruction Transformer
(SepReformer) for more efficient time-domain separation. Based on the ESSD
framework, this model employs an asymmetric encoder-decoder structure with
skip connections based on a temporally multi-scaled sequence model. The
encoder processes a single feature at different temporal resolutions, and each
intermediate feature is used for skip connection. The decoder then gradually
reconstructs fine-grained information from the temporal bottleneck features,
focusing on the discriminative characteristics of separated speech with
auxiliary loss. To achieve this, the Siamese decoder is trained to
discriminate between the features separated by the encoder. Using a common
split layer, we also split intermediate features in the encoder, providing
skip connections for each source to the reconstruction decoder. In addition,
an cross-speaker block is utilized in the decoder to facilitate information
interaction between sequences, as described in [13, 37]. Furthermore, we
design unit blocks for both global and local processing, integrating them
effectively to replace the dual-path model and directly process long sequences
without chunking. The experimental results demonstrated that the ESSD is
effective especially with a small network. Also, simply comprising the
separator network with the proposed global and local Transformer blocks can
sufficiently replace the inter- and intra-chunk processing in dual-path
structure, suggesting significant effectiveness for long feature sequence
processing in speech separation. As a consequence, the proposed SepReformer
that includes both of these achieved state-of-the-art (SOTA) performance with
much less computation than before in various benchmark datasets.

##  2 Related Works

#### TasNet

Conventional source separation has been performed in the STFT domain [26, 33,
11, 41]. In the time-frequency representation, a separator is modeled to
estimate mask values or direct output representations. Then, the inverse STFT
(iSTFT) is operated for the output representations to obtain separated signals
[26, 33]. On the other hand, TasNet [43] replaces STFT with a 1D convolutional
layer. Based on the encoder representation, mask values or direct output
representations [61, 55, 37] are obtained in the separator. Then, the output
representations are decoded by the audio decoder of 1D transposed convolution
instead of iSTFT. Also, unlike the STFT, the convolutional encoder turns out
to work well in a much shorter kernel size. Therefore, TasNet requires the
separator to process the much longer sequences. Therefore, instead of an LSTM-
based separator [43], Conv-TasNet [44] is proposed based on a temporal
convolutional network (TCN) [67, 36] to design a separator for longer
sequence, showing impressive separation results.

#### Dual-path model for long sequence

After Conv-TasNet, the dual-path model is extensively employed to handle long
sequences. In the dual-path model, the sequence is segmented into smaller
chunks, and the sequence is processed alternately as intra-chunk and inter-
chunk, effectively interleaving between local and global contexts. This dual-
path strategy has shown promising performance in TasNet and has been
repeatedly adopted [9, 62, 34, 35, 86, 53, 56, 47, 49, 30]. Especially, it is
shown that, compared to various efficient attention mechanisms [71, 2, 31],
using the dual-path model with the original self-attention mechanism of
Transformer [68] is effective for long sequence [63] in speech separation.
However, when modeling with the dual-path method, the segmentation process may
increase the amount of computation by twice when the overlap between adjacent
chunks is set to 50%. The inter-chunk blocks in the dual-path model are
redundant because their role is mainly to capture the global context.
Therefore, quasi-dual-path network (QDPN) [55] proposes to use downsampling
instead of inter-chunk processing to reduce the redundancy. Inspired by QDPN,
we design efficient global attention to capture the global contexts without
chunking process. Finally, based on Transformer structure, we consider a
hybrid approach of global and local modules which are focusing on global and
local contexts, respectively.

#### Multi-scale model for efficiency

Instead of the dual-path model, based on U-Net structure [57], some studies
have suggested using multi-scaled sequence model [61, 45, 19, 28, 39, 7].
SuDoRM-RF model [66] used a stacked U-Net structure to reduce the
computational cost. The SuDoRM-RF approach can be regarded as a substitution
of the TCN block in Conv-TasNet with U-ConvBlock as U-Net sub-block. Although
SuDoRM-RF reduces the computational cost, it still has the disadvantages of
having a fixed receptive field size and not considering the global context.
More recently, TDANet [39] has efficiently improved performance with top-down
attention and unfolding as in A-FCRNN [28]. However, these conventional
methods with multi-scaled sequences prefer stacked or recurrent structures
with U-Net sub-block to improve performance. Instead, we consider a single
U-Net architecture and explicitly divide the roles of encoder and decoder as
separation and reconstruction.

#### Siamese networks for discriminative learning

Siamese networks [4] are weight-sharing neural networks widely used in modern
contrastive learning [10, 6, 22] including speaker verification [84, 54, 15].
On the other hand, some studies on speech separation proposed to exploit
speaker identity as discriminative information to address the case that the
similar voices are mixed [48, 47, 83]. Therefore, given the success of siamese
networks in contrastive learning, we utilized the siamese network to
reconstruct separated speech by extracting distinct speech representations for
corresponding speakers. To separate the mixture, Siamese decoder directly
learns to focus discriminative features without the need for additionally
designed, for example, speaker loss using an additional speaker embedding
extractor. As a result, based on discriminative learning, the Siamese network
strengthens the dominant speaker’s components on each separated sequence,
respectively.

##  3 Method

###  3.1 Overall pipeline

When input mixture $\mathbf{x}\in\mathbb{R}^{1\times N}$, the 1D convolution
audio encoder, followed by GELU activation [24], encodes $\mathbf{x}$ to the
input representation, as
$\mathbf{X}=\mathcal{E}(\mathbf{x})\in\mathbb{R}^{F_{o}\times T},$where
$F_{o}$ and $T$ denote the number of convolutional filter of encoder and the
number of frames, respectively. The kernel and stride size are $L$ and $H$,
respectively. Then, the $J$ output representations $\hat{\mathbf{Y}}_{j}$ are
estimated from the separator and decoded by the audio decoder, expressed as
$\hat{\mathbf{s}}_{j}=\mathcal{D}(\mathbf{Y}_{j})\in\mathbb{R}^{1\times
N},1\leq j\leq J.$ Following the recent works [56, 73], we design the
separator to directly map the output signals instead of masking.

Figure 2: The architecture of the separator in the proposed SepReformer. The
separator consists of three parts: separation encoder, speaker split module,
and reconstruction decoder.

###  3.2 Architecture of separator

The detailed architecture of the separator of the proposed SepReformer is
illustrated in Figure 2. The separator is constructed on the basis of the ESSD
framework with a separation encoder and a reconstruction decoder in temporally
multi-scaled U-Net structure.

#### Separation encoder

The input representation is first projected to $F$ dimension by the input
layer. The input layer is composed of the linear layer and Layer Normalization
(LN) [1] applied to each frame independently. In the encoder, the projected
feature sequence is successively downsampled $R$ times from the sequence
length of $T$ to $T/2^{R}$. The downsampling is performed by a 1D depth-wise
convolution (Dconv) layer with a stride of 2 and a kernel size of 5, followed
by Batch Normalization (BN) and GELU activation [24]. Each encoder stage
processes the single sequence feature by $B_{E}$ stacks of global and local
Transformer blocks.

Figure 3: Speaker split module

#### Speaker split

The encoded features in all stages of the encoder are expanded by the number
of speakers $J$ to transmit the speaker-wise features from the encoder to the
decoder. Therefore, the speaker split layer is placed in the middle, and it
commonly separates the intermediate encoder features used for skip connections
as well as the bottleneck feature. As shown in Figure 3, this module consists
of two linear layers with gated linear unit (GLU) activation [18]. Each
feature is then normalized by LN and processed by the decoder.

#### Reconstruction decoder

For temporal reconstruction, the upsampled sequence feature from the previous
stage is concatenated with the skip connection followed by linear layer. Then,
$B_{D}$ stacks of global and local Transformer blocks process the $J$ feature
sequences as a Siamese network to discriminate between the separated features.
By incorporating the separation objective function into the Siamese decoder,
the network directly learns to capture the discriminative features. Then, the
output of the last decoder stage is projected back to $F_{o}$ dimension by an
output layer. The output layer consists of two linear layers with GLU
activation.

#### Cross-speaker (CS) Transformer

During the discrimination process by the Siamese decoder, the speech elements
can be mistakenly clustered into other speaker channels. As a result, it would
be beneficial to attend to each other in order to effectively recover the
distorted speech elements. Therefore, to improve the interaction of contexts
between speakers within the decoder, we incorporate a Transformer-based CS
module as in [13, 37]. Based on multi-head self-attention (MHSA) module
without positional encoding, the CS block performs an attention operation on
speaker dimension while temporal dimension is processed independently.
Therefore, the CS block learns to identify the interfering components of the
opposing sequences within the same temporal frame. For convenience, we call
ESSD with CS as a separation-and-reconstruction (SepRe) method.

###  3.3 Global and local Transformer for long sequences

Instead of the dual-path model based on chunking, we directly process a long
sequence using global and local processing blocks, similar to QDPN [55] or
Conformer [23]. In particular, global and local blocks replace inter- and
intra-chunk processing, respectively. The design of the blocks follows a
Transformer block structure to ensure structural effectiveness [20, 64, 80,
25]. This structure consists of two sub-modules: temporal mixing and frame-
wise channel mixing. These modules are stacked together with a pre-norm
residual unit [70, 51] and LayerScale [65] to facilitate faster training of
deep networks. Also, in all residual units, we apply dropout [60] for
regularization.

(a)

(b)

Figure 4: Block diagrams of global and local Transformer for sequence
processing. $\downarrow$ and $\uparrow$ in EGA denote downsampling with
average pooling and upsampling with nearest interpolation. Note that the
point-wise convolution (Pconv) layer performs an equivalent operation to the
linear layer as channel mixing. The hidden dimension of GCFN is set to $3F$
after GLU to maintain a similar parameter size to the FFN with a hidden size
of $4F$. Therefore, while the FFN has parameter size of $8F^{2}$, GCFN has a
slightly larger size of about $9F^{2}$.

#### Gated convolutional feed-forward network (GCFN)

Instead of using the conventional feed-forward network (FFN) [20, 68] for
channel mixing, we improve it by incorporating temporal Dconv with a small
kernel size of 3 and substituting GELU with GLU activation [18] as shown in
Figure 4(a). This GCFN can effectively process channel features by considering
the adjacent frame context. Several studies also have demonstrated the
effectiveness of these enhancements in FFN [59, 81, 72].

#### Global Transformer with efficient global attention (EGA)

In Figure 4(a), the global block consists of an EGA module for temporal mixing
and GCFN. The EGA module is based on the MHSA with relative positional
encoding [17]. However, to reduce the computation and focus on global
information in the attention layer, the downsampled sequence is processed and
upsampled back. Sequences $T/2^{r}$ at all stages $0\leq r\leq R-1$ are
downsampled to $T/2^{R}$, which is equal to the length in the bottleneck. To
compensate for downsampling, the upsampled features are multiplied by the gate
value obtained from an additional branch with a linear layer and sigmoid
function $\sigma$. The simple strategy allows the effective capture of global
contexts while maintaining local contexts.

#### Local Transformer with convolutional local attention (CLA)

For the local block, we design a CLA module based on 1D temporal convolution
with a large kernel of $K$ in Figure 4(b). Inspired by [23, 78], the CLA
module first processes the feature with the Pconv layer and GLU activation to
facilitate capturing local contexts attentively. After the temporal Dconv, two
Pconv layers are used. They have a hidden dimension of $2F$ and employ BN and
GELU activation.

###  3.4 Boosting discriminative learning by multi-loss

The objective function is given as scale-invariant signal-to-noise ratio (SI-
SNR) [58, 43] defined as

$\mathcal{L}=-\sum_{j=1}^{J}\min\left(20\log_{10}{\frac{\|\gamma_{j}\mathbf{s}_{j}\|_{2}}{\|\gamma_{j}\mathbf{s}_{j}-\hat{\mathbf{s}}_{j}\|_{2}}},\tau\right),$
(1)

where
$\gamma_{j}=\hat{\mathbf{s}}^{T}_{j}\mathbf{s}_{j}/\|\mathbf{s}_{j}\|_{2}^{2}$
and $\|\cdot\|_{2}$ denotes L2-norm. The clipping value of $\tau$ limits the
influence of the best training prediction [83, 77]. Notably, the output of the
decoder stages can be trained for progressive reconstruction as the feature
sequences are already separated in the decoder stages as in the progressive
multi-stage strategy [50, 85, 82, 14]. In particular, Siamese decoder in each
stage can be trained clearly for discriminative learning with stage-specific
separation objective. This multi-loss strategy is also considered to guide
intermediate features in audio separation [49, 5, 55, 56, 37].

Therefore, the source signal can be estimated as
$\hat{\mathbf{s}}_{j,r}=\mathcal{D}_{r}(\mathbf{X}\odot\mathbf{M}_{j,r})\in\mathbb{R}^{1\times
N}$ when $\mathbf{M}_{j,r}\in\mathbb{R}^{F_{o}\times T}$ is estimated with
additional output layers for $\mathbf{L}_{j,r}$ and the nearest upsampling.
$\odot$ denotes an element-wise multiplication, and $\mathcal{D}_{r}(\cdot)$
is an auxiliary audio decoder, which is also additional required with
additional output layers. Therefore, we can set the auxiliary objective
function as

$\mathcal{L}_{r}=-\sum_{j=1}^{J}\min\left(20\log_{10}{\frac{\|\gamma_{j,r}\mathbf{s}_{j}\|_{2}}{\|\gamma_{j,r}\mathbf{s}_{j}-\hat{\mathbf{s}}_{j,r}\|_{2}}},\tau\right),$
(2)

where
$\gamma_{j,r}=\hat{\mathbf{s}}^{T}_{j,r}\mathbf{s}_{j}/\|\mathbf{s}_{j}\|_{2}^{2}$.
Note that, when calculating the output from intermediate features, we opt for
masking instead of direct estimation because the temporal resolutions of the
feature sequences are deficient. Then, the multi-loss can be set to
$\hat{\mathcal{L}}=(1-\alpha)\mathcal{L}+{\alpha}\sum_{r=1}^{R}\mathcal{L}_{r}/R$.
Moreover, we alternatively calculate the intermediate loss $\mathcal{L}_{r}$
using the magnitude values of $\mathbf{s}_{j}$ and $\hat{\mathbf{s}}_{j}$ in
the STFT domain as it provided more stable training and no actual separated
signals are required from the intermediate outputs.

##  4 Experimental Settings

###  4.1 Dataset

We evaluated our proposed SepReformer on WSJ0-2Mix [26], WHAM! [76], WHAMR!
[46], and LibriMix [16], which are popular datasets for monaural speech
separation. To ensure generality, the mixtures in the test set were generated
by the speakers that were not seen during training. For all the datasets,
networks were trained with 4-s-long segments at a 8-kHz sampling rate.

#### WSJ0-2Mix

WSJ0-2Mix is the most popular dataset to benchmark the monaural speech
separation task. It contains 30, 10, and 5 hours for training, validation, and
evaluation sets, respectively. Each mixture was artificially generated by
randomly selecting different speakers from the corresponding set and mixing
them at a random relative signal-to-noise ratio (SNR) between -5 and 5 dB.

#### WHAM!/WHAMR!

WHAM!/WHAMR! is a noisy/noisy-reverberant version of the WSJ0-2Mix dataset. In
the WHAM! dataset, speeches were mixed with noise recorded in scenes such as
cafes, restaurants, and bars. The noise was added to get mixtures at SNRs
uniformly sampled between -6dB and 3dB, making the mixtures more challenging
than those in the WSJ0-2Mix.

#### Libri2Mix

In Libri2Mix dataset, the target speech in each mixture was randomly selected
from a subset of LibriSpeech’s train-100 [52] for faster training. Each source
was mixed with uniformly sampled Loudness Units relative to Full Scale (LUFS)
to get a mixture at an SNR between -25 and -33 dB. We used the clean version
as in previous studies [9, 39].

###  4.2 Training and model configuration

We trained the proposed SepReformer for a maximum of 200 epochs with an
initial learning rate of $1.0e^{-3}$. We used a warm-up training scheduler for
the first epoch, and then the learning rate decayed by a factor of 0.8 if the
validation loss did not improve in three consecutive epochs. As optimizer,
AdamW [40] was used with a weight decay of $0.01$, and gradient clipping with
a maximum L2-norm of 5 was applied for stable training. All models were
trained with Permutation Invariant Training (PIT) [33]. When the multi-loss in
Subsection 3.4 was applied, the $\alpha$ was set to 0.4, and after 100 epochs,
it decayed by a factor of 0.8 at every five epochs. $\tau$ was set to 30 as in
[83]. SI-SNRi and SDRi [69] were used as evaluation metrics. Also, we compared
the parameter size and the number of multiply-accumulate operations (MACs) for
16000 samples. The number of heads in MSHA was commonly set to 8, and the
kernel size $K$ in the local block was set to $65$. Also, we evaluated our
model in various model sizes as follows:

  * •

SepReformer-T/B/L: $F=64/128/256$, $F_{o}=256$, $L=16$, $H=4$, $R=4$

  * •

SepReformer-S/M: $F=64/128$, $F_{o}=256$, $L=8$, $H=2$, $R=5$

Note that we did not train the models multiple times, as the deviations in the
results are negligible below the significant digits. Also, all experiments
were conducted on a server with GeForce RTX 3090 $\times$ 6\. The Pytorch
implementation of the models is publicly available.

##  5 Results

###  5.1 Ablation studies of SepRe method

#### Decoder Design

In Table 1(a), we evaluated various decoder structures (See Appendix A for
detailed structures) to validate the effectiveness of Siamese decoder
structure. In particular, the model using multiple decoders after an early
split yielded a performance comparable to that of using a large decoder after
a late split. In contrast, by sharing a decoder after an early split, the
separation result increased significantly, suggesting that the ESSD structure
effectively discriminates between the separated features. This impact was more
noticeable in the tiny models by showing increase of 1.5dB. Finally, applying
CS to ESSD improved the performance especially on the tiny model, leading to
the SepRe mechanism.

#### Effects of multi-loss

Furthermore, we experimented with the effects of multi-loss on various decoder
structures in Table 1(b). Compared to a late split, the case with an early
split increased more significantly with multi-loss because an early split
structure could be trained with a clearer objective for discriminative
learning using an intermediate loss at each stage. In particular, while
applying only CS without multi-loss resulted in a marginal improvement,
combining CS with multi-loss led to a substantial gain. The results
demonstrated that stage-specific objective functions induce each CS-equipped
Siamese decoder stage to effectively learn simultaneously how to discriminate
between and attend to each other. As a result, our proposed SepRe method using
ESSD and CS significantly improved separation performance by applying stage-
specific objective functions and inducing progressive reconstruction of
separated feature sequences. See Appendix B for detailed discussions.

Case | Param. | SI-SNRi
---|---|---
(M) | (dB)
late split + origin dec. | 3.3/11.6 | 19.0/21.6
late split + large dec. | 5.7/20.1 | 19.7/22.0
early split + multi dec. | 5.8/20.9 | 19.8/22.1
early split + shared dec. | 2.8/11.6 | 21.3/23.1
early split + shared dec. + CS | 3.7/14.2 | 22.4/23.8
(a) Decoder design.

Case | ML | Param. | SI-SNRi
---|---|---|---
(M) | (dB)
late split + origin dec. |  | 11.6 | 21.2
late split + origin dec. | ✓ |  13.2∗ | 21.6
early split + shared dec. |  | 11.6 | 22.4
early split + shared dec. | ✓ |  12.2∗ | 23.1
early split + shared dec. + CS |  | 14.2 | 22.6
early split + shared dec. + CS | ✓ | 14.8∗ | 23.8
(b) Effects of multi-loss.

Table 1: Experimental evaluation of SepRe method on the WSJ0-2Mix dataset. ML
denotes the multi-loss. In (a), all the methods were trained with ML, and the
numbers in the left and right of the ’/’ symbol were obtained for the tiny and
base models, respectively. In (b), when ML was used for training, we indicated
the numbers of parameters including the additional output layer for an
auxiliary output for $\hat{\mathbf{s}}_{j}$, which were denoted with asterisk
∗. Note that the additional output layers were not required during inference.

Case | ESSD | CS | ML |  Param. (M) |  SI-SNRi (dB)
---|---|---|---|---|---
1 (origin.) |  |  |  | 5.1 | 15.3
2 | ✓ |  |  | 5.4 | 17.5
3 | ✓ |  | ✓ | 5.5 | 17.8
4 (SepRe) | ✓ | ✓ |  | 5.7 | 19.2
5 (SepRe) | ✓ | ✓ | ✓ | 5.7 | 19.5
(a) Conv-TasNet with SepRe method.

Case | ESSD | CS | ML |  Param. (M) |  SI-SNRi (dB)
---|---|---|---|---|---
1 (origin.) |  |  |  | 26.0 | 20.4
2 | ✓ |  |  | 27.1 | 21.3
3 | ✓ |  | ✓ | 27.2 | 22.0
4 (SepRe) | ✓ | ✓ |  | 28.0 | 21.6
5 (SepRe) | ✓ | ✓ | ✓ | 28.0 | 22.7
(b) Sepformer with SepRe method.

Table 2: Application of SepRe to other networks. From the original separator
of Conv-TasNet and Sepformer, we applied the SepRe method with multi-loss (ML)
and evaluated on the WSJ0-2Mix dataset.

###  5.2 Effects of the SepRe method in other networks

To validate the general applicability of the SepRe method, we incorporated the
SepRe method with multi-loss into the original separators of Conv-TasNet [44]
and Sepformer [62] and conducted experiments on WSJ0-2MIX. The experimental
results in Table 2 demonstrated a significant performance improvement when
ESSD was applied for both networks. Also, applying CS and multi-loss in
addition to the ESSD framework improved the performance further, which
confirms the effectiveness of SepRe with multi-loss.

###  5.3 Ablation studies of unit blocks

#### Depth of encoder-decoder

In Table 3(a), we experimented the depth of encoder and decoder to determine
the optimal configuration in terms of the number of block repetition $B_{E}$
and $B_{D}$. Generally, experimental results showed that using more blocks in
the reconstruction decoder had a greater impact on performance improvement
than in the separation encoder. It demonstrate that it is more important to
discriminate the features more elaborately in Siamese decoder than to analyze
the features in encoder in speech separation. In particular, optimal
performance was achieved with $B_{E}=2$ and $B_{D}=3$, which were used as the
common configuration for subsequent experiments.

#### EGA module design

Next, we validated our proposed EGA module by ablating its components (see
Appendix C for detailed structures) in Table 3(b). First of all, using vanilla
MHSA on a long sequence without chunking was infeasible due to the extremely
large computational requirements. Therefore, one approach was to perform
downsampling before applying MHSA, similar to the method used in QDPN [55].
However, this naive approach had the drawback of losing detailed frame-wise
information. Although another consideration was to simply multiply the
features to reflect the fine-grained frame-wise information, this method still
could not significantly improve performance. In contrast, the optimal
performance was achieved by estimating gate values based on a linear layer and
a sigmoid function. As a result, it is shown that our proposed Global
Transformer with EGA module and Local Transformer with CLA module have
effectively replaced conventional sequence models with smaller computations.
See Appendix D for additional experiments.

#### FFN module design

Also, by improving the design of FFN with Dconv and GLU activation, we could
achieve the significant improvement of performance with slight increase of
parameters as shown in Table 3(c).

$(B_{E},B_{D})$ | Param. | SI-SNRi
---|---|---
(M) | (dB)
$(1,4)$ | 14.6 | 23.6
$(2,3)$ | 14.2 | 23.8
$(3,2)$ | 13.8 | 22.8
$(4,1)$ | 13.4 | 21.7
(a) Depth of encoder-decoder.

Case | Param. | SI-SNRi
---|---|---
(M) | (dB)
MHSA w/ d.s & u.s | 13.9 | 23.3
EGA w/o linear gate | 13.9 | 23.2
EGA | 14.2 | 23.8
(b) EGA module design.

Case | Param. | SI-SNRi
---|---|---
(M) | (dB)
FFN [20] | 13.3 | 23.0
FFN w/ Dconv | 13.4 | 23.4
FFN w/ GLU | 14.1 | 23.3
GCFN | 14.2 | 23.8
(c) FFN module design.

Table 3: Ablation studies for unit blocks on our SepReformer-B on the
WSJ0-2Mix dataset. Various configurations of $B_{E}$ and $B_{D}$ were
evaluated to assess the relative importance of encoder and decoder. Also, we
validated the proposed EGA and GCFN modules.

###  5.4 Comparison with existing models

Finally, we compared our SepReformer models with existing separation models on
various benchmark datasets in Table 4. Although we evaluated SepReformer
trained with standard pairs from the training set in Table 4(a), SepReformer-L
was trained with dynamic mixing (DM) [83, 62] for data augmentation and
compared to other existing large models with DM in Table 4(b). In Table 4(a),
with almost the smallest computational loads in terms of MACs, our tiny model
showed the best performance except for TF-GridNet in the WSJ0-2Mix dataset
which was a powerful model recently proposed. It demonstrated the efficiency
of the SepRe method in speech separation. Also, SepReformer-M without DM in
Table 4(a) showed competitive separation performance on WSJ0-2Mix compared to
the large models with data augmentation in Table 4(b). In particular,
SepReformer-L with DM achieved the SOTA performance of 25 dB of SI-SNRi on
WSJ0-2Mix, showing significantly improved performance over other conventional
methods.

In Table 4(a), the smallest SepReformer-T among the proposed models even
showed significant improvements on WHAM! and Libri2Mix datasets compared to
the conventional methods. It suggested that the proposed SepRe method can be
efficiently applied to a speech separation task in general. Also,
SepReformer-L with DM showed the SOTA performance on WHAM! and WHAMR!
datasets, as well as WSJ0-2Mix, which demonstrated that the proposed method
can be trained effectively in a large model.

System | Params. | MACs | WSJ0-2Mix | WHAM! | Libri2Mix
---|---|---|---|---|---
(M) | (G/s) | SI-SNRi | SDRi | SI-SNRi | SDRi | SI-SNRi | SDRi
(dB) | (dB) | (dB) | (dB) | (dB) | (dB)
Conv-TasNet [44] | 5.1 | 10.5 | 15.3 | 15.6 | 12.7 | - | 12.2 | 12.7
SuDoRM-RF [66] | 6.4 | 10.1 | 18.9 | - | 13.7 | 14.1 | 14.0 | 14.4
TDANet [39] | 2.3 | 9.1 | $\hskip 3.41432pt18.5^{\dagger}$ | $\hskip 3.41432pt18.7^{\dagger}$ | 15.2 | 15.4 | 17.4 | 17.9
Sandglasset [35] | 2.3 | 28.8 | 20.8 | 21.0 | - | - | - | -
S4M [7] | 3.6 | 38.4 | 20.5 | 20.7 | - | - | 16.9 | 17.4
SepReformer-T | 3.7 | 10.4 | 22.4 | 22.6 | 17.2 | 17.5 | 19.7 | 20.2
SepReformer-S | 4.5 | 21.3 | 23.0 | 23.1 | 17.3 | 17.7 | 20.6 | 21.0
DPRNN [42] | 2.6 | 88.5 | 18.8 | 19.0 | 13.7 | 14.1 | 16.1 | 16.6
DPTNet [9] | 2.7 | 102.5 | 20.2 | 20.3 | 14.9 | 15.3 | 16.7 | 17.1
Sepformer [62] | 26.0 | 86.9 | 20.4 | 20.5 | 14.7 | 16.8 | 16.5 | 17.0
WaveSplit‡ [83] | 29.0 | - | 21.0 | 21.2 | 16.0 | 16.5 | 16.6 | 17.2
A-FRCNN [28] | 6.1 | 125.0 | 18.3 | 18.6 | 14.5 | 14.8 | 16.7 | 17.2
SFSRNet [56] | 59.0 | 124.2 | 22.0 | 22.1 | - | - | - | -
ISCIT [47] | 58.4 | 252.2 | 22.4 | 22.5 | 16.4 | 16.8 | - | -
QDPN [55] | 200.0 | - | 22.1 | - | - | - | - | -
TF-GridNet [73] | 14.5 | 460.8 | 23.5 | 23.6 | - | - | - | -
SepReformer-B | 14.2 | 39.8 | 23.8 | 23.9 | 17.6 | 18.0 | 21.7 | 22.1
SepReformer-M | 17.3 | 81.3 | 24.2 | 24.4 | 17.8 | 18.1 | 22.1 | 22.5
(a) Comparison of SepReformer to existing models.

System | Params. | MACs | WSJ0-2Mix | WHAM! | WHAMR!
---|---|---|---|---|---
(M) | (G/s) | SI-SNRi | SDRi | SI-SNRi | SDRi | SI-SNRi | SDRi
(dB) | (dB) | (dB) | (dB) | (dB) | (dB)
Sepformer [63] | 26.0 | 86.9 | 22.3 | 22.5 | 16.4 | 16.7 | 14.0 | 13.0
WaveSplit‡ [83] | 29.0 | - | 21.0 | 21.2 | - | - | 13.2 | 12.2
SFSRNet [56] | 59.0 | 466.2 | 24.0 | 24.1 | - | - | - | -
ISCIT‡ [47] | 58.4 | 252.2 | 24.3 | 24.4 | 16.9 | 17.2 | - | -
QDPN [55] | 200.0 | - | 23.6 | - | - | - | 14.4 | -
Mossformer(L) [86] | 42.1 | 86.1 | 22.8 | - | 17.3 | - | 16.3 | -
Mossformer2(L) [87] | 55.7 | - | 24.1 | - | 18.1 | - | 17.0 | -
SepReformer-L | 55.3 | 155.5 | 25.1 | 25.2 | 18.5 | 18.8 | 17.2 | 16.1
(b) Comparison of SepReformer-L to existing large models with DM.

Table 4: Evaluation on various benchmark dataset of WSJ0-2MIX, WHAM!, WHAMR!,
and Libri2Mix. “$\dagger$” denotes that they were not reported in the original
paper but evaluated using the code provided by an author. "$\ddagger$" denotes
that the networks use additional speaker information.

##  6 Conclusion

In this work, we introduced the SepRe method, in which the asymmetric encoder
and decoder perform separation and reconstruction, respectively. The encoder
analyzes and separates a feature sequence, and the separated sequences are
reconstructed by a weight-sharing network and a cross-speaker network. We
demonstrated that the SepRe method can be applied to conventional separators
in general and utilizing multi-loss significantly improves the performance.
Moreover, we replaced the dual-path model with presented global and local
Transformer blocks to address a long sequence. The separator using the
presented unit blocks has shown enhanced separated results efficiently, and
combining a U-Net structure to exploit the multi-scale sequence model has
further increased the efficiency. Finally, not only did our presented
SepReformer outperform the most conventional methods in speech separation even
with almost the smallest computational resources, but our large models
achieved SOTA performance with large margins compared to the conventional
models on various speech separation datasets.

#### Limitations and future work.

Our study focuses on 2-speaker mixture situation to assess our models in
various model sizes and in the extensive datasets including noise and
reverberation. Consequently, we believe that further investigation is needed
to validate for more than 2-speaker mixture scenarios. Also, an important
future direction is to separate mixtures for an unknown number of speakers as
it is impractical to assume that the number of speakers to be separated is
known in advance. Finally, although we experimentally validated our SepRe
method, we believe that further investigation is necessary to figure out its
underlying mechanism.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * [2] Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer.  CoRR, abs/2004.05150, 2020.
  * [3] Albert S Bregman.  Auditory scene analysis: The perceptual organization of sound.  MIT press, 1994.
  * [4] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  Signature verification using a "siamese" time delay neural network.  In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume 6. Morgan-Kaufmann, 1993.
  * [5] Jaeuk Byun and Jong Won Shin.  Monaural speech separation using speaker embedding from preliminary separation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2753–2763, 2021.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912–9924. Curran Associates, Inc., 2020.
  * [7] Chen Chen, Chao-Han Huck Yang, Kai Li, Yuchen Hu, Pin-Jui Ku, and Eng Siong Chng.  A neural state-space model approach to efficient speech separation, 2023.
  * [8] Hangting Chen, Yi Yang, Feng Dang, and Pengyuan Zhang.  Beam-Guided TasNet: An Iterative Speech Separation Framework with Multi-Channel Output.  In Proc. Interspeech 2022, pages 866–870, 2022.
  * [9] Jingjing Chen, Qirong Mao, and Dong Liu.  Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation.  In Proc. Interspeech 2020, pages 2642–2646, 2020.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR, 13–18 Jul 2020.
  * [11] Zhuo Chen, Yi Luo, and Nima Mesgarani.  Deep attractor network for single-microphone speaker separation.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 246–250. IEEE Press, 2017.
  * [12] E. Colin Cherry.  Some Experiments on the Recognition of Speech, with One and with Two Ears.  The Journal of the Acoustical Society of America, 25(5):975–979, 06 2005.
  * [13] Srikanth Raj Chetupalli and Emanuël Habets.  Speech Separation for an Unknown Number of Speakers Using Transformers With Encoder-Decoder Attractors.  In Proc. Interspeech 2022, pages 5393–5397, 2022.
  * [14] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko.  Rethinking coarse-to-fine approach in single image deblurring.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4641–4650, October 2021.
  * [15] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee-Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and Icksang Han.  In Defence of Metric Learning for Speaker Recognition.  In Proc. Interspeech, pages 2977–2981, 2020.
  * [16] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent.  LibriMix: An open-source dataset for generalizable speech separation, 2020.
  * [17] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [18] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 933–941. PMLR, 06–11 Aug 2017.
  * [19] Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis R. Bach.  Music source separation in the waveform domain.  CoRR, abs/1911.13254, 2019.
  * [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [21] Hakan Erdogan, John R. Hershey, Shinji Watanabe, Michael I. Mandel, and Jonathan Le Roux.  Improved MVDR Beamforming Using Single-Channel Mask Prediction Networks.  In Proc. Interspeech 2016, pages 1981–1985, 2016.
  * [22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [23] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.  Conformer: Convolution-augmented Transformer for Speech Recognition.  In Proc. Interspeech 2020, pages 5036–5040, 2020.
  * [24] Dan Hendrycks and Kevin Gimpel.  Bridging nonlinearities and stochastic regularizers with gaussian error linear units.  CoRR, abs/1606.08415, 2016.
  * [25] Hyun-Jun Heo, Ui-Hyeop Shin, Ran Lee, YoungJu Cheon, and Hyung-Min Park.  Next-TDNN: Modernizing multi-scale temporal convolution backbone for speaker verification.  In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11186–11190, 2024.
  * [26] John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe.  Deep clustering: Discriminative embeddings for segmentation and separation.  In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 31–35, 2016.
  * [27] Jahn Heymann, Lukas Drude, Aleksej Chinaev, and Reinhold Haeb-Umbach.  BLSTM supported gev beamformer front-end for the 3rd chime challenge.  In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 444–451, 2015.
  * [28] Xiaolin Hu, Kai Li, Weiyi Zhang, Yi Luo, Jean-Marie Lemercier, and Timo Gerkmann.  Speech separation using an asynchronous fully recurrent convolutional neural network.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22509–22522. Curran Associates, Inc., 2021.
  * [29] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le.  Transformer quality in linear time.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9099–9117. PMLR, 17–23 Jul 2022.
  * [30] Xilin Jiang, Cong Han, and Nima Mesgarani.  Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation, 2024.
  * [31] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient Transformer.  In International Conference on Learning Representations, 2020.
  * [32] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al.  Siamese neural networks for one-shot image recognition.  In ICML deep learning workshop, volume 2. Lille, 2015.
  * [33] Morten Kolbæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen.  Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1901–1913, 2017.
  * [34] Max W. Y. Lam, Jun Wang, Dan Su, and Dong Yu.  Effective low-cost time-domain audio separation using globally attentive locally recurrent networks.  In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 801–808, 2021.
  * [35] Max W. Y. Lam, Jun Wang, Dan Su, and Dong Yu.  Sandglasset: A light multi-granularity self-attentive network for time-domain speech separation.  In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5759–5763, 2021.
  * [36] Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager.  Temporal convolutional networks for action segmentation and detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [37] Younglo Lee, Shukjae Choi, Byeong-Yeol Kim, Zhong-Qiu Wang, and Shinji Watanabe.  Boosting unknown-number speaker separation with transformer decoder-based attractor, 2024.
  * [38] Andong Li, Wenzhe Liu, Chengshi Zheng, and Xiaodong Li.  Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement.  In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6487–6491, 2022.
  * [39] Kai Li, Runxuan Yang, and Xiaolin Hu.  An efficient encoder-decoder architecture with top-down attention for speech separation.  In The Eleventh International Conference on Learning Representations, 2023.
  * [40] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [41] Yi Luo, Zhuo Chen, and Nima Mesgarani.  Speaker-independent speech separation with deep attractor network.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(4):787–796, 2018.
  * [42] Yi Luo, Zhuo Chen, and Takuya Yoshioka.  Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46–50, 2020.
  * [43] Yi Luo and Nima Mesgarani.  TasNet: Time-domain audio separation network for real-time, single-channel speech separation.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 696–700, 2018.
  * [44] Yi Luo and Nima Mesgarani.  Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–1266, 2019.
  * [45] Craig Macartney and Tillman Weyde.  Improved speech enhancement with the wave-u-net.  arXiv preprint arXiv:1811.11307, 2018.
  * [46] Matthew Maciejewski, Gordon Wichern, Emmett McQuinn, and Jonathan Le Roux.  WHAMR!: Noisy and reverberant single-channel speech separation.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 696–700, 2020.
  * [47] Zhaoxi Mu, Xinyu Yang, and Wenjing Zhu.  Multi-dimensional and multi-scale modeling for speech separation optimized by discriminative learning.  In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023.
  * [48] Hannah Raphaelle Muckenhirn, Ignacio Lopez Moreno, John Hershey, Kevin Wilson, Prashant Sridhar, Quan Wang, Rif A. Saurous, Ron Weiss, Ye Jia, and Zelin Wu.  Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking.  In ICASSP 2019, 2018.
  * [49] Eliya Nachmani, Yossi Adi, and Lior Wolf.  Voice separation with an unknown number of multiple speakers.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7164–7175. PMLR, 13–18 Jul 2020.
  * [50] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [51] Toan Q. Nguyen and Julian Salazar.  Transformers without tears: Improving the normalization of self-attention.  In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2-3 2019. Association for Computational Linguistics.
  * [52] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: An ASR corpus based on public domain audio books.  In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015.
  * [53] Shuangqing Qian, Lijian Gao, Hongjie Jia, and Qirong Mao.  Efficient monaural speech separation with multiscale time-delay sampling.  In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6847–6851, 2022.
  * [54] F A Rezaur rahman Chowdhury, Quan Wang, Ignacio Lopez Moreno, and Li Wan.  Attention-based models for text-dependent speaker verification.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5359–5363, 2018.
  * [55] Joel Rixen and Matthias Renz.  QDPN - Quasi-dual-path Network for single-channel Speech Separation.  In Proc. Interspeech 2022, pages 5353–5357, 2022.
  * [56] Joel Rixen and Matthias Renz.  SFSRnet: Super-resolution for single-channel audio source separation.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11220–11228, Jun. 2022.
  * [57] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [58] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey.  SDR – half-baked or well done?  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 626–630, 2019.
  * [59] Noam Shazeer.  GLU variants improve transformer.  CoRR, abs/2002.05202, 2020.
  * [60] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15(1):1929–1958, jan 2014.
  * [61] Daniel Stoller, Sebastian Ewert, and Simon Dixon.  Wave-U-Net: A multi-scale neural network for end-to-end audio source separation.  In Emilia Gómez, Xiao Hu, Eric Humphrey, and Emmanouil Benetos, editors, Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, September 23-27, 2018, pages 334–340, 2018.
  * [62] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong.  Attention is all you need in speech separation.  In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21–25, 2021.
  * [63] Cem Subakan, Mirco Ravanelli, Samuele Cornell, François Grondin, and Mirko Bronzi.  Exploring self-attention mechanisms for speech separation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2169–2180, 2023.
  * [64] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  MLP-Mixer: An all-MLP architecture for vision.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 24261–24272. Curran Associates, Inc., 2021.
  * [65] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 32–42, October 2021.
  * [66] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis.  Sudo RM -RF: Efficient networks for universal audio source separation.  In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6, 2020.
  * [67] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  In Arxiv, 2016.
  * [68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [69] E. Vincent, R. Gribonval, and C. Fevotte.  Performance measurement in blind audio source separation.  IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462–1469, 2006.
  * [70] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao.  Learning deep transformer models for machine translation.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [71] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  CoRR, abs/2006.04768, 2020.
  * [72] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A general U-shaped transformer for image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17683–17693, June 2022.
  * [73] Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe.  TF-Gridnet: Making time-frequency domain models great again for monaural speaker separation.  In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023.
  * [74] Zhong-Qiu Wang, Peidong Wang, and DeLiang Wang.  Complex spectral mapping for single- and multi-channel speech enhancement and robust ASR.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1778–1787, 2020.
  * [75] Zhong-Qiu Wang, Peidong Wang, and DeLiang Wang.  Multi-microphone complex spectral mapping for utterance-wise and continuous speech separation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2001–2014, 2021.
  * [76] Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux.  WHAM!: Extending Speech Separation to Noisy Environments.  In Proc. Interspeech 2019, pages 1368–1372, 2019.
  * [77] Scott Wisdom, Hakan Erdogan, Daniel P. W. Ellis, Romain Serizel, Nicolas Turpault, Eduardo Fonseca, Justin Salamon, Prem Seetharaman, and John R. Hershey.  What’s all the FUSS about free universal sound separation data?  CoRR, abs/2011.00803, 2020.
  * [78] Zhanghao Wu*, Zhijian Liu*, Ji Lin, Yujun Lin, and Song Han.  Lite transformer with long-short range attention.  In International Conference on Learning Representations, 2020.
  * [79] Takuya Yoshioka, Hakan Erdogan, Zhuo Chen, and Fil Alleva.  Multi-microphone neural speech separation for far-field multi-talker speech recognition.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5739–5743, 2018.
  * [80] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10819–10829, June 2022.
  * [81] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  In CVPR, 2022.
  * [82] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14821–14831, June 2021.
  * [83] Neil Zeghidour and David Grangier.  Wavesplit: End-to-end speech separation by speaker clustering.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2840–2849, 2021.
  * [84] Chunlei Zhang, Kazuhito Koishida, and John H. L. Hansen.  Text-independent speaker verification based on triplet convolutional neural network embeddings.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(9):1633–1644, 2018.
  * [85] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz.  Deep stacked hierarchical multi-patch network for image deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [86] Shengkui Zhao and Bin Ma.  Mossformer: Pushing the performance limit of monaural speech separation using gated single-head transformer with convolution-augmented joint self-attentions.  In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023.
  * [87] Shengkui Zhao, Yukun Ma, Chongjia Ni, Chong Zhang, Hao Wang, Trung Hieu Nguyen, Kun Zhou, Jiaqi Yip, Dianwen Ng, and Bin Ma.  Mossformer2: Combining transformer and rnn-free recurrent network for enhanced time-domain monaural speech separation, 2023.

## Appendix / supplemental material

This appendix is organized as follows:

  * •

Appendix A describes the various decoder designs in Table 1(a).

  * •

Appendix B interprets the discriminative learning mechanism by Siamese decoder
in SepReformer.

  * •

Appendix C illustrates the cases of ablating EGA modules in Table 3(b)

  * •

Appendix D experiments on proposed global and local Transformer blocks for
modelling long sequence in speech separation.

  * •

Appendix E plots the separation performance on WSJ0-2Mix versus computations.

  * •

Appendix F provides sample demo link.

##  Appendix A Architecture of various decoder design

(a)

(b)

(c)

(d)

Figure 5: Block diagrams of various decoder designs experimented in Table 1 of
subsection 5.1. In all cases, the encoder and decoder consists of $R$ stages
and the blocks were stacks of global and local Transformer block in our cases.

In Figure 5, the block diagrams of various decoder designs are illustrated,
which were experimented on in Table 1. In Table 1(a), the case of a late
split, corresponding to the first and second rows, processes a single feature
sequence in both the encoder and decoder, forming a symmetric encoder-decoder
structure. In contrast, the case of an early split with multiple decoders,
shown in Figure 5(b), has each decoder block processing the separated
sequences from the encoder. Thus, Figure 5(b) can be seen as a special case of
a late split with a large decoder of $2F$, where the large decoder performs
group convolution with the number of groups equal to the number of speakers
$J$. On the other hand, in the ESSD method illustrated in Figure 5(c), the
decoder shares weights in a Siamese network structure to process the early
split feature sequences. Therefore, even without interaction between the
separated feature sequences, the decoder can learn discriminative
characteristics by sharing the weights. Furthermore, in the proposed SepRe
method shown in Figure 5(d), the decoder learns to attend to each other using
the CS block to additionally recover the deviated elements.

##  Appendix B Visualization of discriminative learning

Figure 6: Plot of cosine similarities for the two separated features in the
first decoder stage using a sample mixture in WSJ0-2Mix dataset.

To analyze the roles of each transformer block within the shared decoder of
the ESSD framework with multi-loss application, we calculate the cosine
similarities between the separated features from $\mathbf{Z1}$ to
$\mathbf{Z4}$ as shown in Figure 6.

First, we observe that $\mathbf{Z1}$ has higher cosine similarity compared to
the others, indicating that the separated features initially share similar
characteristics before being processed by the Siamese blocks.

However, after the Siamese global block, the similarities in $\mathbf{Z2}$ are
generally lower compared to $\mathbf{Z1}$. This result suggests that the
Siamese global blocks capture discriminative characteristics, making the
features more dissimilar in terms of cosine similarities.

The further decrease in similarity in $\mathbf{Z3}$ compared to $\mathbf{Z2}$
and $\mathbf{Z1}$ demonstrates the effect of the Siamese structure of the
local block. As features pass through subsequent local blocks, more region-
specific traits are refined. The local block, which handles these localized
characteristics, further enhances the distinctiveness of the local features,
resulting in decreased similarity.

In contrast, features processed through the CS block exhibit increased
similarity, unlike the Siamese structure. This increase in similarity can be
understood as the separated features attending to and becoming more similar to
each other, as the CS structure is designed to cross-reference information
between the speech features. During this process, the CS block preserves
distinct features and restores degraded information by leveraging mutual
information.

As the Siamese structure emphasizes unique characteristics, the split features
can deviate from the original characteristics of the speech within the same
frames, where the influence of each speaker’s information is similar. This
deviation occurs because emphasizing features in such frames can increase
interference from other speakers’ speech components, potentially distorting
and degrading these features. Therefore, the CS block after the Siamese block
is expected to recover the deviated features by attending to each other within
the frames.

Figure 7: The block diagram of ablation studies for EGA in Table 3(b).

##  Appendix C Illustration of ablation for EGA module

In Figure 7, we drew a block diagram of ablation studies for EGA in Table
3(b). In the fisrt row of Table 3(b), MHSA is simply performed with
downsampling and upsampling to reduce the sequence length. In the second row,
the upsampled output sequences of MHSA with downsampling is multiplied to the
input features before downsampling in order to reflect detailed temporal
information. However, simply multiplying the input features does not improved
the performances. To reflect the frame-level details to the upsampled feature,
we added the linear layer and sigmoid function to make gate values, leading to
our proposed EGA module.

##  Appendix D Comparison with various long sequence models

In Table 5, we evaluated the network by stacking our proposed global-local
Transformer blocks to assess the performance of modeling a long sequence. Note
that we did not apply the ESSD structure and multi-loss to our separator in
this experiment. We could observe that the network based on dual-path sequence
models requires high computation resources in terms of MACs while multi-scale
sequence models are more efficient. The recently proposed Mossformer based on
efficient gate attention unit (GAU) mechanism [29] showed improved performance
with relatively smaller computations compared to the networks with dual-path
model. In particular, the proposed model showed improved separation
performance with similar MACs, which demonstrated the capacity as a model for
a long sequence. It also suggested that the proposed block can sufficiently
replace the dual-path models with fewer computations. Furthermore, by
combining the U-Net structure into global-local Transformer blocks, the
network became more efficient with the similar separation performance.

Separator | Long sequence model | Param. (M) | MACs (G/s) |  SI-SNRi (dB)
---|---|---|---|---
Conv-TasNet [44] |  TCN [67] | 5.1 | 10.5 | 15.6
DPRNN [42] | Dual-path + BLSTM | 2.6 | 88.5 | 18.8
SuDoRM-RF [66] | Multi-scale + Convolution | 6.4 | 10.1 | 18.9
Sepformer [62] | Dual-path + Transformer | 26.0 | 86.9 | 20.4
MossFormer(S) [86] |  GAU [29] | 10.8 | 44.0 | 20.9
S4M [7] | Multi-scale + SSM | 3.6 | 38.4 | 20.5
Ours | Global-Local Transformer | 11.9 | 43.1 | 21.3
Ours + U-Net | Multi-scale + Global-Local Transformer | 11.6 | 18.3 | 21.2
Table 5: Comparison with various long sequence models in speech separation of
WSJ0-2Mix. MS denotes multi-scale. For our model, global and local blocks were
repeated 22 times with $F=128$. Figure 8: Si-SNRi results on WSJ0-2Mix versus
MACs (G/s) for the conventional methods and the proposed SepReformer. The
circle outlined in red indicates the use of DM method for training. The radius
of circle is proportional to the parameter size of the networks.

##  Appendix E Plot of Si-SNRi versus computations on WSJ0-2Mix

Figure 8 compares the separation performance of various existing methods in
terms of SI-SNRi versus MACs. These results correspond to those in Table 4.
From the figure, we can observe the effectiveness of the proposed SepReformer
in the speech separation task.

##  Appendix F Sample demo

The sample audio can be found at https://fordemopage.github.io/SepReformer/.