DS-TDNN: Dual-stream Time-delay Neural Network with Global-aware Filter
  for Speaker Verification
Conventional time-delay neural networks (TDNNs) struggle to handle long-range
context, their ability to represent speaker information is therefore limited in
long utterances. Existing solutions either depend on increasing model
complexity or try to balance between local features and global context to
address this issue. To effectively leverage the long-term dependencies of audio
signals and constrain model complexity, we introduce a novel module called
Global-aware Filter layer (GF layer) in this work, which employs a set of
learnable transform-domain filters between a 1D discrete Fourier transform and
its inverse transform to capture global context. Additionally, we develop a
dynamic filtering strategy and a sparse regularization method to enhance the
performance of the GF layer and prevent overfitting. Based on the GF layer, we
present a dual-stream TDNN architecture called DS-TDNN for automatic speaker
verification (ASV), which utilizes two unique branches to extract both local
and global features in parallel and employs an efficient strategy to fuse
different-scale information. Experiments on the Voxceleb and SITW databases
demonstrate that the DS-TDNN achieves a relative improvement of 10\% together
with a relative decline of 20\% in computational cost over the ECAPA-TDNN in
speaker verification task. This improvement will become more evident as the
utterance's duration grows. Furthermore, the DS-TDNN also beats popular deep
residual models and attention-based systems on utterances of arbitrary length.
Yangfu Li, Jiapan Gan, Xiaodan Lin
# Efficient Dual-stream Time-delay Neural Network with Global-aware Filter for Speaker Verification

###### Abstract

Conventional time-delay neural networks (TDNNs) face a challenge in handling long-range context, which affects their ability to represent speaker information, particularly for long-duration utterances. Existing solutions either significantly increase model complexity or achieve a poor trade-off between local features and global context. In this paper, we introduce a novel module called Global-aware Filter layer (GF layer), which employs a set of learnable transform-domain filters between a 1D discrete Fourier transform and its inverse transform to efficiently capture global context. Additionally, we innovatively design a dynamic filtering strategy and a sparse regularization method to enhance the performance of the GF layer and prevent overfitting. Based on the GF layer, we present a dual-stream TDNN architecture called DS-TDNN for automatic speaker verification (ASV). It utilizes two unique branches to extract both local and global features in parallel and employs a straightforward yet efficient strategy to fuse different-scale information. Experiments on the Voxceleb and SITV databases demonstrate that the DS-TDNN achieves a relative 10% improvement together with a relative 20% decline in computational cost over the ECAPA-TDNN in speaker verification. This improvement will become more significant with the growth of utterance durations. Furthermore, the DS-TDNN also beats popular deep residual models and attention-based baseline systems for arbitrary-duration utterances.

 Time-delay neural network, dual-stream network, text-independent speaker verification, global context.

## I Introduction

Automatic Speaker Verification (ASV) systems aim to determine whether a given utterance is from an enrolled speaker, which has been widely applied in user authentication, access control, multimedia forensics, and other areas [1, 2, 3]. Typically, an ASV system consists of two main components: a front-end that extracts low-dimensional discriminative speaker embeddings from variable-length utterances and a back-end that determines whether two embeddings are from the same speaker. With the development of deep neural networks (DNNs), the front-end has undergone a significant transformation from conventional probabilistic models [4, 5, 6] to DNN-based methods [7, 8, 9]. In particular, x-vector [9] is a state-of-the-art architecture for speaker embedding. The x-vector is based on the Time-Delay Neural Network (TDNN) [10] layers designed for end-to-end speech processing, where share-weight filters are employed to pay attention on the whole frequency axis and a time-delay strategy is applied to capture context between consecutive frames.

Despite its great success, TDNN still has limitations. Due to the short-term stationarity of human speech, both local features and global context are essential for extracting robust speaker representation. However, typical TDNN mainly focuses on local features but lacks global context fusion because of the small receptive field in each hidden layer. As a result, TDNN-based models are less resistant to abrupt noise and exhibit suboptimal performance in real-life applications, especially when facing long-duration utterances. A natural idea to strengthen global context is to extend the depth of TDNN. [11] introduces the residual connection [12] to construct an extremely deep TDNN that captures context information over a longer time span than the standard TDNN. In [13], the authors extend TDNN by inserting dense layers between each pair of its hidden layers. To prevent overfitting, [14] introduce dropouts [15] to TDNN and propose a carefully designed filter with varying temporal resolution for more powerful context representation. To further deep TDNN within affordable parameters, [16] makes a trade-off between the width and the depth. However, thin TDNN may have poor speaker representations. To solve this problem, [17] proposes a split-transform-merge structure, which can help the deep TDNN with limited width learn more discriminative speaker representations. These improvements make remarkable progress in performance but also significantly increase complexity. Additionally, they do not consider the fusion of local and global features. To solve these problems, many techniques have been proposed, which are divided into two categories in this paper.

One technique is to enhance the filters with multi-scale information. [18] introduces the Res2Net [19] structure into TDNN, which inserts skip connections between each grouped filter to construct a different-scale receptive field and proposes the multi-layer feature aggregation to fuse local and global features, achieving state-of-the-art performance. To enhance the Res2Net, [20] proposes a context-aware filter. This method further divides each grouped filter applied in Res2Net into two different-scale filters: a normal receptive field filter for local features and a large receptive field filter for global context, which are combined using the dynamic integration strategy. Authors in [21] propose another type of dynamic filter, whose value is determined by an element-wise summation between the local features calculated using moving average and the global context calculated using global average pooling. Recently, [22] presents a variable-resolution filter, which employs a kernel selection mechanism to find the optimal receptive field. Although these enhanced filters can dynamically choose to focus on local features or global context as required, none