NeuroSpex: Neuro-Guided Speaker Extraction with Cross-Modal Attention
Dashanka De Silva Siqi Cai Saurav Pahuja Tanja Schultz Haizhou Li



Geometry of temporal chiral structures 
    Andres F. Ordonez^1,2, Aycke Roos ^3, Pablo M. Maier ^3, David Ayuso^1,2,3 and Olga Smirnova^3,4
    September 9, 2024
====================================================================================================



§ ABSTRACT 
In the study of auditory attention, it has been revealed that there exists a robust correlation between attended speech and elicited neural responses, measurable through electroencephalography (EEG). Therefore, it is possible to use the attention information available within EEG signals to guide the extraction of the target speaker in a cocktail party computationally. In this paper, we present a neuro-guided speaker extraction model, i.e. NeuroSpex,  using the EEG response of the listener as the sole auxiliary reference cue to extract attended speech from monaural speech mixtures. We propose a novel EEG signal encoder that captures the attention information. Additionally, we propose a cross-attention (CA) mechanism to enhance the speech feature representations, generating a speaker extraction mask. Experimental results on a publicly available dataset demonstrate that our proposed model outperforms two baseline models across various evaluation metrics.



Speaker extraction, EEG, selective auditory attention, cocktail party effect




§ INTRODUCTION


Humans inherently can focus on a specific audio source amidst multiple competing sources. This remarkable ability is referred to as selective auditory attention (SAA) in cocktail party scenarios <cit.>. It was found that two regions of the auditory cortex manage such selective auditory attention. They allow us to separate and enhance the interested voices. However, when we suffer from hearing loss or other forms of impairments, we find it hard to distinguish voices <cit.> because our ears are missing some frequencies. Despite much progress, existing algorithms still struggle to isolate and enhance the targeted speech amidst background noise effectively <cit.>. 

Studies in neuroscience have revealed a strong correlation between the attended speech and the neural response it elicits <cit.>, paving the way for auditory attention detection (AAD) from neural activities <cit.>. Such neural responses include Electrocorticography (ECoG) <cit.>, Magnetoencephalography (MEG) <cit.>, and EEG <cit.>. Among them, EEG provides a non-invasive, comparably easy-to-wear, and affordable choice for SAA tasks, especially for neuro-steered hearing aids <cit.>.


Speech separation and speaker extraction emulate human SAA to solve cocktail party problems. Speech separation algorithms are designed to separate a speech mixture into individual speech streams <cit.>. Conversely, speaker extraction <cit.> algorithms extract a target speaker's voice associated with an auxiliary reference cue. This cue serves as a distinct marker indicating the specific speech signal of interest to be isolated, typically providing identifiable information about the target or attended speech. Various auxiliary reference cues have been explored to guide speaker extraction systems. Among them, pre-enrolled speech signal <cit.> from the interested speaker is a commonly used reference signal. Moreover, inspired by human attention mechanisms, visual reference cues derived from video streams capturing hand and body gestures <cit.>, lip movements <cit.>, and direction information <cit.> have been studied. Furthermore, there's a growing interest in multimodal approaches that integrate audio-visual cues <cit.>. However, in real-world scenarios, it is not feasible to always access pre-enrolled speech from numerous speakers, and visually tracking the target speaker is often impractical.

In this paper, we seek to use the elicited EEG signal as the sole reference cue as it provides feedback from the human brain regarding attended speech. We hypothesize that the neural response can effectively inform a speaker extraction system about the content of the attended speech in real-time. Therefore, we propose a novel end-to-end speaker extraction model at the utterance level in this work, which utilizes the EEG signal temporally aligned with speech as the auxiliary reference cue. We train the model in a trial-independent setting on a public dataset and compare it with several baseline models on multiple evaluation metrics.

This paper is organized as follows. Section 2 summarizes the related work; Section 3 describes the architecture of our proposed model; Section 4 outlines the experimental setup, including the dataset and baseline models. We report our results in Section 5 and conclude the study in Section 6.



§ RELATED WORK



Recent advancements in EEG-based AAD and speech separation have opened a path to utilize EEG signals for separate attended speech in a multi-source speech mixture environment. The mixture is first separated into multiple single-speech signals and then the signal with the highest correlation with the EEG signal is selected as the attended speech <cit.>. However, these methods are limited by the requirement of the number of speakers to be known in advance and the high computational consumption during separation.

In the studies of AAD, a clean speech signal is often compared with the EEG signal to find their correlation. It was shown that <cit.> the performance of the AAD systems improves as the decision window size increases. However, such clean single-speech signals are not always available in real-world scenarios. This calls for the study on how to exploit the information of the attended speech from within the EEG signals. There have been studies on reconstructing the attended speech envelope <cit.> from EEG signals. This allows us to establish a relationship between the speech stimulus and its neural responses.  

Multiple end-to-end time domain studies have been proposed addressing different aspects. The Brain-Informed Speech Separation (BISS) <cit.> uses the reconstructed attended speech envelope from the EEG signal as the reference cue for speaker extraction. Inspired by BISS, the Brain-Enhanced Speech Denoiser (BESD) <cit.> and follow-up work U-shaped BESD (UBESD) <cit.> followed a dual-module approach based on Temporal Convolutional Neural network (TCN) to model EEG signals with speech together to denoise the speech mixture to obtain the attended speech. The Brain Assistend Speech Enhancement Network (BASEN) <cit.> used a TCN and Convolutional multi-layer Cross-Attention (CMCA) module for feature extraction and fusion, respectively. As a follow-up work, sparsity-driver BASEN <cit.> has proposed two EEG channel selection methods during speech enhancement: residual Gumbel selection <cit.> and convolutional regularization selection. 

Most recently, the neuro-steered speaker extraction (NeuroHeed) <cit.> was introduced, which consists of a self-attention (SA) <cit.> based EEG encoder that generates the reference cue. The speaker extraction task is performed in either an online or offline manner. The online system adopted auto-regressive feedback from previously extracted speech. Following up, the NeuroHeed+ <cit.> was introduced. It added an auxiliary AAD module to NeuroHeed to reduce the speaker confusion error. 

Despite much progress, it has not been explored in the prior work how both spatial and temporal information from EEG signals can be exploited to build knowledge-rich clues. Moreover, previous research has not studied how to fuse EEG and speech embeddings to make use of the temporal correlation and complementary information of two types of signals during speech mask generation for speaker extraction. We are motivated to explore the above unused potentials for neuro-guided speaker extraction.











§ METHODOLOGY



We propose a neuro-guided speaker extraction algorithm, an end-to-end EEG-based speaker extraction model or NeuroSpex, comprising four components: speech encoder, EEG encoder, speaker extractor, and speech decoder as depicted in Fig. 1. This model is built based on ConvTasNet <cit.>, consisting of a speech encoder, separator, and decoder. Our model also has an EEG encoder module to guide the extractor by providing information on the attended speech. Given the mixture of attended and interfering speech in the time domain, our model aims to extract the attended speech using elicited neural response as the sole reference cue. 





 §.§ Speech Encoder


The speech encoder transforms the single-channel input mixture signal segment x ∈ℝ^T_s into a sequence of utterance-based embeddings X ∈ℝ^N_x× T_x, similar to the frequency analysis with short-time Fourier transform. This is achieved by applying time domain speech encoding <cit.>, that is a 1D Convolutional (Conv1D) layer followed by Rectified Linear units (ReLu) as the activation function:

    X = ReLu(Conv1D(x,1,N,K)) ∈ℝ^N_x× T_x


Here, the input channel size is 1 for the single-channel speech utterance with a sequence length T_s, which is a 4-second segment sampled at 8 kHz (as discussed in Section 4.1). The output X is a sequence of T_x embeddings, each with N_x dimensions, which are 3200 and 256, respectively. The kernel size K of the Conv1D is set to 20, while stride and padding are set to K/2. 





 §.§ EEG Encoder

The proposed EEG encoder seeks to exploit complementary information of the attended speech contained in the multi-channel EEG signals and develops the EEG  embeddings or the reference signal to guide the speaker extractor. The EEG encoder takes a 4-second segment of a 64-channel EEG signal y, sampled at 128 Hz (detailed in Section 4.1), as input and generates Y ∈ℝ^N_y× T_y, where T_y = 512 and N_y = 64. This output Y serves as the reference signal for the speaker extractor. 
    
The EEG encoder has a pre-convolutional layer followed by a series of multi-head attention and depth convolution blocks that serve as AdC blocks as illustrated in Fig. <ref> (b). Specifically, the pre-convolutional layer (preConv) acts as a preliminary feature extractor, which extracts important features for subsequent layers from EEG signal segment y. Then we have n number of sequentially layered stack of AdC blocks, where each Adc block consists of a multi-head attention <cit.> and 1D depthwise-convolutional layer <cit.>. We employ multi-head attention due to its ability to effectively capture temporal dependencies from sequential data as applied in previous SAA research <cit.>. Furthermore, motivated by recent work <cit.> to encode EEG signals, we use depth-wise convolutions to capture frequency-specific spatial information from channels. Both multi-head attention (mHA) and depth-convolutional (dConv) layers have residual connections followed by layer normalization (LN). That is,

    Y_0 = preConv(y) ∈ℝ^N_y× T_y


    Y_i = AdC_i(Y_i-1) ∈ℝ^N_y× T_y

where Y_0 is the output of the preConv layer that is fed to the first AdC block (AdC_1). Y_i is the output of i^th AdC block where i ∈{1,...,n }. Each AdC block is designed as follows:

    Y_mha = LN_i^mHA( mHA_i(Y_i-1,N,H) + Y_i-1) ∈ℝ^N_y× T_y


    Y_i = LN_i^dConv( dConv_i(Y_mha,N,K) + Y_mha) ∈ℝ^N_y× T_y

where Y_mha is the output of the  multi-head attention and N is the channel size that is 64 for all cases. The number of heads of the attention layer H and the kernel size of the depth-convolutional layer K are set to 2 and 10, respectively. The EEG encoder outputs Y from the final AdC block.












































 §.§ Speaker Extractor


The speaker extractor generates the estimation mask to separate the attended speech from the mixture background. The masked speech S is created by element-wise multiplication of speech mixture embeddings X and the generated mask M as shown in Equation (<ref>). The extracted speech has an embedding sequence length T_x of dimension N_x, the same as the input speech mixture embedding X.

    S = X ⊗ M ∈ℝ^N_x× T_x

As shown in Fig. 1, we propose a speaker extractor based on the ConvTasNet backbone with TCNs <cit.> to increase the receptive field, and CA modules to fuse speech mixture and EEG embeddings as shown in Fig. 2 (a). 
We adopt cross-modal attention <cit.> to integrate information from both the auditory stimuli (speech mixture) and brain responses (EEG), mirroring the brain's natural process of speech mixture perception during speech comprehension. 

The cross-attention (CA) mechanism is designed to combine and enhance insights from the key-value given the query <cit.>. Before fusing EEG and speech mixture embeddings, the EEG embedding (i.e., the reference signal Y) is linearly interpolated from length T_y to T_x (512 to 3200) to match the embedding sequence length required by the CA block, thereby generating Y^'∈ℝ^N_y× T_x. Subsequently, the key and value inputs to the CA block are the speech mixture embeddings X, while the query input is the interpolated reference signal Y^'. The output of the CA block X_ca is computed as shown in Equation <ref>, where Q, K, and V are query, key, and value, respectively. The X_ca also has the same dimensionality of X and is then fed to the TCN block with a residual connection.

    X_ca = CA(K=X, V=X, Q=Y^') ∈ℝ^N_x× T_x

As shown in Fig. 1, we repeat CA and TCN pairs 4 times (m = 4) in the speaker extractor. 











 §.§ Speech Decoder


The speech decoder reconstructs time-domain single-channel speech waveform s from masked speech embeddings S. It performs an inverse operation of the speech encoder by passing S through a linear layer (linear) and an Overlap-and-Add (OvlpAd) function to restore the audio signal from speech representation. That is,

    s = OvlpAd(linear(S,N,L), L/2) ∈ℝ^T_s

where T_s denotes the length of the output speech utterance. The linear layer has input size of N and output size of L, which are 64 and 20, respectively. The Overlap-and-Add operation also has a L/2 frame shift.



 §.§ Loss Function


We train the proposed model end-to-end using the scale-invariant signal-to-distortion ratio (SI-SDR) <cit.> between the reference attended speech signal s_t and the extracted speech signal s_e. SI-SDR loss is typically used in time-domain speaker extraction tasks <cit.>. The extracted speech is scaled to remove changes that occurred during the reconstruction and helps maintain stability during the training phase. It is calculated with dB and a higher SI-SDR indicates better speech quality. Therefore, a negative SI-SDR is used as the loss function to train the model. The SI-SDR is defined as follows:

    ℒ_SI-SDR(s_t,s_e) = -10log_10(∥s_e^Ts_t/∥ s_t ∥^2s_t ∥^2/∥ s_e- s_e^Ts_t/∥ s_t ∥^2s_t ∥^2)




§ EXPERIMENTAL SETUP





 §.§ Dataset


We used the publicly available KULeuven (KUL) <cit.> dataset, which includes EEG signals from 16 normal-hearing subjects, collected using the BioSemi ActiveTwo system with 64 channels at a sampling rate of 8,192 Hz. The dataset comprises 20 trials per subject, though our experiments used only the first 8 to avoid repetition in attending to the same speech stimulus. Each trial involves subjects listening to simultaneous speech recordings from two male speakers narrating Dutch short stories, delivered dichotically via plugged-in earphones. Subjects focused on the speech from one speaker while ignoring the other. The stimuli, generated from four stories, were balanced for loudness and presented in randomized speaker direction and identity. The speech signals are sampled at 8 kHz and mixture signals are created by mixing attended and unattended speech signals at 0 dB to have the same power for both speakers.

EEG recordings were referenced to the average of all electrodes, band-pass filtered between 1 and 32 Hz, and downsampled to 128 Hz. This frequency range was selected to align with auditory attention detection protocols. Data normalization was performed trial-by-trial to standardize mean and variance across the dataset. In total, the dataset comprises 128 trials amounting to 12.8 hours of speech-EEG parallel data.

We followed a speaker-dependent trial-independent training approach to train all configurations of the proposed NeuroSpex and baseline models. Therefore, we divided the data into three sets: train, test, and validation. For the test set, we randomly selected 1 trial from each subject, thus having 16 trials for all subjects. Since each trial has a pair of attended and unattended audios, we made sure no same pair is chosen more than twice from all subjects. Similarly, from the rest of the data, we randomly selected 4 trials to form the validation set. The remaining trials are used for the training set.

Each trial was cut into 4-second segments with a hop length of 1 second for all sets based on experimental observations and computational resources. This segmentation approach yielded 5,712 segments for the test set, 1,428 segments for the validation set, and 38,556 segments for the training set. Specifically, the test set comprised data from 16 subjects, with each subject contributing 357 segments, totaling 5,712 segments, each lasting 4 seconds. We train the model on the training data and evaluate its performance on the validation set at the end of each epoch to monitor progress and adjust hyper-parameters accordingly. After training, the test set is used to obtain the final evaluation metrics to report the performance.





 §.§ Evaluation Metrics


We employ three evaluation metrics and an extension of one of them. SI-SDR <cit.> and SI-SDR improvement (SI-SDRi) which quantifies the improvement in the quality of the extracted speech signal compared to the mixture signal. Perceptual Evaluation of Speech Quality (PESQ) <cit.> assesses the quality of extracted speech by comparing it to the clean single speech signal, providing a score indicating intelligibility and naturalness. Short-Term Objective Intelligibility (STOI) <cit.> evaluates the intelligibility of extracted speech by comparing it to the clean single speech signal, providing a measure of speech clarity and understandability. All evaluation metrics are higher the better.



 §.§ Baseline Models


We benchmark NeuroSpex against several baseline models. The baselines are justified as follows.

Firstly, we use blind speech separation with permutation invariant training BSS (PIT) <cit.> model based on DPRNN <cit.>. This model blindly separates a multi-talker speech mixture into multiple single-talker streams. It then selects the target speech by comparing the best permutation with the target speech based on the SI-SDR metric, without relying on a reference cue to guide the separation. This method is optimized to separate speech streams in a discriminative manner. It represents the full speaker extraction potential when we have perfect neural decoding. Thus, it is seen as the upper bound of NeuroSpex in terms of speaker extraction performance.

Secondly, two versions of NeuroHeed are considered, which operate with a speaker extractor. They are DPRNN <cit.> and ConvTasNet <cit.> based on recurrent neural networks and TCNs, respectively. NeuroHeed is the current state-of-the-art for neural response-based speaker extraction. NeuroHeed with ConvTasNet is similar to NeruroSpex in terms of model architecture, while NeuroHead with DPRNN has fewer parameters than NeuroSpex. They make two relevant baselines for our benchmarking.

Thirdly, we use the BASEN model which also adopts TCN and CMCA EEG-Speech processing in terms of model architecture. Each model has been used in its vanilla form and trained in the same conditions as NeuroSpex. BASEN is being chosen as a baseline also because it represents the state of the art before NeuroHeed, and it also adopts TCN and CA in its architecture. 



 §.§ Model Training


All implementations utilize the PyTorch framework with distributed training and data distributed sampler techniques across 2 Nvidia RTX A6000 GPUs. To ensure reproducibility, we used random seeds to generate consistent results. The models are trained end-to-end using the Adam optimizer with an initial learning rate of 0.0001. A learning rate scheduler with a decay factor of 0.5 when the best validation loss does not improve within 5 consecutive epochs and early stopping is applied when the best validation loss does not improve within the last 25 epochs. Training is conducted for around 100 epochs or until stable training with batch size of 16. 
















§ RESULTS






In this section, we discuss the empirical evaluations of our study by comparing our model with baseline models, exploring different model configurations, and conducting an ablation study. Performance on the trial-independent test and validation sets is reported using the evaluation metrics outlined in section <ref>. The results are summarized in tables and illustrated with SI-SDRi violin plots. Statistical significance was assessed using paired t-tests. 





 §.§ Ablation Study






We conducted the ablation study to understand the effect of the contributing components. Table <ref> shows the performance of models, evaluated using multiple metrics on the validation set, for different EEG encoders and feature fusion methods including direct EEG signal input and direct fusion. Note that we only used a single AdC block (AdC_1) in the EEG encoder. The results mainly show that introducing CA to speech-EEG embeddings fusion increases the performance on every evaluation metric. Furthermore, AdC block also improves the performance over SA used in NeuroHeed and Convolution-based EEG encoders.











 §.§ Effect of AdC blocks


We evaluated the effect of the number of AdC blocks in the EEG encoder to find the appropriate number. Table <ref> reports the performance for 5 progressive number of AdC blocks. For all evaluation metrics, 6 AdC blocks return the best performance. This shows that the number of AdC blocks increases the performance from 1 to 6 blocks with smaller margins. Hence, we present our best-proposed model with 6 AdC blocks. Furthermore, Fig. 3 depicts the violin plots of SI-SDRi for each of the 16 subjects in the test set for our best-proposed model. All subjects show consistent and centered distribution with smaller variances except for subjects 8, 11, and 14.



 §.§ Comparison with Baseline Models


We compare our best model (with 6 AdC blocks) with multiple baseline models as shown in Table 3 including the main architectural difference between models and evaluation metrics. The violin plots in Fig. 4 summarize the SI-SDRi for all trials in the test set for all models. The BSS (PIT) shows the best results with the least variance as it performs direct speech separation and sets an upper bound for the comparison. Our proposed model outperforms all baselines significantly (p < 0.001) except BSS (PIT) on all evaluation metrics showing that it performs the speaker extraction with better signal quality perpetual quality, and intelligibility. Furthermore, the proposed NeuroSpex model contains fewer parameters compared to the ConvTasNet based NeuroHeed model but more than DPRNN counterparts due to the TCN based speaker extractor. According to violin plots, the proposed model has a better-centered distribution around a higher median and less variance compared to both NeuroHeed and BASEN baselines.





§ CONCLUSION


In this study, we propose an end-to-end speaker extraction model operating in the time domain. The model utilizes neural responses as a reference cue to extract attended speech in a cocktail party scenario. The novel contributions of this work include multi-head attention and depth-wise convolution-based EEG encoding, and CA-based EEG-speech embeddings fusion, which seek to enhance the overall quality of speaker extraction. The results demonstrate significant improvements in extraction performance over several competitive baseline models. Thus, we show that our proposed model effectively extracts EEG embeddings correlated with attended speech and achieves superior speech-EEG feature fusion to generate the speaker extraction mask, hence validating our hypothesis. For future research, we recommend exploring speaker-specific information during extraction and conducting subject-independent studies to enhance generalizability and realism.













IEEEbib




