Relating EEG to continuous speech using deep neural networks: a review
Objective. When a person listens to continuous speech, a corresponding
response is elicited in the brain and can be recorded using
electroencephalography (EEG). Linear models are presently used to relate the
EEG recording to the corresponding speech signal. The ability of linear models
to find a mapping between these two signals is used as a measure of neural
tracking of speech. Such models are limited as they assume linearity in the
EEG-speech relationship, which omits the nonlinear dynamics of the brain. As an
alternative, deep learning models have recently been used to relate EEG to
continuous speech. Approach. This paper reviews and comments on
deep-learning-based studies that relate EEG to continuous speech in single- or
multiple-speakers paradigms. We point out recurrent methodological pitfalls and
the need for a standard benchmark of model analysis. Main results. We gathered
29 studies. The main methodological issues we found are biased
cross-validations, data leakage leading to over-fitted models, or
disproportionate data size compared to the model's complexity. In addition, we
address requirements for a standard benchmark model analysis, such as public
datasets, common evaluation metrics, and good practices for the match-mismatch
task. Significance. We present a review paper summarizing the main
deep-learning-based studies that relate EEG to speech while addressing
methodological pitfalls and important considerations for this newly expanding
field. Our study is particularly relevant given the growing application of deep
learning in EEG-speech decoding.
Corentin Puffay, Bernd Accou, Lies Bollens, Mohammad Jalilpour Monesi, Jonas Vanthornhout, Hugo Van hamme, Tom Francart
# Relating EEG to continuous speech using deep neural networks: a review.

###### Abstract

_Objective._ When a person listens to continuous speech, a corresponding response is elicited in the brain and can be recorded using electroencephalography (EEG). Linear models are presently used to relate the EEG recording to the corresponding speech signal. The ability of linear models to find a mapping between these two signals is used as a measure of neural tracking of speech. Such models are limited as they assume linearity in the EEG-speech relationship, which omits the nonlinear dynamics of the brain. As an alternative, deep learning models have recently been used to relate EEG to continuous speech, especially in auditory attention decoding (AAD) and single-speech-source paradigms. _Approach._ This paper reviews and comments on deep-learning-based studies that relate EEG to continuous speech in AAD and single-speech-source paradigms. We point out recurrent methodological pitfalls and the need for a standard benchmark of model analysis. _Main results._ We gathered 29 studies. The main methodological issues we found are biased cross-validations, data leakage leading to over-fitted models, or disproportionate data size compared to the model's complexity. In addition, we address requirements for a standard benchmark model analysis, such as public datasets, common evaluation metrics, and good practices for the match-mismatch task. _Significance._ We are the first to present a review paper summarizing the main deep-learning-based studies that relate EEG to speech while addressing methodological pitfalls and important considerations for this newly expanding field. Our study is particularly relevant given the growing application of deep learning in EEG-speech decoding.

## 1 Introduction

Electroencephalography (EEG) is a non-invasive method that measures the electrical activity in the brain. When a person listens to speech, the EEG signal measured has been shown to contain information related to different features of the presented continuous speech. We can relate these speech features to the EEG activity using machine learning models to investigate if and how the brain processes continuous speech. Reasons for doing this include (1) understanding neural mechanisms of speech processing in the brain; (2) objectively measuring processes in the brain
related to speech processing using single sound source stimulus-response models, which in turn are useful for research and clinical diagnostics of hearing; (3) designing auditory prostheses that incorporate auditory attention decoding (AAD) and use it to steer noise suppression to improve speech understanding in so-called cocktail party scenarios. AAD systems are designed to decode to which speaker from a mixture a listener attends; and (4) providing architectures from EEG speech processing modeling to interested readers from any field.

Currently, primarily linear models are used to relate continuous speech to EEG (e.g., Ding and Simon, 2012; Vanthornhout et al., 2018; Crosse et al., 2016; de Cheveigne et al., 2018; Crosse et al., 2021). Such models are used to either predict EEG from speech (forward modeling) or to reconstruct speech from EEG (backward modeling). Once the EEG (or speech) is approximated, a correspondence measure between the predicted and the ground truth signal is computed and considered a measure of neural tracking. For single sound source stimulus-response models, the end-task is to maximize the prediction (or reconstruction) quality, whereas for AAD models, the task is to decode the attended speaker or direction.

For single sound source stimulus-response models, the main correspondence measure for prediction (or reconstruction) quality is Pearson correlation, and it can be used either between the predicted and ground truth signal, or between two maximally-correlated embedded representations as implemented in the canonical component analysis (CCA) method (de Cheveigne et al., 2018). However, comparison of correlations across experiments is delicate if the preprocessing of the target signals is different. As an example, predicting an EEG in a narrow frequency band is easier than predicting the whole EEG spectrum. This implies that to maximize correlation, one can decide to filter the EEG in a given frequency band, while some speech-related information is contained in the filtered-out EEG (de Cheveigne et al. (2021)).

To circumvent that issue, an simple objective task called the match-mismatch (MM) task (de Cheveigne et al., 2018), was developed. This classification task requires a model to decide whether a segment of EEG was evoked by a given segment of speech or not.

Linear models are limited as they assume linearity between the EEG and speech signals, inadequately fitting the nonlinear nature of the auditory system. For example, it is well known that depending on the level of attention and state of arousal of a person response latencies can change (Ding and Simon, 2012), which cannot be modeled with a single linear model.

Deep neural networks (DNNs) have been recently introduced to this field. Many studies have shown the ability of deep learning models to relate EEG to speech (see Section 2), be it for neural tracking assessment (e.g., Katthi and Ganapathy, 2021; Accou et al., 2021; Monesi et al., 2020; Thornton et al., 2022) or to decode auditory attention (e.g., de Taillez et al., 2020; Ciccarelli et al., 2019; Kuruvila et al., 2021). On certain tasks, DNNs have outperformed linear decoder baselines (e.g., Accou et al., 2023, 2021; Monesi et al., 2020; Puffay et al., 2023), but it is still not a general finding.

In this paper, we summarize the methods present in the literature to relate EEG to continuous speech using deep learning models. In Section 2, we first review different experiment steps of the gathered studies and the different approaches chosen by authors. These steps include the task used to relate EEG to speech, the network
architecture used, the dataset's nature, the preprocessing methods employed, the dataset segmentation, and the evaluation metric. In Section 3, we then address the methodological pitfalls to avoid when using such models and we recommend to establish a standard benchmark for models' analyses. In Section 7 and 8, we summarize the results of each study individually for multiple sound source and single sound source respectively.

## 2 Review of deep-learning-based studies to relate EEG to continuous speech

Using Google Scholar, IEEE Xplore, Science Direct, Pubmed and Web of Science, we collected papers using search queries reported in Table 1. As a last step, we pruned the selection manually to exclude studies not including EEG data, continuous speech stimuli or deep learning models, and stopped searching for new papers in December 2022.

In this section, we go through different features of the gathered studies including the task used to relate EEG to speech, the different architectures used, the dataset's nature, the preprocessing methods employed, the dataset segmentation, and the evaluation metrics. More detailed summaries of individual studies can be found in Section 7 and 8.

\begin{table}
\begin{tabular}{l l} \hline Search engine & Search query \\ \hline Google Scholar & (”EEG” OR ”Electroencephalography” OR ”Electroencephalogram”) AND speech \\  & AND (”deep learning” OR ”deep neural networks”) \\ IEEE Xplore & ((”All Metadata”:EEG) OR (”AlklMetadata”:Electroencephalogram)) \\  & AND (”All Metadata”:speech) \\  & AND ((”All Metadata”:deep neural networks) \\  & OR (”All Metadata”: non-linear) OR (”All Metadata”: nonlinear) ) \\ Science Direct & (EEG OR Electroencephalogram) AND (”continuous speech” OR ”natural speech”) \\  & AND ( ”neural network”)) \\ Pubmed & (EEG OR Electroencephalography OR Electroencephalogram) \\  & AND (speech) AND ( deeplearning OR deep learning OR neural networks) \\  & NOT (imagined[title]) NOT (motor[title]) NOT emotion[title] \\ Web of Science & (((((((((TS=((”EEG” or ”Electroencephalography” or ”Electroencephalogram”)))) AND TS=((”speech” or ”audio” or ”auditory” )))) \\  & AND TS=((”artificial neural network*” or ”ANN” or ”deep learning” or ”deeplearning” or ”CNN” or ”convolutional” or ”recurrent” or ”LSTM”)))) NOT TI=(”imagined” or ”motor imagery” or ”parkinson”)) \\  & NOT TI=(”emotion”)) NOT TS=((”dysphasia” or ”alzeimer**”)))) \\  & NOT TS=(”seizure”)))) AND DOP=(2010-01-01/2022-05-15) \\ \hline \end{tabular}
\end{table}
Table 1: Search queries for each search engine during paper collection.
### Tasks relating EEG to speech

To relate EEG to speech, we identified two main tasks, either involving multiple simultaneous speech sources or a single speech source.

#### 2.1.1 Multiple sound sources

When more than one speaker talks simultaneously, the brain of a listener must cope with multiple speech sources. One of the main challenges arising from this scenario is AAD. The interest in this topic is two-fold: it provides a basis to overcome current hearing aid limitations in cocktail party scenarios, but also to investigate attention mechanisms in the brain. Linear methods have been widely used for AAD (review by Geirnaert et al. (2021c)), and we here solely investigate deep-learning-based studies that we report in Table 2.

To decode attention, one possibility is to classify the speaker identity (i.e., direct classification). In the majority of studies, the focus has been on identifying the left or right speaker in a situation where two speakers are competing for attention. From 2016 to 2020, only three studies were reported (Shree et al., 2016; de Taillez et al., 2020; Tian and Ma, 2020), however since 2021 this has increased a lot(Su et al., 2021; Kuruvila et al., 2021; Lu et al., 2021; Zakeri and Geravanchizadeh, 2021; Vandecappelle et al., 2021; Hosseini et al., 2021; Xu et al., 2022b, a; Thornton et al., 2022a).

Another approach is to decode the directional focus of attention. It presents advantages such as avoiding separating speech sources, but also enabling models to use brain lateralization (i.e. the directional focus is encoded spatially in the brain), which is an instantaneous spatial feature, rather than a temporal feature. We report only one deep-learning-based study attempting to decode locus of attention (Vandecappelle et al., 2021).

#### 2.1.2 Single sound source

Another popular task is to relate EEG to a single speech source. This approach usually aims to quantify the time-locking of a brain response to a single speech source, often referred to as neural tracking. Neural tracking of speech can be used in multiple applications, notably to model the speech processing in the brain but also as an objective measure of hearing or understanding speech.

We identified two main approaches: using a match-mismatch task, and direct regression of stimulus or EEG (see Table 3).

In the match-mismatch (MM) paradigm (de Cheveigne et al., 2018) a model is trained to associate a segment of EEG with the corresponding segment of speech. The accuracy obtained on this task is defined as a measure of neural tracking. We report four studies in which the MM task was used (Monesi et al., 2020, 2021; Accou et al., 2021b, a).

EEG can also be related to speech in a reconstruction/prediction (R/P) task. In this case a stimulus feature is reconstructed from the EEG (or the EEG predicted from the speech, respectively), and correlated to the original signal. This relates to the commonly used linear backward (or forward, respectively) models. Four studies concern backward models (Krishna et al., 2021a, b; Sakthi et al., 2019; Thornton et al., 2022a), while only one was reported for forward modeling (Krishna et al., 2021a).

In another linear approach, canonical component analysis (CCA), the stimulus and
EEG are projected to separate subspaces and correlated in those subspaces de Cheveigne et al. (2018). Variations on this method have also been explored with deep learning (Reddy Katthi and Ganapathy, 2021; Katthi and Ganapathy, 2021).

The MM and the R/P tasks were the most used methods, however some studies used different tasks to relate EEG to speech such as semantic incongruuities classification (Motomura et al., 2020), or sentence classification (Sakthi et al., 2021). Bollens et al. (2022) utilized solely an embedded representation to classify segments of speech.

### Model architectures

The field of deep learning is evolving rapidly, and constantly providing novel architectures. Multiple layer types were integrated to AAD and single speech source decoding models. Globally, architectures to solve the tasks mentioned in Section 2.1 were inspired from other fields (e.g., automatic speech recognition, ASR). We provide a more in-depth description of each architecture in Section 6.

Early attempts used general regression neural networks (GRNNs) (Shree et al., 2016) or fully-connected neural network (FCNN) models (de Taillez et al., 2020). As fully connected layers are very computationally expensive, later studies implemented convolutional neural network (CNN)-based models (Ciccarelli et al., 2019; Tian and Ma, 2020; Thornton et al., 2022; Vandecappelle et al., 2021).

As an attempt to compensate for the delay in the brain response to speech, models with recurrent layers such as long-short term memory (LSTM) (Monesi et al., 2020, 2021; Kuruvila et al., 2021; Lu et al., 2021; Xu et al., 2022b), Bi-LSTM (Zakeri and Gervanchizadeh, 2021), gated recurrent unit (GRU) (Krishna et al., 2020, 2021b, a; Sakthi et al., 2021) or Bi-GRU (Motomura et al., 2020) were implemented.

To enable the model to provide more weight to certain time points (Motomura et al., 2020; Krishna et al., 2021), or certain EEG electrodes (Su et al., 2021), channel attention mechanisms were integrated into the models. In one study (Xu et al., 2022a), channel attention was integrated into a transformer, an well-known architecture from automatic speech recognition (ASR) that allows parallel computation (to reduce training time), and reduces performance drops due to long dependencies (Vaswani et al., 2017a).

Other popular model types such as generative adversarial networks (GANs) (Krishna et al., 2021b), or autoencoders (AEs) Bollens et al. (2022); Hosseini et al. (2021) were utilized. AEs find a compressed meaningful representation of a signal. They can be constrained to extract speech-related information in EEG, hence working as a denoiser.

### Datasets

As explained in Section 2.2, deep learning architectures have very useful properties to relate EEG to speech. Compared to linear models, they often have a high number of parameters, which means lots of data are required to train properly, and to avoid overfitting. As collecting EEG data is tedious and time-consuming, research groups often work with their own small datasets, sometimes containing a few minutes of speech or less per participant (Lu et al., 2021; Shree et al., 2016; Krishna et al., 2020,
2021a,b; Tian and Ma, 2020). The other studies we reported used at least 30 min of data per subject, while some even published their datasets (Das et al., 2019; Fuglsang et al., 2018; Bollens et al., 2023a). We discuss the importance of the dataset and generalization in Section 3.3.

### Preprocessing

Once a dataset is available, both the EEG and the presented speech can be preprocessed in various manners.

#### 2.4.1 Eeg

The preprocessing of EEG signals varies across studies. While it is not specific to deep learning, many models can learn to conduct steps that would traditionally be considered preprocessing. Hence we provide a general EEG preprocessing procedure, and invite readers to read the original individual studies to get more specific details. In addition, extensive considerations about preprocessing are reported by Crosse et al. (2021).

Most of the studies we reviewed start with filtering the signal, first with a high-pass filter to remove any unwanted DC shifts or slow drift potentials, secondly with a low-pass filter to remove high frequencies as the SNR becomes lower in higher frequency ranges. In linear studies, models typically use low frequencies (e.g., between 0.5 and 8 Hz), while some deep learning studies report benefits from including higher frequencies (Puffay et al., 2022). A re-referencing step can then be added, typically by subtracting the mean over all channels from each individual channel. This contributes to increase the signal-to-noise ration (SNR). Downsampling is commonly performed to reduce the computational time during training. It is often done in accordance with previous filtering to avoid temporal aliasing as mentioned in Crosse et al. (2021). Typical sampling rates are 128 or 64 Hz. Finally, an artifact removal algorithm based on different methods such as multi-channel Wiener filtering (MWF) (Somers et al., 2018), or independent component analysis (ICA) (Hyvarinen and Oja, 2000) is used to remove different artifacts (e.g., eye-blink, neck movement).

The majority of the studies we reviewed provided their models with EEG signals pre-processed as stated above. However, two of them engineered more specific features from EEG, such as a latent representation optimized through the training of an AE (Bollens et al., 2022), or source-spatial feature images (Tian and Ma, 2020).

#### 2.4.2 Speech

From the raw speech signal, it is common to investigate the neural tracking of different features of speech. Most studies we report here used acoustic features, such as the speech envelope (e.g, Ciccarelli et al., 2019; Su et al., 2021; de Taillez et al., 2020; Lu et al., 2021; Xu et al., 2022b,a), or the Mel spectrogram (e.g., Krishna et al., 2020, 2021a,b; Monesi et al., 2021; Kuruvila et al., 2021). A study even used the fundamental frequency of the voice (Puffay et al., 2022).

To investigate the processing of different speech units in the brain, higher-level speech features at the level of the sentence (Motomura et al., 2020; Sakthi et al., 2021), word (Monesi et al., 2021) or phonemes (Sakthi et al., 2021; Monesi et al., 2021) were also used.
As mentioned by Crosse et al. (2021), these preprocessing steps should be conducted on the entire dataset or on the training, validation and test sets prior to segmenting data to avoid introducing artifacts that could affect the model's performance.

### Data segmentation

The training paradigm is crucial and should be carefully chosen to avoid biases and overfitting. In the gathered studies, two methods were employed: cross-validation (e.g., Ciccarelli et al., 2019), and regular training, validation, and test sets split (e.g., Monesi et al., 2021; de Taillez et al., 2020). Cross-validation performs multiple training iterations with different portions of the dataset used for training and validation, while regular training uses a unique training and validation sets.

In AAD tasks, trials are defined as a period of time during which a subject attends to a target speaker, and are labelled accordingly (e.g., left/right). The split can therefore be done in different manners, notably within (e.g., Lu et al., 2021) and between trials (e.g., Thornton et al., 2022a). The split choice can lead to overfitting, as discussed in Section 3.2.

### Evaluation metric

Although various intermediate metrics can be used to select the attended speaker or locus, the reported metric is common in the multiple sound sources studies we report. The attention decoding (i.e., speaker identity or direction classification) accuracy is used in all studies. The only metric difference is the chance level defined by the number of sound sources.

For single sound source paradigms, various metrics are employed. Classification metrics are used, such as the match-mismatch accuracy (Monesi et al., 2020, 2021; Accou et al., 2021b, a), subject-classification accuracy (Bollens et al., 2022) or sentence classification accuracy (Motomura et al., 2020). For regression studies, the main metric we noted is Pearson correlation (Katthi et al., 2020; Katthi and Ganapathy, 2021; Thornton et al., 2022a), while one research group used root-mean-squared error (RMSE) and Mel cepstral distortion (MCD) (Krishna et al., 2020, 2021a, b).

The use of multiple metrics is problematic when comparing model performances across studies. For instance, even when the same metric is used, it does not guarantee a fair comparison (see Section 3.4).


## 3 Overfitting, interpretation of results, recommendations

### Preamble

In our own practice with auditory EEG we noticed how easily the deep learning models overfit to specific trials, subjects or datasets. This is mainly due to the relatively small amount of data typically available, compared to other domains such as image or speech recognition. A very careful selection of the test set is therefore needed, and the results of a number of the studies reviewed above may be overly optimistic. In the following experiments, we demonstrate how this can happen and propose a number of good practices to avoid overfitting and how to calculate results on a sufficiently independent test set. We first introduce two different datasets which we use for respectively single speech source or multiple speech source tasks.

#### 3.1.1 Single speech source (N=1) dataset

For the single speech source experiments, namely subsections 3.4, 3.5 and 3.6 below, we select a publicly available dataset (1). We selected 48 subjects from a dataset that now consists of EEG data from 85 normal hearing subjects while they listened to 10 unique stories of roughly 14 minutes each, narrated in Flemish (Dutch).

We use the LSTM-based model proposed by Monesi et al. (2020) trained on the match-mismatch task defined in Section 6. An exception is made for subsection 3.4 which uses linear decoders as correlation analyses require a regression task rather than a match-mismatch task. We use linear decoders in this subsection as they face the same issue as DNNs but are computationally less expensive.

#### 3.1.2 Multiple speech sources (N\(>\)1) dataset

For the multiple-speech source experiment, namely subsection 3.2 below, we use the publicly available dataset from Das et al. (2019). It contains data from 16 subjects. In total, there are 4 Dutch stimuli (i.e., stories), spoken by male speakers, of 12 minutes each. Each stimulus is split up into 2 parts of 6 minutes and all stimuli are played twice, alternating the attended story. Each subject listens to 8 trials of 6 minutes.

We use the CNN model proposed by Vandecappelle et al. (2021) to conduct our experiments. The architecture of this model is depicted in Figure 1.

Figure 1: **Architecture of the CNN model (Vandecappelle et al., 2021).** The input has dimensions \(64\times T\), representing the number of EEG channels and the segment length. Five \(64\times 17\) spatio-temporal filters are shifted over the input matrix.
### Selection of training, validation and test sets

When training neural networks, the split of the dataset into a training, validation, and test partition is an important aspect.

In a two-competing speaker scenario, the task of the model is usually to predict which one of the two speakers is the attended one and which one is the unattended one. When recording the EEG, the measurement is usually spread out over multiple trials. In each trial, the subjects have to pay attention to one of the two speakers. Then, in the next trial, they have to pay attention to another speaker, to generate a balanced dataset. Translating this into output labels, means that there is usually one label per trial, e.g., _left speaker_ or _right speaker_.

A common way to split these datasets up into training/validation and testing sets is to split each trial into a training, validation and test sets (Zakeri and Gervanchizadeh, 2021; Lu et al., 2021; Xu et al., 2022, 2022; Su et al., 2021; Shree et al., 2016; Ciccarelli et al., 2019; de Taillez et al., 2020; Vandecappelle et al., 2021). While this might not seem problematic, this potentially allows models to overfit the training data. With only one label per trial (left/right), the model might learn to identify the trial from which the segment of EEG was taken, rather than solving the auditory attention task. If the validation and test set are taken from within the same trial, they have the same correct label, and information from the training set can leak into the validation and test set. This leads to models that seemingly perform great, but are unable to generalize and do not score well on unseen trials.

To prevent this, we propose to always use held-out trials for the test set (Kuruvila et al., 2021; Thornton et al., 2022; Hosseini et al., 2021; Tian and Ma, 2020). If the dataset contains 10 trials, 8 could be used for training, 1 for validation, and 1 for testing. Since the trial used for testing is never seen by the model before, it cannot rely on identifying which trial the EEG segment is taken from and has to learn to identify the underlying speaker information.

To demonstrate the necessity of this split, we conducted experiments with two different splits of the dataset and show that this leads to substantially different results.

We use the model proposed by Vandecappelle et al. (2021) and follow the training procedure, using two different splits of the dataset. In the first experiment, we split each trial of 6 minutes into an 80:10:10 training/validation/test partition. In the second experiment, we implement a 4-fold cross-validation scheme. Each fold contains unique stories, ensuring that the stories seen in training do not occur in the test set. Looking at Table 4, we divide the folds as (trial1, trial5), (trial2, trial6), (trial3, trial7) and (trial4, trial8). The results of both experiments can be seen in Figure 2. The average accuracy of the first experiment for segments with a window length of 1 second is 76.25 %, while the average accuracy of the second experiment does not exceed 51.10 %, showing the need for a cross-validation scheme when applying deep learning models to the auditory attention paradigm. The leave-one-story+speaker-out method was tested by Vandecappelle et al. (2021), and strong overfitting effects were found when within-trial splitting was performed.

Overall, out of 13 articles gathered with multiple sound source paradigms, only 4 performed a between-trial split, which as shown in this section, biases the model
performance.

### Benchmarking model evaluation using public datasets

Publicly available datasets include (1) for multiple speech sources (AAD): (Das et al. (2019); Fuglasang et al. (2017)), and (2) for a single speech source: Fuglasang et al. (2017), Broderick et al. (2018), Weissbart et al. (2022), and. While we are grateful

\begin{table}
\begin{tabular}{l l l l} \hline Trial & Left stimulus & Right stimulus & Attended side \\ \hline
1 & story1, part1 & Story2, part1 & Left \\
2 & story2, part2 & Story1, part2 & Right \\
3 & story3, part1 & Story3, part1 & Left \\
4 & story4, part2 & Story4, part2 & Right \\
5 & story2, part1 & Story1, part1 & Left \\
6 & story1, part2 & Story2, part2 & Right \\
7 & story4, part1 & Story3, part1 & Left \\
8 & story3, part2 & Story4, part2 & Right \\ \hline \end{tabular}
\end{table}
Table 4: Example division for the Das2019 dataset for 1 subject. Between subjects, the attended direction is alternated.

Figure 2: Results for training the model from Vandecappelle et al. (2021) using different sets of training/test. Box plots shown over 18 subjects. Each point in the boxplot corresponds to the auditory attention detection accuracy for one subject, averaged over all segments. Split within trials: each trial of 6 minutes is split into 80/10/10 training/validation/test. Split between trials: out of the 8 trials per subject, use 6 for training, 1 for validation and 1 for test.
to the authors for making this data available, unfortunately, as EEG data collection is expensive and time-consuming, the total amount of data up to now has remained relatively small in the context of deep learning: 60 hours for AAD, and 30 hours for single sound source paradigm. The lack of a larger public dataset makes it difficult to benchmark the models. Moreover, training and evaluating on a specific dataset can result in overfitting and lack of generalizability. We recently made available a much larger dataset Bollens et al. (2023a) of 85 subjects, with a total of approximately 200 hours of data. While this remains a very small dataset compared to those available in the fields of automatic image and speech recognition, we believe it is a substantial step forward, and hope that it paves the way towards standardized benchmarking such as demonstrated in the recent IEEE Auditory EEG Challenge (Bollens et al., 2023b).

A potential point of improvement for most of the papers from this review is to _additionally_ evaluate the developed architectures on multiple datasets recorded with various EEG devices (e.g., different number and location of electrodes) and experimental set-ups (e.g., different signal-to-noise ratios, inserted phones or speakers). To illustrate good practices, some generalization experiments were conducted in Accou et al. (2023): the authors trained a model on their dataset and they evaluated it on a publicly available dataset (Fuglsang et al., 2017).

Among articles gathered in this study, only 9 out of 29 (Monesi et al., 2020, 2021; Accou et al., 2021, 2021; Bollens et al., 2022; Su et al., 2021; Kuruvila et al., 2021; Vandecappelle et al., 2021; Puffay et al., 2023) involved the use of a publicly available dataset, and only one attempted to evaluate generalization to another dataset (Thornton et al., 2022b). Ideally, we recommend to evaluate trained model on multiple publicly available datasets, to ensure generalization capabilities of a model.

Considering the above-stated issues, the solution to share data publicly seems straightforward. However, sharing EEG data is complicated due to their biological nature. In many countries, the participants have to agree explicitly to their data being shared (anonymized/pseudoinmyzed) in a publicly available dataset.

Therefore, we encourage the research groups to work towards establishing a common dataset to facilitate model comparison. This will be a huge time gain and be a good control for possible pitfalls in recordings, preprocessing, or model evaluation. As a comparison, most deep learning models in ASR are evaluated on shared datasets (e.g., Librispeech ASR corpus from Panayotov et al. (2015)) and with common error measures such as word error rate.

### Interpretation of correlation scores

When decoding continuous speech features such as the envelope from EEG, decoding quality is often estimated by correlating the reconstructed speech envelope with the presented stimulus envelope. In several papers we collected (Reddy Katthi and Ganapathy, 2021; Katthi and Ganapathy, 2021; Thornton et al., 2022b; Sakthi et al., 2019), a correlation metric is reported as a metric of the performance of the model being used. While correlation metrics are important for interpretation and possible applications (e.g. hearing tests), they depend on the training, evaluation and architecture of a model, the experimental paradigm, and the nature, quality, size and preprocessing of the datasets used.

Standard statistical tests for correlations are ill-equipped to deal with non-independent
sample data, such as (low-pass filtered) EEG and speech envelopes (Combrisson and Jerbi, 2015; Crosse et al., 2021), as estimated correlations between distant segments can be high by chance. Therefore, an appropriate null distribution has to be constructed to detect whether a model can effectively use neural data to decode speech from EEG (or predict EEG channels from a speech feature). For the encoding/decoding case, Crosse et al. (2021) proposed to use a permutation test using randomly (circular) shifted versions of the predicted data with regards to the actual data to estimate the null distribution. Among all the papers we gathered performing a R/P task, only Thornton et al. (2022b) used this method. The percentiles of this null distribution can be used to measure the significance of the results.

An example of the proposed permutation test is visualized for a linear decoder with an integration window of 250 ms in Figure 3 (a). The decoder was trained on data of a single (representative) recording of the 48-subject dataset in 6-fold cross-validation. Both EEG and speech envelope data were filtered between 0.5-4 Hz with an eighth order Butterworth filter. The decoder was evaluated on 1 minute windows with 80% overlap. The null distribution was constructed using 100 permutations of circular shifted speech envelopes, similar to the approach suggested by Crosse et al. (2021). The mean of the predicted correlation scores (0.137 Pearson correlation) is greater than the 95th percentile of the null distribution (0.099 Pearson correlation), showing significance at \(\alpha\)=0.05.

To illustrate the effect of preprocessing and evaluation paradigm, the same model is trained and evaluated on the same recording, but with a broadband speech envelope

Figure 3: Null and actual (prediction) distributions for a linear decoder with a 250 ms integration window, trained in 6-fold cross validation. For (a) EEG and speech data was filtered between 0.5-4Hz and evaluation was performed on 1 minute windows with 90% overlap. For (b) EEG was filtered between 0.5-4Hz and the speech envelope highpass filtered above 0.5Hz. Evaluation was performed on 5 second windows with 80% overlap. The purple lines represent the mean of the null and actual distribution respectively. The red striped line represents the 95-percentile of the null distribution. Note that while the same model is used in (a) and (b), and the mean correlation score in (b) is higher than (a) (0.146 vs. 0.142 respectively), only (a) is statistically significant.
(highpass filtered with an order 8 Butterworth filter, using 0.5 Hz as a cutoff frequency) and evaluated on 5 seconds windows with 80% overlap. The null distribution was again calculated using permutations of circular shifted speech envelopes. While the mean correlation scores of the predictions increased from 0.137 to 0.146 Pearson correlation, due to the increased variance of the null distribution (from 0.002 to 0.039), the 95th percentile of the null distribution has also increased from 0.099 to 0.327 Pearson correlation, rendering the obtained correlation scores not significantly different from the null distribution. This example illustrates why null distributions should be constructed methodically for each obtained result to ensure significance. While this example used a simple linear model for clarity and computational efficiency, the general methodology is also applicable to deep learning models (even without retraining the model) and forward modeling.

Using the permutation of random circular shifts has a few drawbacks. As mentioned in the study of Crosse et al. (2021), discontinuities might appear at the end/beginning 'of the recording, possibly leading to an inappropriate null distribution. If the recording is sufficiently long however, this risk decreases. Secondly, sufficient permutations have to be computed to obtain an accurate estimate of the percentile of the null distribution.

When comparing models based on prediction quality, the choice of preprocessing techniques and datasets should be taken into account. For example, studies commonly filter EEG data into separate bands (e.g., delta band [0.5-4 Hz], theta band [4-8 Hz], etc.) These bands have been linked to different processing stages (e.g., Etard and Reichenbach (2019)). When filtering data, caution has to be taken when filtering the target signal (i.e., EEG in forward models, speech features in backward models), as this directly influences the difficulty of the task (e.g., a narrowly bandpassed low-frequency target signal is easier to predict than a broadband target signal), possibly making the task trivial. This also complicates using correlation scores as a metric for general model performance, as some models might perform well using broadband EEG/stimuli features (e.g., Accou et al. (2021a)), while others might benefit from more narrowband features (e.g., linear decoders Vanthornhout et al. (2018)). Finally, auditory EEG datasets are often recorded with varying equipment, varying methodologies and different languages of both stimuli and listeners, which can influence the obtained correlation scores and thus make correlation scores unfit for comparison of model performance across datasets.

Our recommendations are as follows: Firstly, construct an appropriate null distribution for each experimental result, and compare it to the correlations between the predicted and original signal. Secondly, when comparing models based on correlation scores of predictions, one must be aware of the influence of external factors (preprocessing, dataset choice, training/evaluation paradigm,...) on the obtained correlation values and interpret the obtained correlation scores with caution.

We also identified studies that used MSE and MCD as a reconstruction (or prediction) performance metric (Krishna et al., 2020, 2021a,b), which does not enable direct comparison with correlation values. We therefore also recommend to provide multiple metrics including Pearson correlation to allow comparison with other studies. A recent study also implemented a model predicting the EEG signal from speech for both a match and a mismatch segment, in order to get an accuracy value from a forward model (Puffay et al., 2023), opening the path for a mapping between evaluation metrics.
### Model generalization to unseen subjects

Subject-specific models sometimes have an performance advantage over subject-independent models as they can be fine-tuned to idiosyncrasies of a given subject and are not required to generalize to other subjects. However, subject-independent models are particularly attractive as they do not require training data for new subjects and much larger datasets an can be used to train them.

Across subjects, the EEG cap placement can vary and so does the brain activity. Training models on multiple subjects enables the model to learn these differences. That remark also applies to different EEG systems with different densities and locations of electrodes, or experiment protocols.

The performance of subject-independent models, especially on subjects not seen during training, depends on the training data. To illustrate this, the LSTM model of Monesi et al. (2020) is trained on 1 up to 28 subjects of the 48-subject dataset, and evaluated on the test set of the 20 remaining subjects. The results are displayed in Figure 4. With this analysis, we show that given the model and the collected dataset, the performance seems to reach a plateau, as the standard deviation of the last 10 medians (from subject number 18 until subject number 28) is less than 1%.

Among the 29 studies we gathered, 7 used a subject-independent training paradigm. If one has a limited amount of data per subject, we recommend to use subject-independent training. One can still fine-tune a subject-independent model (i.e. train a subject-independent model on all subjects, keep its weights and train it on the subject of interest before evaluation) to boost its performance on a given subject. Please note that fine-tuning is likely be more efficient if the subject fine-tuned on belongs to a similar group to subjects used for prior training (e.g., healthy young normal-hearing).

### Negative samples selection in MM task

When training models on the MM task, the choice of the mismatched segments (negative samples) is important to make sure the model can generalize well. The main idea is that the negative samples should be what is called "hard negatives" in the deep learning and contrastive learning literature (van den Oord et al., 2018). The negative samples should be challenging enough such that it forces the model to learn to relate neural responses to the positive samples (here the matched segments) rather than only learning to distinguish between positive and negative samples. For example, if we sample the mismatched segments from white noise or any signal that has a very different distribution than that of the positive samples, then a model, with enough parameters and capacity, will not learn the relation between EEG and speech but it will rather learn that there is a difference between positive and negative speech samples.

It is shown on speech data that taking negative samples from the same stimulus or same speaker yields the best accuracy in a phone classification task (Robinson et al., 2021). This is in line with the theory of using hard negatives to train contrastive models as mentioned above. As a result, to train a model in match-mismatch that can relate EEG to speech, it is better to sample mismatched segments from the same speech stimulus (i.e. story) the matched segments are drawn from, such that we have
a similar distribution for the matched and mismatched segments. In addition, we recommend designing the training in such a way that the mismatched segments also appear as matched segments with other EEG segments (see Figure 6), such that the only way to determine whether a candidate speech segment is matched or mismatched is to use the corresponding EEG segment.

In our work (Monesi et al., 2020; Accou et al., 2021), we have used mismatched segments from the same speech stimulus. More specifically, we selected the mismatched segment 1 second after the end of the matched segment. We have also tried taking mismatched segments from the past instead of the future, which led to the same match-mismatch classification performance. In our setup, we make sure that the mismatched segments are temporally close enough to the matched segments. But most importantly, we make sure that a mismatched segment is also a matched segment with another EEG segment as mentioned above. Finally, we report the results of two experiments to illustrate the importance of the above-mentioned points while training the LSTM-based model in the MM task.

To support the choice of a 1 s shift as a robust hard negative, we conducted the following experiment: we train our LSTM model on 48 subjects from the dataset with

Figure 4: The LSTM model of Monesi et al. (2020) trained on 1-28 subjects of the 48 subject dataset, and evaluated on the test-set of the remaining 20 subjects. Each point in the boxplot corresponds to the match-mismatch accuracy for one subject, averaged over segments.
matched and mismatched segments selected with a 1 s shift. We then evaluate our model under two conditions. First on a test set generated with the same 1 s shift for the 48 subjects, second with a given shift between 1 s and 20 s selected randomly for each subject. We show the results in Figure 5a.

We observe no significant difference in accuracy between the random and the 1 s shift. This result alleviates all doubts about the possibility for the model to learn from the serial correlation structure of the data over the 1 s span, causing a lack of generalization to other shifts. In addition, we observe a slight increase of the random shift condition, suggesting that larger shifts facilitates the task for the model. We believe that taking systematically large enough shifts would lead to a significant performance increase.

In a second experiment, we designed our match-mismatched segments in a way that violated the setup in Figure 6 such that the mismatched segments were never matched segments exactly. More specifically, we used 65 time samples (instead of 64) as a space between end of the matched and start of the mismatched segment in combination with using a window shift of 64 time samples (one second). As a result, matched segments had overlap with mismatched segments but they were never exactly mismatched segments with other EEG segments, thus resulting in two different sets for matched and mismatched segments. Note that other spacing lengths between end of matched and start of the mismatched segment would also result in mismatched segments never exactly being matched segments with other EEG segments. More generally, if the sum of the window length and the spacing is divisible by the window shift, then mismatched segments will also appear as matched segments (our recommended setup for training). We used the 48-subjects dataset where subjects listened to 8 stories. Each recording was split into training, validation, and test sets using 80%, 10%, and 10% ratios, respectively. The training set comprised 40% from the start and 40% from end of the recording and the remaining 20% was further split into validation and test sets. As shown in Figure 5b, the model performs poorly when mismatched segments are never matched segments. Note that the dataset has only around 2.5 hours of unique speech. As a result, the model succeeds in remembering the matched and the mismatched speech segments (of the training set, the training accuracy is around 90%) instead of relating them to EEG.

In a third experiment, we took our mismatched speech candidates from another stimulus (story). The other stimulus was randomly chosen from a set of 7 stories available for the subject. We compare the results with our proposed default setup where we take mismatched speech candidates from the same stimulus. For each training scenario, we evaluated the trained model on both of the setups. When we choose mismatched segments from another stimulus, the model does not generalize well (52% classification accuracy) to unseen data and it even performs poorly (53% classification accuracy) on the same setup it is trained on. On the other hand, we observe that the model trained in our recommended setup, in which mismatched speech segments are taken from the same stimulus, performs well on the default setup that is trained (84% classification accuracy) as well as on the new setup (84% classification accuracy). This implies that the model has learned to find the relation between EEG and the stimulus.
Figure 5: **Match-mismatch accuracy following different training set-ups.** (a) Match-mismatch accuracy of a LSTM model as a function of the mismatched shift. Our LSTM model was trained on the training and validation set of our 48 subjects dataset with a shift of 1 s between the matched and mismatched segments. This model was evaluated on the test set of the same subjects with a 1 s shift and a randomly picked shift (between 1 s and 20 s) per subject. ; (b) Classification accuracy of the LSTM-based model in the match-mismatch task. Box plots are shown over 48 subjects. Our recommended setup: our proposed training setup where a mismatched segment is also a matched segment with another EEG segment. Mismatched speech is never matched speech: in this setup, a mismatched speech segment will never be exactly a matched segment. As a result, some speech segments will only be matched and others only mismatched.
## 4 Conclusions

We gave an overview of the methods to relate EEG to continuous speech using deep learning models. Although many different network types have been implemented across studies, there is no consensus on which one gives the best performance. Performance is difficult to compare across studies as most research groups use their own dataset (e.g., EEG device, participants) and training paradigms. As we suspected many cases of overfitting, we suggested guidelines to make the performance evaluation less biased, and more comparable across studies.

The first point addressed the importance of the training, validation and test set selection. We demonstrated with an experiment that in multiple speech sources paradigms, the split must not be done within trials (i.e., the subjects have to pay attention to one of the two speakers during a defined amount of time) but between them. Some studies we reviewed have done such a split and show implausibly high decoding accuracies (Lu et al., 2021; Su et al., 2021, e.g.,), possibly remembering each trial's label when the split is done within trials.

We then addressed the need to use and share public datasets to encourage researchers to improve models and have a common general evaluation benchmark to do so. Gathering diverse data is also necessary to make models more generalizable across devices or experimental setups. We propose to proceed similarly to ASR and computer vision research by gathering large and diverse public datasets rather than working separately on small personal datasets.

While correlation metrics are important for interpretation and possible applications (e.g. hearing tests), they depend on the training, evaluation and architecture of a model, the experimental paradigm, and the nature, quality, size and preprocessing of the datasets used. It is necessary to construct an appropriate null distribution for each experimental result to see if a model performs significantly better than chance. When comparing models based on correlation scores of predictions, researchers should be aware of the influence of external factors (preprocessing, dataset choice, training/evaluation paradigm,...) on the obtained correlation values and interpret the obtained correlation scores with caution.

Figure 6: Illustration of the choice of mismatched segments in the match-mismatch task. Two important points to consider: 1. The mismatched segments are taken from the same sequence. 2. The mismatched segments are also matched segments (and vice versa) depending on the EEG segment. For example, segment 2 speech is the mismatched candidate with segment1 EEG but it will be a matched speech candidate with segment2 EEG.
Subject-independent models are very convenient because, when trained on a sufficient amount of data, they can cope with dataset diversity due to, e.g., EEG devices, protocols, brain anatomy or speech content. Although in certain cases (e.g., hearing aid device), an individual's good performance prevails over an ability to generalize, deep learning models require lots of data, which is not clinically ideal to collect from individual subject. We therefore recommend to use subject-independent models when the amount of data is limited.

For pratical applications, we need deep learning models to generalize and researchers to test their ability to do so, notably by evaluating models on other datasets or ensuring they were trained on enough data to reach their optimal performance. As an example experiment, we characterized an LSTM model's performance as a function of the number of subjects included in the training.

Finally, we highlight the importance of the negative sample selection in a match-mismatch task. With such temporally auto-correlated signals, the difficulty level of a match-mismatch task is also defined through the negative sample selection. Hence, two important characteristics of the negative sample selection are that the mismatched segment is taken from the same speech segment and that each mismatch speech segment is also a matched speech segment with another EEG segment. These two points constrain the model to use the EEG data provided to the model, ensuring the model cannot find the matched segment solely from the speech data.

## 5 Acknowledgements

Funding was provided by the KU Leuven Special Research Fund C24/18/099 (C2 project to Tom Francart and Hugo Van hamme), FWO fellowships to Bernd Accou (1S89622N), Corentin Puffay (1S49823N), Lies Bollens (1SB1423N) and Jonas Vanthornhout (1290821N).
