SGM

    score-based generative model
SNR

    signal-to-noise ratio
GAN

    generative adversarial network
VAE

    variational autoencoder
DDPM

    denoising diffusion probabilistic model
STFT

    short-time Fourier transform
iSTFT

    inverse short-time Fourier transform
SDE

    stochastic differential equation
ODE

    ordinary differential equation
OU

    Ornstein-Uhlenbeck
VE

    Variance Exploding
DNN

    deep neural network
PESQ

    Perceptual Evaluation of Speech Quality
SE

    speech enhancement
T-F

    time-frequency
ELBO

    evidence lower bound
WPE

    weighted prediction error
PSD

    power spectral density
RIR

    room impulse response
LSTM

    long short-term memory
POLQA

    Perceptual Objectve Listening Quality Analysis
SDR

    signal-to-distortion ratio
ESTOI

    Extended Short-Term Objective Intelligibility
ELR

    early-to-late reverberation ratio
TCN

    temporal convolutional network
DRR

    direct-to-reverberant ratio
NFE

    number of function evaluations
RTF

    real-time factor
MOS

    mean opinion scores
ASR

    automatic speech recognition
WER

    word error rate
AVSE

    audio-visual speech enhancement
SSL

    self-supervised learning
DL

    deep learning
ROI

    region-of-interest

# Audio-Visual Speech Enhancement with Score-Based Generative Models

Julius Richter1, Simone Frintrop2, Timo Gerkmann1

###### Abstract

This paper introduces an audio-visual speech enhancement system that leverages
score-based generative models, also known as diffusion models, conditioned on
visual information. In particular, we exploit audio-visual embeddings obtained
from a self-supervised learning model that has been fine-tuned on lipreading.
The layer-wise features of its transformer-based encoder are aggregated, time-
aligned, and incorporated into the noise conditional score network.
Experimental evaluations show that the proposed audio-visual speech
enhancement system yields improved speech quality and reduces generative
artifacts such as phonetic confusions with respect to the audio-only
equivalent. The latter is supported by the word error rate of a downstream
automatic speech recognition model, which decreases noticeably, especially at
low input signal-to-noise ratios.

Index Terms— audio-visual speech enhancement, diffusion models, self-
supervised learning.

††This work has been funded by the German Research Foundation (DFG) in the
transregio project Crossmodal Learning (TRR 169). We would like to thank J.
Berger and Rohde&Schwarz SwissQual AG for their support with POLQA.

##  1 Introduction

Robust speech processing, e.g. in telecommunications or for automatic speech
recognition (ASR), often requires improved speech quality and intelligibility.
For this purpose, speech enhancement aims at estimating clean speech signals
from noisy audio recordings [1]. However, in various real-world scenarios,
such as in negative signal-to-noise ratios, speech enhancement systems may
face limitations, typically resulting in speech distortions and impaired
speech recognition.

To mitigate these limitations, promising approaches have been proposed that
leverage both audio and video information which is referred to as audio-visual
speech enhancement (AVSE) [2]. In fact, incorporating visual information into
audio-based systems becomes an emerging research direction and improves both
the robustness and the overall performance [3]. The principle is in general
twofold: first, the visual information (e.g. lip movements) is usually not
affected by the acoustic environment, and second, it has been proven that
visual information is able to provide additional speech and speaker-related
cues [4, 5].

Prevailing deep learning approches proposed for AVSE typically involve models
with modality-specific encoders, feature fusion, and a decoder, all trained
end-to-end in a supervised fashion [6, 7]. Their goal is to learn audio-visual
representations suitable for speech enhancement which is usually defined as a
predictive regression task. Another approach to speech enhancement is to use
generative models which follow a different learning paradigm. They aim to
model the underlying statistical distribution of clean speech and do not
directly estimate the mapping from noisy input to clean output.

Recently, score-based generative models have been proposed for speech
enhancement [8, 9]. The idea is to corrupt clean speech with slowly increasing
levels of noise and train a deep neural network (DNN) to reverse this
corruption iteratively. This exciting new research direction provides an
alternative to other generative approaches and differs from predictive
approaches in its methodology and results. Since the models are trained to
generate clean speech conditioned on the noisy input, they tend to hallucinate
for signals with low SNRs [9]. This can lead to speech-like sounds with poor
articulation and no linguistic meaning or cause phonetic confusions, which may
be problematic in many practical applications.

In this paper, we build upon our previous work on score-based generative
models for speech enhancement [9] and develop an audio-visual extension of the
generative approach. We condition the noise conditional score network on
audio-visual embeddings, that are obtained from AV-HuBERT [10], a self-
supervised learning (SSL) model fine-tuned on lipreading (see Fig. 1). This
follows a recent trend in deep learning-based speech processing to leverage
powerful representations from pre-trained models, which have been trained on
large-scale datasets using SSL techniques.

Experimental evaluations show that the proposed AVSE system provides better
speech quality and reduces generative artifacts, such as phonetic confusions,
compared to its audio-only counterpart. This observation is supported by the
reduction of the word error rate (WER) of a downstream ASR model, which is
particularly evident at low input SNRs.

Figure 1: Proposed audio-visual speech enhancement system. The score model
iteratively refines noisy speech $\mathbf{y}$ to obtain a clean estimate
$\widehat{\mathbf{x}}$ and is conditioned on audio-visual embeddings
$\mathbf{f}_{AV}$ obtained from AV-HuBERT.

##  2 Related Work

###  2.1 Score-based Generative Models

Score-based generative models, also known as diffusion models, have recently
gained attention due to their ability to generate high-quality samples and
provide flexible and efficient training procedures. Originally inspired by
non-equilibrium thermodynamics [11], they exist in several variants [12, 13].
All of them share the idea of gradually turning data into noise, and training
a DNN that learns to invert this diffusion process for different noise scales.
In an iterative refinement procedure, the DNN is evaluated multiple times to
generate data, achieving a trade-off between computational complexity and
sample quality.

Song et al. [14] showed that the diffusion process can be expressed as a
solution of a stochastic differential equation (SDE), establishing a
continuous-time formulation for diffusion models. The forward SDE can be
inverted in time, resulting in a corresponding reverse SDE which depends only
on the score function of the perturbed data [15].

###  2.2 Audio-Visual Speech Enhancement

Early works on AVSE can be traced back to the 1990s [16]. More recently, a
number of interesting algorithms have been developed in the context of deep
learning, including methods based on spectral mapping/masking that are trained
in a supervised fashion [6, 7]. These methods typically consist of two
modality-specific encoders, one for processing noisy speech and another for
processing video information. The resulting audio and visual features are then
concatenated and processed to obtain an audio-visual embedding that is fed
into a decoder to predict the clean speech.

Alternatively, Sadeghi et al. [17] proposed a generative approach to AVSE
using a conditional variational autoencoder trained on clean speech only. The
model learns a clean speech prior that is conditioned on visual information.
At inference, this prior is combined with a noise model based on nonnegative
matrix factorization to estimate a probabilistic Wiener filter for denoising.

Other approaches consider AVSE as a speech resynthesis task, i.e., to create
an output that closely resembles the original signal in terms of sound
quality, prosody, and other acoustic characteristics [18]. Yang et al. [19]
developed a speaker-dependent model that generates clean speech codes of a
neural audio codec conditioned on noisy audio and visual inputs. The discrete
codes obtained are used to reconstruct a mel-spectrogram which is then fed
into a neural vocoder. Hsu et al. [20] break AVSE into audio-visual speech
recognition and text-to-speech exploiting hidden units from a pre-trained SSL-
model. Mira et al. [21] proposed a two-stage approach that predicts clean mel-
spectrograms from audio-visual speech and converts them into waveform audio
using a neural vocoder.

###  2.3 Self-Supervised Learning

Recently, SSL has gained significant attention in the speech community, with
well-known models such as wav2vec 2.0 [22] and HuBERT [23] for audio-only and
AV-HuBERT [10] for audio-visual speech representation learning. In contrast to
supervised learning, which involves direct optimization of models for a
specific task, SSL adopts a two-step approach. First, the model undergoes pre-
training on large-scale unlabeled data to extract task-agnostic
representations. Then, these learned representations are either used as input
to a supervised model, or the model itself is fine-tuned for a particular
downstream task. The expected outcome is to either improve the downstream
task’s performance or reduce the amount of labeled data required for training.

Pasad et al. [24] have analyzed the layer-wise features of wav2vec 2.0 and
found that the transformer layers follow an autoencoder-like behavior in which
the representation deviates from the input speech features as the depth of the
model increases, followed by a reverse trend in which even deeper layers
become more similar to the input. Furthermore, the evolution of the
representations follows an acoustic-linguistic hierarchy. The shallowest
layers encode acoustic features, followed by phonetic, word identity, and word
meaning information [24]. Huang et al. [25] have investigated SSL model for
speech enhancement and separation and concluded although they are not designed
for waveform generation tasks like enhancement and separation, some of them
achieve remarkable improvements over spectrogram magnitudes. Interestingly,
instead of extracting representations from the last hidden layer, they
proposed to aggregate embeddings from all layers with a learnable weighted
sum.

##  3 Method

In this section, we describe the proposed AVSE system, which we name _AV-Gen_.
Fig. 1 shows an overview of AV-Gen which consists of a score model and a pre-
trained AV-HuBERT model as an upstream video processor.

###  3.1 Score Model

Following the work in [9], we cast speech enhancement as a conditional
generation task using score-based generative models. For a pair of clean and
noisy spectrograms $\mathbf{x}_{0},\mathbf{y}\in\mathbb{C}^{d}$ with $d$
denoting the number of time-frequency bins, we define a stochastic diffusion
process $\\{\mathbf{x}_{t}\\}_{t=0}^{T}$, called _forward process_ , as the
solution to the SDE,

$\mathrm{d}{\mathbf{x}_{t}}=\gamma(\mathbf{y}-\mathbf{x}_{t})\mathrm{d}t+g(t)\mathrm{d}\mathbf{w}\,,$
(1)

where $\mathbf{x}_{t}$ denotes the process state at time $t\in[0,T]$,
$\gamma\in\mathbb{R}$ controls the transition from $\mathbf{x}_{0}$ to
$\mathbf{y}$, and $g(t)\in\mathbb{R}$ is the diffusion coefficient that
controls the amount of Gaussian white noise induced by a standard Wiener
process $\mathbf{w}$.

The forward process has an associated _reverse process_ which is given by the
reverse SDE [15, 14],

$\begin{split}\mathrm{d}\mathbf{x}_{t}=[-\gamma(\mathbf{y}-\mathbf{x}_{t})+g(t)^{2}\underbrace{\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|\mathbf{y})}_{\approx\,\mathbf{s}_{\theta}(\mathbf{x}_{t},\mathbf{y},\mathbf{f}_{AV},t)}]\mathrm{d}t+g(t)\mathrm{d}\bar{\mathbf{w}}\,,\end{split}$
(2)

where the score function $\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|\mathbf{y})$ is intractable and therefore approximated by
the score model $\mathbf{s}_{\theta}$ with learnable parameters $\theta$. As
in [9], we use the Noise Conditional Score Network (NCSN++) [14] as the score
model. The training objective is denoising score matching, as defined in [9,
Eq. (15)]. Note, however, that in contrast to [9], the score model is
additionally conditioned on the video modality by taking conditional features
$\mathbf{f}_{AV}$ as input.

At inference, we can generate a clean speech estimate $\widehat{\mathbf{x}}$
by solving the reverse SDE in Eq. (2). For this purpose, a numerical SDE
solver is employed which solves the task of speech enhancement iterating
through the reverse process starting at $t=T$ and ending at $t=t_{\epsilon}$,
where $t_{\epsilon}$ is a small minimum process time to avoid numerical
instabilities.

###  3.2 AV-HuBERT as Conditional Model

Figure 2: Network architecture: Each encoder layer $k$ of the score model is
conditioned on a conditional feature $\mathbf{f}_{k}$ which is a weighted sum
of the layerwise embedding $\mathbf{e}_{l}$ of the AV-HuBERT’s transformer
encoder.

As an upstream video processor for the score model, we use a pre-trained AV-
HuBERT model consisting of a hybrid ResNet-transformer architecture for which
we freeze the parameters at training. In particular, we exploit audio-visual
embeddings from each layer of its transformer encoder, as visualized in Fig.
2. For each encoder layer $k$ of the score model, we learn a conditional
feature $\mathbf{f}_{k}$ by aggregating the layer-wise embeddings
$\mathbf{e}_{l}$ of AV-HuBERT’s transformer encoder, as

$\mathbf{f}_{k}=\operatorname{TempAlign}_{k}\left(\sum_{l=1}^{L}w_{kl}\,\mathbf{e}_{l}\right),\quad
k=1,\dots,K\,,$ (3)

with trainable weights $w_{kl}$ with the properties $w_{kl}\geq 0$ and
$\sum_{l}w_{kl}=1$ for all $k$. The operation
$\operatorname{TempAlign}_{k}(\cdot)$ ensures that the features are temporally
aligned to the feature rates of the score model’s embeddings for different
resolutions. In particular, we use linear interpolation for upsampling and
trainable Conv1D layers with kernel size and stride $d$ for downsampling with
factor $d$. The features are broadcasted over the frequency dimension to match
the feature maps of the score model and the number of channels is adopted in
the temporal alignment operation. Finally, the conditional features
$\mathbf{f}_{AV}=\\{\mathbf{f}_{1},\dots,\mathbf{f}_{K}\\}$ have the same
shapes as the embeddings of the score model before concatenation.

##  4 Experimental Setup

###  4.1 Dataset

For audio-visual speech data, we use the LRS3 dataset [26], a large-scale
dataset primarily designed for the task of audio-visual speech recognition.
Besides the audio tracks sampled at $16\,\text{kHz}$, it includes face tracks
sampled at $25\,\text{Hz}$ from over 400 hours of TED and TEDx videos. Note
that recording conditions vary among the files and the data sometimes contain
reverberant speech and is not as clean as data recorded in controlled
environments such as a recording studio. For practical reasons, we limit the
size of training data and use the low-resource 30 hours subset introduced in
[10]. To meet the requirements for evaluating speech quality [27], we only
consider files from the test set with a minimum length of $3.2\,\text{s}$
resulting in 244 test files in total.

We generate pairs of clean and noisy speech by mixing the audio tracks of LRS3
with noise signals from the CHiME3 dataset [28] at SNRs sampled uniformly
between -6 and 12 dB for the training, validation, and test. Consequently, we
name the resulting data set LRS3-CHiME3.

###  4.2 Baselines

The primary objective of this paper is to investigate the impact of
incorporating visual information into SGMSE+ [9], our previous work on score-
based generative models for speech enhancement. To accomplish this, we train
SGMSE+ using identical settings as those employed for AV-Gen. This encompasses
the input representation, SDE parameters, and matching training
configurations, including batch size and learning rate. Considering that both
methods employ the same DNN as the score model, with the only distinction
being the visual conditioning, it is reasonable to regard SGMSE+ as the audio-
only counterpart to AV-Gen.

Furthermore, we consider VisualVoice [29] as an audio-visual baseline, a
method originally proposed for audio-visual speech separation. This method
learns a direct mapping from noisy to clean speech conditioned on visual
information. To this end, it utilizes an encoder-decoder architecture (54,7M
parameters) for complex mask prediction, in which the audio features are
combined with visual features extracted from a lip motion analysis network.
Here, we only use the mask prediction loss and omit the facial analysis
network, as we adopt the method for AVSE.

Method | AV | POLQA | PESQ | ESTOI | SI-SDR [dB] | WER | S | D | I
---|---|---|---|---|---|---|---|---|---
Mixture |  | $1.68\pm 0.55$ | $1.20\pm 0.20$ | $0.77\pm 0.13$ | $\>\>2.6\pm 5.1$ | $0.28\pm 0.30$ | $2.2$ | $1.5$ | $\mathbf{0.1}$
SGMSE+ [9] |  | $2.73\pm 0.61$ | $2.08\pm 0.56$ | $0.89\pm 0.08$ | $10.6\pm 4.8$ | $0.22\pm 0.25$ | $2.3$ | $0.6$ | $0.2$
VisualVoice [29] | ✓ | $2.78\pm 0.60$ | $2.11\pm 0.57$ | $0.89\pm 0.07$ | $11.0\pm 4.0$ | $0.15\pm 0.18$ | $1.5$ | $\mathbf{0.3}$ | $0.2$
AV-Gen (ours) | ✓ | $\mathbf{2.87\pm 0.55}$ | $\mathbf{2.23\pm 0.53}$ | $\mathbf{0.90\pm 0.06}$ | $\mathbf{11.4\pm 3.9}$ | $\mathbf{0.11\pm 0.14}$ | $\mathbf{1.1}$ | $\mathbf{0.3}$ | $0.2$

Table 1: Average speech enhancement performance obtained for the LRS3+CHiME3
test set. Values indicate mean and standard deviation for the 244 test files.

###  4.3 Hyperparameter Setting

####  4.3.1 Input Representation

The complex spectrogram is computed with the short-time Fourier transform
(STFT) using a Hann window of length 510 and a hop size of length 160
resulting in 256 unique frequency bins and an audio frame rate of
$100\,\text{Hz}$. Consequently, the audio frame rate is a multiple of the
video frame rate, which provides simple temporal alignment for feature fusion
to ensure audio-visual synchronization. We use the same spectrogram
transformation as in [9]. Unlike the original work, we do not normalize the
signal in the time domain, which should make the score model more robust to
varying input conditions.

For video pre-processing, we follow the protocol in [10] which stems from
prior works in lipreading [30, 31]. However, instead of using dlib for facial
landmark detection, we use the Face Alignment Network [32] because we found
better robustness when inspecting the processed data. Using the 68 facial
landmarks, each video clip is aligned to a reference face frame via affine
transformation and cropped to a $88\times 88$ region-of-interest (ROI)
centered on the mouth.

####  4.3.2 Score model and training configuration

We parameterize the SDE in Eq. (1) and the NCSN++ architecture with the same
configuration as in [9], resulting in 76M learnable parameters. We train the
score model with variable length sequences (between $2$ and $12$ seconds),
using the MaxTokenBucketizer from TorchData, which forms batches of variable
batch size. We utilize four Quadro RTX 6000 (24 GB memory each) and train for
60 epochs using the distributed data-parallel (DDP) approach in PyTorch
Lightning, which takes about $2.5$ days. We use the Adam optimizer [33] with a
learning rate of $5\cdot 10^{-4}$.

####  4.3.3 AV-HuBERT

We use AV-HuBERT Base with $L=12$ transformer layers. We use the fine-tuned
model for visual speech recognition that used LRS3 as pretraining data and
LRS3-433h as finetuning data. This model takes only the video modality as
input. We also experimented with the pre-trained model trained on LRS3 using
both modalities as input, which resulted in inferior performance.

###  4.4 Metrics

For an instrumental evaluation of speech quality and intelligibility, we use
standard metrics, including POLQA [34], PESQ [27], ESTOI [35], and SI-SDR
[36].

To evaluate the effect of speech enhancement on ASR, we use QuartzNet
(QuartzNet15x5Base-En) from the NeMo toolkit [37] as a downstream ASR system.
We measure the WER, which is defined as

$\operatorname{WER}=\frac{S+D+I}{N}\,,$ (4)

where $S$ is the number of substitutions, $D$ is the number of deletions, $I$
is the number of insertions needed to get from the predicted to the target
sequence, and $N$ is the number of words in the target sequence.

##  5 Results

In Tab. 1, we report the speech enhancement performance obtained for the
LRS3-CHiME3. Comparing AV-Gen to its audio-only counterpart SGMSE+, we see
that the inclusion of video information improves the speech quality, indicated
by an improvement of $0.14$ in POLQA and $0.15$ in PESQ. ESTOI is only
marginally better, suggesting that both methods are fairly similar in terms of
intelligibility. However, the evaluation on ASR shows an improved WER of 0.11
on average. We argue that this is mainly because SGMSE+ tends to produce
generative artifacts like phonetic confusions at low input SNRs [9], which is
alleviated by using video information. In fact, the reduced number of
substitutions and deletions on average reveals a stronger robustness with
respect to generative hallucinations, which is also verified by informal
listening. Moreover, AV-Gen seems to be competitive with the audio-visual
baseline VisualVoice, outperforming it in all metrics.

Figure 3: Mean performance in PESQ evaluated for different ranges of input
SNRs. Figure 4: Mean performance in WER evaluated for different ranges of
input SNRs

In informal listening, we find that AV-Gen occasionally generates a more
reverberant output compared to the noisy input. This could be due to the
generative behavior of the proposed model, as the data used for training was
recorded in the wild and also contains reverberant speech. To prevent this
effect, the training data could be cleaned beforehand.

To gain more insights into the effect of exploiting video information in AV-
Gen as opposed to SGMSE+, we evaluate PSEQ and WER at different input SNRs.
Fig. 3 shows the mean performance in PESQ, and Fig. 4 for WER, when evaluated
at specific ranges of input SNRs. As expected, both metrics degrade for
decreasing input SNRs. However, the proposed audio-visual method shows
stronger robustness, indicated by larger relative improvements. This is
particularly evident for WER, with drastic degradations for SGMSE+ at negative
input SNRs, at which AV-Gen remains fairly robust. Please find some listening
examples on our project page111https://uhh.de/inf-sp-av-gen.

##  6 Conclusion

In this paper, we presented an AVSE system that utilizes score-based
generative models conditioned on visual information. To this end, we leverage
audio-visual embeddings obtained from AV-HuBERT, a self-supervised learning
model that was fine-tuned on lipreading. The experimental evaluations
demonstrated that the proposed method improves speech quality and noticeably
reduces generative artifacts, such as phonetic confusions, in comparison to
its audio-only counterpart. The effectiveness of the system is further
supported by the significant reduction in WER, as observed in a downstream ASR
model, particularly for low input SNRs.

## References

  * [1] T. Gerkmann and E. Vincent, “Spectral masking and filtering,” in Audio Source Separation and Speech Enhancement (E. Vincent, T. Virtanen, and S. Gannot, eds.), John Wiley & Sons, 2018.
  * [2] D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y. Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio-visual speech enhancement and separation,” IEEE Trans. on Audio, Speech, and Language Proc. (TASLP), vol. 29, pp. 1368–1396, 2021.
  * [3] H. Zhu, M.-D. Luo, R. Wang, A.-H. Zheng, and R. He, “Deep audio-visual learning: A survey,” Int. Journal of Automation and Computing, vol. 18, pp. 351–376, 2021.
  * [4] W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,” The Journal of the Acoustical Society of America, vol. 26, no. 2, pp. 212–215, 1954.
  * [5] H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976.
  * [6] J.-C. Hou, S.-S. Wang, Y.-H. Lai, Y. Tsao, H.-W. Chang, and H.-M. Wang, “Audio-visual speech enhancement using multimodal deep convolutional neural networks,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2, no. 2, pp. 117–128, 2018.
  * [7] T. Afouras, J. S. Chung, and A. Zisserman, “The conversation: Deep audio-visual speech enhancement,” Proc. Interspeech 2018, pp. 3244–3248, 2018.
  * [8] S. Welker, J. Richter, and T. Gerkmann, “Speech enhancement with score-based generative models in the complex STFT domain,” ISCA Interspeech, pp. 2928–2932, 2022.
  * [9] J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based generative models,” IEEE Trans. on Audio, Speech, and Language Proc. (TASLP), 2023.
  * [10] B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” in Int. Conf. on Learning Repr. (ICLR), 2022.
  * [11] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” Int. Conf. on Machine Learning (ICML), pp. 2256–2265, 2015.
  * [12] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Inf. Proc. Systems (NeurIPS), vol. 33, pp. 6840–6851, 2020.
  * [13] Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” Advances in Neural Inf. Proc. Systems (NeurIPS), vol. 32, 2019.
  * [14] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” Int. Conf. on Learning Repr. (ICLR), 2021.
  * [15] B. D. Anderson, “Reverse-time diffusion equation models,” Stochastic Processes and their Applications, vol. 12, no. 3, pp. 313–326, 1982.
  * [16] L. Girin, G. Feng, and J.-L. Schwartz, “Noisy speech enhancement with filters estimated from the speaker’s lips.,” in Eurospeech, 1995.
  * [17] M. Sadeghi, S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Horaud, “Audio-visual speech enhancement using conditional variational auto-encoders,” IEEE Trans. on Audio, Speech, and Language Proc. (TASLP), vol. 28, pp. 1788–1800, 2020.
  * [18] A. Mehrish, N. Majumder, R. Bhardwaj, and S. Poria, “A review of deep learning techniques for speech processing,” arXiv preprint arXiv:2305.00359, 2023\.
  * [19] K. Yang, D. Marković, S. Krenn, V. Agrawal, and A. Richard, “Audio-visual speech codecs: Rethinking audio-visual speech enhancement by re-synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8227–8237, 2022.
  * [20] W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y. Adi, “ReVISE: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18795–18805, 2023\.
  * [21] R. Mira, B. Xu, J. Donley, A. Kumar, S. Petridis, V. K. Ithapu, and M. Pantic, “LA-VocE: Low-SNR audio-visual speech enhancement using neural vocoders,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023.
  * [22] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Inf. Proc. Systems (NeurIPS), vol. 33, 2020.
  * [23] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE Trans. on Audio, Speech, and Language Proc. (TASLP), vol. 29, pp. 3451–3460, 2021.
  * [24] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 914–921, IEEE, 2021\.
  * [25] Z. Huang, S. Watanabe, S.-w. Yang, P. García, and S. Khudanpur, “Investigating self-supervised learning for speech enhancement and separation,” in IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP), pp. 6837–6841, 2022.
  * [26] T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018.
  * [27] A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual evaluation of speech quality (PESQ) - a new method for speech quality assessment of telephone networks and codecs,” IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP), vol. 2, pp. 749–752, 2001.
  * [28] J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third ’CHiME’ speech separation and recognition challenge: Dataset, task and baselines,” IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 504–511, 2015.
  * [29] R. Gao and K. Grauman, “Visualvoice: Audio-visual speech separation with cross-modal consistency,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15490–15500, IEEE, 2021.
  * [30] B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6319–6323, 2020.
  * [31] P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7613–7617, 2021.
  * [32] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks),” in International Conference on Computer Vision, 2017.
  * [33] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Int. Conf. on Learning Repr. (ICLR), 2015.
  * [34] ITU-T Rec. P.863, “Perceptual objective listening quality prediction,” Int. Telecom. Union (ITU), 2018.
  * [35] J. Jensen and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,” IEEE Trans. on Audio, Speech, and Language Proc. (TASLP), vol. 24, no. 11, pp. 2009–2022, 2016.
  * [36] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR–half-baked or well done?,” IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP), pp. 626–630, 2019.
  * [37] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook, et al., “NeMo: a toolkit for building AI applications using neural modules,” arXiv preprint arXiv:1909.09577, 2019.

  *[ASR]: automatic speech recognition
  *[AVSE]: audio-visual speech enhancement
  *[DNN]: deep neural network
  *[SNRs]: signal-to-noise ratio
  *[SSL]: self-supervised learning
  *[WER]: word error rate
  *[SDE]: stochastic differential equation
  *[STFT]: short-time Fourier transform
  *[ROI]: region-of-interest


# A hybrid decision approach for the association problem in heterogeneous
networks

Salah Eddine Elayoubi  Orange Labs
38-40 Rue du General Leclerc
92130 Issy-Les-Moulineaux, France
<EMAIL_ADDRESS>Eitan Altman  INRIA Sophia Antipolis
10 route des Lucioles
06902 Sophia Antipolis, France
<EMAIL_ADDRESS>Majed Haddad, Zwi Altman  Orange Labs
38-40 Rue du General Leclerc
92130 Issy-Les-Moulineaux, France
<EMAIL_ADDRESS>
###### Abstract

The area of networking games has had a growing impact on wireless networks.
This reflects the recognition in the important scaling advantages that the
service providers can benefit from by increasing the autonomy of mobiles in
decision making. This may however result in inefficiencies that are inherent
to equilibria in non-cooperative games. Due to the concern for efficiency,
centralized protocols keep being considered and compared to decentralized
ones. From the point of view of the network architecture, this implies the co-
existence of network-centric and terminal centric radio resource management
schemes. Instead of taking part within the debate among the supporters of each
solution, we propose in this paper hybrid schemes where the wireless users are
assisted in their decisions by the network that broadcasts aggregated load
information. We derive the utilities related to the Quality of Service (QoS)
perceived by the users and develop a Bayesian framework to obtain the
equilibria. Numerical results illustrate the advantages of using our hybrid
game framework in an association problem in a network composed of HSDPA and 3G
LTE systems.

##  I Introduction

In order to handle the growing wireless traffic demand, operators are often
faced with the need to install new base stations. This could result in
splitting cells into smaller ones, or in having several base stations covering
the same cell. The second option may be preferred when the traffic has high
variability (in time and space) in which case it may be advantageous to have
the possibility to allocate resources from both base stations to any point in
the cell. This flexibility comes at a cost of having to include an access
control that takes the proper association decisions for the mobiles, that of
deciding to which base station (BS) to connect. To achieve efficient use of
the resources, these decisions should be based not only on the current system
state but also on expected future demand which may interact with traffic
assigned in the present.

We wish to avoid completely decentralized solutions of the association problem
in which all decisions are taken by the mobiles, due to well known
inefficiency problems that may arise when each mobile is allowed to optimize
its own utility. This inefficiency is inherent to the non-cooperative nature
of the decision making. On the other hand, we wish to delegate to the mobiles
a large part in the decision making in order to alleviate the burden from the
base stations.

The association schemes actually implemented are fully centralized: the
operator tries to maximize his utility (revenue) by assigning the users to the
different systems [1]-[3]. However, distributed RRM mechanisms are gaining in
importance: Users may be allowed to make autonomous decisions in a distributed
way. This has lead to game theoretic approaches to the association problems in
wireless networks, as can be found in [4]-[8]. The potential inefficiency of
such approaches have been known for a long time. The term ”The Tragedy of the
Commons” has been frequently used for this inefficiency [9]; it describes a
dilemma in which multiple individuals acting independently in their own self-
interest can ultimately destroy a shared limited resource even when it is
clear that it is not in anyone’s long term interest for this to happen.

We propose in this paper association methods that combine benefits from both
decentralized and centralized design. Central intervention is needed during
severe congestion periods. At those instants, we assume that the mobiles
follow the instructions of the base stations. Otherwise the association
decision is left to the mobiles, who make the decision based on aggregated
state information from the base stations. The decision making is thus based on
partial information that is signaled to the mobiles by the base station. A
central design aspect is then for the base stations to decide how to aggregate
information which then determines what to signal to the users. Note that this
decision making at the BS can be viewed as a mechanism design problem, or as a
Bayesian game.

##  II Problem statement

###  II-A System description

We consider a network composed of $S$ systems operated by the same operator.
Even if the model we develop is applicable to different kinds of situations,
we will focus on the more realistic and cost effective case where the operator
uses the same cell sites to deploy the new system (e.g. 3G LTE), while keeping
the old ones (e.g. HSDPA). As, in each cell, there are different radio
conditions following the position of the user regarding the cell site, the
peak throughput that can be obtained by the user connected to system $s$, if
served alone by a cell, differs following his position in the cell, as
illustrated in Figure 1 for a cell served by HSDPA and 3G LTE. For simplicity,
we consider that there are $N$ classes of radio conditions and that users with
radio condition $n$ have a peak rate $D_{n}^{s}$ if connected to system $s$.
The network state is then defined by the vector: ${\bf
M}=(M_{1}^{1},...,M_{N}^{1},...,M_{1}^{S},...,M_{N}^{S})$, $M_{n}^{s}$ being
the number of users with radio condition $n$ connected to system $s$.

Figure 1: LTE and HSDPA peak throughputs for different user positions.

We assume that the network broadcasts a partial load information $l$ ($1\leq
l\leq L$), e.g., an aggregated load information indicating for each system if
it is in low, medium, or high load state. An example of this load information
is described in Figure 2 for a network composed of HSDPA and LTE systems.

Figure 2: Aggregated load information.

###  II-B Policy definition

As stated before, users are only aware of the load information $l$ sent by the
network. Their policies are then based on this information. Let ${\bf P}$ be a
policy defined by the actions taken by mobiles in the different load
conditions. ${\bf P}$ is a $N\times L$ matrix whose element $P_{nl}$ is equal
to $s$ if class-$n$ users connect to system $s$ when the network broadcasts
information $l$.

Let $\mathcal{A}$ be the space of feasible states, $\mathfrak{P}$ be the set
of all possible policies and let $\mathfrak{L}$ be the set of load
information. An assignment $f:\displaystyle\mathcal{A}\rightarrow\mathfrak{L}$
specifies for each network state ${\bf M}$ the corresponding load information
$f({\bf M})$. On the other hand, when the load information is equal to $l$ and
the policy is ${\bf P}$, we can determine the system to which users of class
$n$ will connect by the value $P_{nl}$. As an example, knowing the function
$f(.)$ and the policy ${\bf P}$, if the network is in state ${\bf M}$, a class
$n$ user will connect to system $P_{nl}$, where $l=f({\bf M})$.

There are some important remarks to keep in mind when speaking about policies.
The first is that we suppose that a user connected to a system stays within it
until the end of his communication in order to avoid vertical handovers and
their signaling overhead. Furthermore, even if the decision is distributed,
all users will have the same policy and learn together how to enhance it.
However, a policy change will occur after an observation time, long enough to
insure that the steady state of the network has been reached. Note also that
users can connect to a system only if there is room in it, otherwise they are
directed by the network to an available system or blocked if all systems are
saturated.

##  III Utilities

We analyze a system offering streaming calls. The goal of a streaming user is
to achieve the best throughput, knowing that the different codecs allow a
throughput between an upper (best) $T_{max}$ and a lower (minimal) $T_{min}$
bounds. His utility is thus expressed by the quality of the streaming flow he
receives, which is in turn closely related to his throughput. Indeed, a
streaming call with a higher throughput will use a better codec offering a
better video quality. This throughput depends not only on the peak throughput,
but also on the evolution of the number of calls in the system where the user
decides to connect. Note that a user that cannot be offered this minimal
throughput in neither of the available systems is blocked in order to preserve
the overall network performance.

###  III-A Steady state analysis

####  III-A1 Instantaneous throughput

The instantaneous throughput obtained by a user in a system depends on the
state of the system. The throughput of a user with radio condition class $n$
connected to system $s$ is given by:

$t_{n}^{s}({\bf M})=\min\left[D_{n}^{s}\frac{G({\bf
M})}{\sum_{m=1}^{N}\sum_{r=1}^{S}M_{m}^{r}},T_{max}\right]$ (1)

where $G({\bf M})$ is the scheduler gain. Note here that the admission control
will insure that $t_{n}^{s}({\bf M})\geq T_{min}$ by blocking new arrivals.
The space of feasible states $\mathcal{A}$ is thus the set of all states ${\bf
M}$ where this constraint is ensured:

$\frac{\sum_{m=1}^{N}\sum_{r=1}^{S}M_{m}^{r}}{G({\bf
M})}\leq\frac{T_{min}}{D_{n}^{s}},\forall n,s|M_{n}^{s}>0$ (2)

####  III-A2 Steady state probabilities

The throughput achieved by a user depends on the number of ongoing calls. This
latter is a random variable whose evolution is governed by the arrival and
departure processes. We assume that the arrival process of new connections
with radio condition $n$ is Poisson with rate $\lambda_{n}$. Each arriving
user makes a streaming connection whose duration is exponentially distributed
with parameter $1/\mu$.

Within the space of feasible states $\mathcal{A}$, transitions are due to:

  * •

Arrivals of users of radio condition $n$. Let ${\cal G}_{n}^{s}({\bf M})$
denote the state of the system if we add one mobile of radio conditions $n$ to
system $s$: ${\cal G}_{n}^{H}({\bf
M})=(M_{1}^{1},...,M_{N}^{1},...,M_{1}^{s},...,M_{n}^{s}+1,...,M_{N}^{s},...,M_{1}^{S},...,M_{N}^{S})$.
The transition from state ${\bf M}$ to ${\cal G}_{n}^{s}({\bf M})$ happens if
the policy implies that system $s$ is to be chosen for the load information
corresponding to state ${\bf M}$, and if the state ${\cal G}_{n}^{s}({\bf M})$
is an admissible state. The corresponding transition rate is thus equal to:

$q({\bf M},{\cal G}_{n}^{s}({\bf M})|{\bf P})=\lambda_{n}\cdot I_{P_{n,f({\bf
M})}=s}\cdot I_{{\cal G}_{n}^{s}({\bf M})\in\mathcal{A}}$ (3)

where $I_{C}$ is the indicator function equal to 1 if condition $C$ is
satisfied and to 0 otherwise.

  * •

Departures of users of radio condition $n$. Let ${\cal D}_{n}^{s}({\bf M})$
denote the state with one less mobile of class $(n,s)$. The transition from
state ${\bf M}$ to ${\cal D}_{n}^{s}({\bf M})$ is equal to:

$q({\bf M},{\cal D}_{n}^{s}({\bf M})|{\bf P})=M_{n}^{s}\cdot\mu\cdot
I_{M_{n}^{s}>0}$ (4)

The transition matrix $\textbf{Q}({\bf P})$ of the Markov process is written
for each policy ${\bf P}$ knowing that its diagonal element is:

$q({\bf M},{\bf M}|{\bf P})=-\sum_{n=1}^{N}\sum_{s=1}^{S}(q({\bf M},{\cal
D}_{n}^{s}({\bf M})|{\bf P})+q({\bf M},{\cal G}_{n}^{s}({\bf M})|{\bf P}))$
(5)

The steady-state distribution is then obtained by solving:

$\left\\{\begin{array}[]{l}{\bf\Pi}({\bf P})\cdot\textbf{Q}({\bf P})=0\\\
{\bf\Pi}({\bf P})\cdot{\bf e}=1;\end{array}\right.$ (6)

${\bf\Pi({\bf P})}$ being the vector of the steady-state probabilities
$\pi({\bf M}|{\bf P})$ under policy ${\bf P}$ and ${\bf e}$ is a vector of
ones.

Once the vector ${\bf\Pi}$ is obtained, the global performance indicators can
be calculated, e.g., the blocking rate of class-$n$ calls knowing that the
load information is equal to $l$:

$b_{n}(l|{\bf P})=\frac{\sum_{{\bf M}\in\mathcal{A};{\cal G}_{n}^{s}({\bf
M})\notin\mathcal{A},\forall s\in[1,S]}\pi({\bf M}|{\bf P})}{\sum_{{\bf
M}\in\mathcal{A};f({\bf M})=l}\pi({\bf M}|{\bf P})}$ (7)

In this equation, we consider as blocked all calls that arrive in states where
both systems are saturated, i.e., where $t_{n}^{s}({\bf M})<T_{min}$, $\forall
j\in{H,L}$. We also obtain the overall blocking rate:

$b({\bf
P})=\sum_{n=1}^{N}\frac{\lambda_{n}}{\sum_{m=1}^{N}\lambda_{m}}\sum_{{\bf
M}\in\mathcal{A};{\cal G}_{n}^{s}({\bf M})\notin\mathcal{A},\forall
s\in[1,S]}\pi({\bf M}|{\bf P})$ (8)

###  III-B Transient analysis

The steady-state analysis described above is not sufficient to describe the
utility of the users as the throughput obtained by a user at his arrival is
not a sufficient indication about the quality of his communication because of
the dynamics of arrivals/departures. In order to obtain the utility, we modify
the Markov chain in order to allow tracking mobiles during their connection
time. For users of radio condition $n$ connected to system $s$, only states
where there is at least one user $(n,s)$ are considered. The calculation is as
follows:

  1. 1.

Introduce absorbing states $A_{n}^{s}$ corresponding to the departure of
mobiles that have terminated their connections. Additional transitions are
thus added between ${\bf M}$ and $A_{n}^{s}$ with rate equal to:

$\tilde{q}_{n}^{s}({\bf M},A_{n}^{s})=\mu\cdot I_{M_{n}^{s}>0}$ (9)

The transitions to the neighboring states with one less user are then modified
accordingly by subtracting $\mu$ from the original transition rates defined in
equation (4):

$\tilde{q}_{n}^{s}({\bf M},{\cal D}_{n}^{s}({\bf M})|{\bf
P})=(M_{n}^{s}-1)\cdot\mu\cdot I_{M_{n}^{s}>0}$ (10)

The remaining transition rates remain equal to the original transitions:

$\tilde{q}_{n}^{s}({\bf M},{\cal G}_{n^{\prime}}^{s^{\prime}}({\bf M})|{\bf
P})=q({\bf M},{\cal G}_{n^{\prime}}^{s^{\prime}}({\bf M})|{\bf
P}),\quad\forall n^{\prime},s^{\prime}$

and

$\tilde{q}_{n}^{s}({\bf M},{\cal D}_{n^{\prime}}^{s^{\prime}}({\bf M})|{\bf
P})=q({\bf M},{\cal D}_{n^{\prime}}^{s^{\prime}}({\bf M})|{\bf
P}),\quad\forall n^{\prime},s^{\prime}\neq s$

  2. 2.

Define matrix $\tilde{{\bf Q}}_{n}^{s}$ of elements $\tilde{q}_{n}^{s}({\bf
M},{\bf M}^{\prime})$ defined above and with diagonal elements as in equation
(5):

$\tilde{q}_{n}^{s}({\bf M},{\bf M}|{\bf P})=q({\bf M},{\bf M}|{\bf P})$

Under policy ${\bf P}$ , the volume of information $I_{n}^{s}({\bf M}|{\bf
P})$ sent by system $s$ users subject to radio conditions $n$ starting from
state ${\bf M}$ is then equal to the volume of information sent between state
${\bf M}$ and the absorbing state $A_{n}^{s}$. These values can be calculated
by solving the set of linear equations for all states ${\bf M}$:

$\sum\tilde{q}_{n}^{s}({\bf M},{\bf M}^{\prime}|{\bf P})I_{n}^{s}({\bf
M}^{\prime}|{\bf P})=-t_{n}^{s}({\bf M})$ (11)

knowing that $I_{n}^{s}(A_{n}^{s})=0$.

  3. 3.

The utility of a class-$n$ user that has found the network in state ${\bf M}$
and chosen to connect to system $s$ is the volume of information sent starting
from state ${\cal G}_{n}^{s}({\bf M})$. Recall that ${\cal G}_{n}^{s}({\bf
M})$ is defined as the state with one more class-$n$ call connected to system
$s$:

$u_{n}^{s}({\bf M}|{\bf P})=I_{n}^{s}({\cal G}_{n}^{s}({\bf M})|{\bf P})$ (12)

##  IV Optimality, game and control

In this section, we use the utilities of users that we obtained above to
derive the association policies. We first search for the optimal policy, i.e.
the policy that maximizes the global utility of the network. Nevertheless, as
it is not realistic to consider that the users will seek the global optimum,
we show how to find the policy that corresponds to the Nash equilibrium,
knowing that users will try to maximize their individual utility. We will next
show how the operator can control, by sending appropriate load information,
the equilibrium of its wireless users to maximize its own utility.

###  IV-A Optimality

####  IV-A1 Global utility

When a global optimum is sought, it is important to maximize the QoS of all
users. The global utility function can be written as:

$\displaystyle U({\bf
P})=\sum_{n=1}^{N}\frac{\lambda_{n}}{\sum_{i=1}^{N}\lambda_{i}}\sum_{l\in\mathfrak{L}}[(1-b_{n}(l|{\bf
P}))\times$ (13) $\displaystyle\sum_{{\bf M}|f({\bf
M})=l}u_{n}^{(P_{n,l})}({\bf M}|{\bf P})\pi({\bf M}|{\bf P})]$

knowing that $P_{n,l}\in[1,S]$ is the system where new users of class-$n$
connect when they receive the load information $l$ and have the policy ${\bf
P}$.

Note that, in this utility, we consider not only the QoS of accepted users
(throughput), but also the blocking rate as the aim is also to maximize the
number of accepted users. We also weight the users with different radio
conditions with their relative arrival rates.

####  IV-A2 Optimal policy

Knowing the utility in equation (13), the optimal policy is the one among all
possible policies that maximizes this utility:

${\bf P}^{*}=\arg{\hbox{$\underset{{\bf P}}{\max}\,$}}U({\bf P})$ (14)

###  IV-B Equilibrium

####  IV-B1 Individual utility

If the aim is to maximize the individual utility, users of different radio
conditions are interested by maximizing the QoS they obtain given the load
information broadcast by the network. The utility that a class $n$ user might
obtain if he chooses system $s$ when the load information is $l$, while all
other users follow policy ${\bf P}$ is then:

$U_{nl}^{s}({\bf P})=\frac{\sum_{{\bf M}|f({\bf M})=l}u_{n}^{s}({\bf M}|{\bf
P})\pi({\bf M}|{\bf P})}{\sum_{{\bf M}|f({\bf M})=l}\pi({\bf M}|{\bf P})}$
(15)

####  IV-B2 Nash equilibrium

A policy ${\bf P}^{*}$ corresponds to a Nash equilibrium if, for all radio
conditions and all load information, the individual utility obtained when
following ${\bf P}^{*}$ is the largest possible utility under ${\bf P}^{*}$.
Mathematically, this can be expressed by the following inequality for all
radio conditions $n\in[1,N]$ and all load information $l\in[1,L]$:

$U_{nl}^{(P^{*}_{n,l})}({\bf P}^{*})\geq U_{nl}^{s}({\bf P}^{*}),\forall
s\in[1,S]$ (16)

###  IV-C Control

In the previous section, we derived the policy that corresponds to the Nash
equilibrium for a game where players are the wireless users that aim at
maximizing their utility. However, there is another dimension of the problem
related to the information sent by the network and corresponding to the
different load information. Motivated by the fact that the network may guide
users to an equilibrium that optimizes its own utility if he chooses the
adequate information to send, we introduce a control problem, that can also be
modeled as a game between the base station and its users. At the core lies the
idea that introducing a certain degree of hierarchy in non-cooperative games
not only improves the individual efficiency of all the users but can also be a
way of reaching a desired trade-off between the global network performance at
the equilibrium and the requested amount of signaling.

More formally, the way of aggregating the loads in the broadcast information
(expressed by the function $f(.)$) is inherent to the previous analysis. In
particular, the utilities of individual users, calculated in equation (15), is
function of $f(.)$:

$U_{nl}^{s}({\bf P}|f)=\frac{\sum_{{\bf M}|f({\bf M})=l}u_{n}^{s}({\bf M}|{\bf
P})\pi({\bf M}|{\bf P})}{\sum_{{\bf M}|f({\bf M})=l}\pi({\bf M}|{\bf P})}$

This leads the wireless users to a Nash equilibrium that depends on the way
the network aggregates the load information:

${\bf P}^{*}={\bf P}^{*}(f)$

The control problem is thus defined as the maximization of the utility of the
network by tuning the function $f(.)$. If the aim of the operator is to
maximize its revenues by maximizing the acceptance ratio, the optimal solution
is:

$f^{*}=\arg{\hbox{$\underset{f}{\max}\,$}}\frac{1}{b({\bf P}^{*}(f))}$ (17)

with blocking defined as in equation (8).

##  V Results

For illustration, we consider the case of a network composed of HSDPA and 3G
LTE systems. Users are classified between users with good radio conditions (or
cell center users) and users with bad radio conditions (or cell edge users).
The network sends aggregated load information as shown in Figure 2 with the
following thresholds: $[H_{1}=0.3,H_{2}=0.7,L_{1}=0.3,L_{2}=0.7]$, meaning
that a system is considered as highly loaded if its load exceeds $0.7$ and as
low-loaded if its load is below $0.3$.

We also consider a streaming service where users require a minimal throughput
of 1 Mbps and can profit from throughputs up to 2 Mbps in order to enhance
video quality ($D_{min}=1Mbps$ and $D_{max}=2Mbps$). We consider an offered
traffic that varies from 1 to 10 Erlangs and obtain numerically the
equilibrium points.

###  V-A Equilibrium

We focus on the Nash equilibrium when wireless users aim at maximizing their
individual utility. For comparison purposes, we study three different
association approaches:

  * •

Hybrid decision approach: The proposed hybrid scheme where users receive
aggregated load information and aim at maximizing their individual utility. We
illustrate the global utility corresponding to the Nash equilibrium policy.

  * •

Peak rate maximization approach: This is a simple association scheme where
users do not have any information about the load of the systems. They connect
to the system offering them the best peak rate:

$s^{*}=\arg{\hbox{$\underset{s}{\max}\,$}}D_{n}^{s}$

Note that this peak rate can be known by measuring the quality of the
receiving signal.

  * •

Instantaneous rate maximization approach: The network broadcasts ${\bf M}$,
the exact numbers of connected users with different radio conditions. Based on
this information and on the measured signal strength, the wireless users
estimate the throughput they will obtain in both systems. A new user with
radio condition $n$ will then connect to the system $s^{*}$ offering him the
best throughput:

$s^{*}=\arg{\hbox{$\underset{s}{\max}\,$}}\frac{D_{n}^{s}}{1+\sum_{m=1}^{N}\sum_{r=1}^{S}M_{m}^{r}}$

Note that this scheme is not realistic as the network operator will not
divulge the exact number of connected users in each system and each position
of the cell.

We plot in Figure 3 the global utility for the three cases. This global
utility is the one defined in equation (13) and expressed in Mbits, as users
are interested in maximizing the information they send during their transfer
time.

As intuition would expect, the results show that the peak rate maximization
approach has the worst performance as a system that offers the largest peak
throughput may be highly-loaded, resulting in a bad QoS. However, a surprising
result is that the hybrid scheme, based on partial information, is comparable
and even outperforms the full information scheme when traffic increases. This
is due to the fact that streaming users will have relatively long sessions,
visiting thus a large number of network states; knowing the instantaneous
throughput at arrival will not bring complete information about the QoS during
the whole connection. On the contrary, the proposed hybrid, game theoretic,
approach aims at maximizing the QoS during the connection time.

Figure 3: Global utility.

###  V-B Control

We now turn to the second stage of our problem, where the network tries to
control the users’ behavior by broadcasting appropriate information, expected
to maximize its utility while individual users maximize their own utility. We
plot in Figure 4 the blocking rate for different ways of aggregating load
information, obtained when users follow the policy corresponding to Nash
equilibrium. In this figure, we plot the results for three cases: the optimal
thresholds (in red stars) and two other sets of thresholds. We can observe
that the utility of the network (expressed in the acceptance rate) varies
significantly depending on the load information that is broadcast. Such an
accurate modeling of the control problem is a key to understand the actual
benefits brought by the proposed hybrid decision approach.

Figure 4: Blocking rate for different broadcast load information; a vector of
thresholds $[H_{1},H_{2},L_{1},L_{2}]$ means that system $s$ will be
considered as highly loaded if its load exceeds $s_{2}$ and as low-loaded if
its load is below $s_{1}$ ($s=H$ for HSDPA and $L$ for LTE).

##  VI Conclusion

In this paper, we studied hybrid association schemes in heterogeneous
networks. By hybrid schemes we mean distributed decision schemes assisted by
the network, where the wireless users aim at maximizing their own utility,
guided by information broadcast by the network about the load of each system.
We first show how to derive the utilities of flows that are related to the QoS
they receive under the different association policies. We then derive the
policy that corresponds to the Nash equilibrium. Finally, we show how the
operator, by sending appropriate information about the state of the network,
can optimize its own utility. The proposed hybrid decision approach for the
association problem can reach a good trade-off between the global network
performance at the equilibrium and the requested amount of signaling.

## Acknowledgment

This work was supported by the ANR project WiNEM.

## References

  * [1] E. Stevens-Navarro, Yuxia Lin, V. W. S. Wong, ”An MDP-Based Vertical Handoff Decision Algorithm for Heterogeneous Wireless Networks”, IEEE Transactions on In Vehicular Technology, , Vol. 57, No. 2. (2008), pp. 1243-1254.
  * [2] D. Kumar, E. Altman, J.-M. Kelif, ”Globally Optimal User-Network Association in an 802.11 WLAN and 3G UMTS Hybrid Cell”, in: Proc. of 20th International Teletraffic Congress (ITC 20), Ottawa, Canada, Selected for Plenary Presentation, June 17-21, 2007.
  * [3] S. Horrich, S-E. Elayoubi and S. Ben Jamaa, ”On the impact of mobility and joint RRM policies on a cooperative WiMAX/HSDPA network”, IEEE WCNC 2008, Las Vegas, April 2008.
  * [4] Ercetin Ozgur, ”Association Games in IEEE 802.11 Wireless Local Area Networks”, IEEE transactions on wireless communications, 2008, vol. 7 (1), no12, pp. 5136-5143.
  * [5] Srinivas Shakkottai, Eitan Altman, and Anurag Kumar, ”Multihoming of Users to Access Points in WLANs: A Population Game Perspective”, IEEE Journal on Selected Areas in Communications, Vol. 25, No. 6, 2007 , August 2007.
  * [6] S. Shakkottai, E. Altman and A. Kumar, ”The Case for Non-cooperative Multihoming of Users to Access Points in IEEE 802.11 WLANs”, IEEE Infocom, 2006.
  * [7] Libin Jiang, Shyam Parekh and Jean Walrand, ”Base Station Association Game in Multi-cell Wireless Networks”,
  * [8] D. Kumar, E. Altman, J.-M. Kelif. User-Network Association in an 802.11 WLAN & 3G UMTS Hybrid Cell: Individual Optimality, in: Proc. of IEEE Sarnoff Symposium, Princeton, NJ, USA, April 30 - May 2, 2007\.
  * [9] Garrett Hardin, ”The Tragedy of the Commons,” Science, 162(1968):1243-1248.


#  Sparse Multitask Learning for Efficient Neural Representation of Motor
Imagery and Execution 111††thanks: This work was supported by the Institute of
Information & Communications Technology Planning & Evaluation (IITP) grant,
funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial
Intelligence Graduate School Program (Korea University); No.2021-0-02068,
Artificial Intelligence Innovation Hub) and the National Research Foundation
of Korea (NRF) grant funded by the MSIT (No.2022-2-00975, MetaSkin: Developing
Next-generation Neurohaptic Interface Techonology that enables Communication
and Control in Metaverse by Skin Touch.

Hye-Bin Shin
Dept. of Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Kang Yin
Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Seong-Whan Lee
Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

In the quest for efficient neural network models for neural data
interpretation and user intent classification in brain-computer interfaces
(BCIs), learning meaningful sparse representations of the underlying neural
subspaces is crucial. The present study introduces a sparse multitask learning
framework for motor imagery (MI) and motor execution (ME) tasks, inspired by
the natural partitioning of associated neural subspaces observed in the human
brain. Given a dual-task CNN model for MI-ME classification, we apply a
saliency-based sparsification approach to prune superfluous connections and
reinforce those that show high importance in both tasks. Through our approach,
we seek to elucidate the distinct and common neural ensembles associated with
each task, employing principled sparsification techniques to eliminate
redundant connections and boost the fidelity of neural signal decoding. Our
results indicate that this tailored sparsity can mitigate the overfitting
problem and improve the test performance with small amount of data, suggesting
a viable path forward for computationally efficient and robust BCI systems.

Keywords–brain–computer interface, network pruning, sparse multitask learning;

##  I INTRODUCTION

The neural processes underlying imagination, observation, and execution of
movements share a profound and enigmatic connection. Neuroscientific studies
have consistently demonstrated that these processes engage overlapping neural
circuits, suggesting a shared neural subspace[1, 2, 3]. This revelation has
significant implications for EEG-based brain-computer interfaces (BCIs)
leveraging motor imagery (MI) and motor execution (ME) paradigms[4, 5, 6].
Despite the success of machine learning algorithms in uncovering meaningful
and discriminative features from EEG data[7, 8, 9], deep learning has gained
popularity in the EEG-BCI domain due to its ability to provide end-to-end
learning solutions[10, 11]. However, state-of-the-art deep learning models
trained offline on a fixed EEG dataset show severe overfitting due to the
limited amount of training samples, coupled with the high-dimensional nature
of multi-channel EEG data[10].

To enhance generalization in deep learning models for BCI, various strategies
have been employed, from multi-domain learning, which leverages insights
gained from multiple subjects data, to multi-modal and multi-view learning
approaches that integrate diverse physiological signals and feature
representations, thereby enriching the model’s understanding and
representational power[12, 13, 14, 15, 16]. Multi-task learning, on the other
hand, remains relatively underexplored in the EEG-BCI domain and and presents
a unique opportunity to address the challenge of model generalization,
especially given its recent success in other domains, e.g., training large
language models[17]. In the context of motor-related BCI paradigms, a
multitask setup is particularly pertinent as it can exploit the duality of
task imagination and execution inherent in their design, e.g., motor
imagery/execution[15] and imagined/overt speech[18, 19]. Given the
neuroscientific evidence for shared neural substrates underlying these tasks,
multitask learning can potentially uncover the commonalities and learn shared
representation across these related tasks.

In this study, we construct a multitask CNN model for MI and ME classification
and apply sparse training technique to distill the network to its most
informative parameters while pruning away those that are not helpful for
either task[20, 21, 22]. This sparse multitask learning approach can mitigate
model overfitting and improve generalization performance by learning shared
representations across different tasks. The integration of different pruning
and sparse training methods is largely unexplored in BCI, but it can
potentially help to create efficient and compact models suitable for real-time
applications[23, 24, 25, 26]. Our contribution is twofold:

  * •

We demonstrate how multitask learning can be effectively applied to motor
imagery and execution tasks in BCIs, uncovering shared neural representations
that enhance model generalization.

  * •

We explore the integration of neural network pruning within this multitask
framework, showcasing its potential in reducing model complexity while
maintaining, or even improving, performance.

In short, this work aims to address the problem of limited generalization
performance in deep learning-based BCIs by integrating multitask learning with
model pruning techniques. Importantly, we base our construction on the core
assumption of shared neural substrates between imagery and execution in the
human brain and leverage the dual-task structure common in many motor-related
BCI paradigms.

##  II METHDOLOGY

Figure 1: Overview of the proposed sparse multitask learning framework based
on static sparse training or “pruning at initialization.” Given a multitask
CNN model for MI and ME classification, comprising feature extraction modules
and a classification head, the parameters for the shared and private modules
undergo a saliency-based mask generation process that outputs a pruning mask
for each task. The mask for the shared part is generated by feeding the task-
specific masks into an elementwise arbiter function. The final sparsified
model is obtained by applying these masks to the respective network
parameters.

###  II-A Problem Statement

Given $N$ trials of EEG samples $\mathbf{x}_{i}\in\mathbb{R}^{E\times{T}}$,
corresponding labels $y_{i}$, and target sparsity level $\sigma\in(0,1)$, the
following pruning objective can be formulated:

$\displaystyle\min_{\phi}\mathcal{L}(f_{\phi},\mathcal{D})=\min_{\phi}\frac{1}{N}\sum_{i=1}^{N}\ell\left(f_{\phi}\left(x_{i}\right),y_{i}\right)$
(1) $\displaystyle\text{ s.t.
}\quad\phi\in\mathbb{R}^{d},\quad\|\phi\|_{0}\leq(1-\sigma)\cdot d,$

where $\ell(\cdot)$ denotes the pre-defined loss function, $\phi$ represents
the network parameters, $d$ is the total number of parameters, and
$\|\cdot\|_{0}$ indicates the $L_{0}$ norm. The problem can be reformulated in
terms of finding an optimal binary mask $\mathcal{M}$:

$\displaystyle\min_{\mathcal{M}}\mathcal{L}(f_{\phi},\mathcal{M},\mathcal{D})=\min_{\mathcal{M}}\frac{1}{N}\sum_{i=1}^{N}\ell\left(f\left(x_{i};\phi\odot\mathcal{M}\right),y_{i}\right)$
(2) $\displaystyle\text{ s.t.
}\quad\mathcal{M}\in\mathbb{R}^{d},\mathcal{M}\in\\{0,1\\}^{d},\|\mathcal{M}\|_{0}\leq(1-\sigma)\cdot
d.$

In the context of disentangling the subspaces for imagery and execution, we
partition the network parameters into distinct sets of parameters:
$\phi^{\mathcal{I}}$ and $\phi^{\mathcal{E}}$, for task-specific parameters,
and $\phi^{s}$ for shared parameters. The corresponding binary masks
$\mathcal{M}^{\mathcal{I}}$, $\mathcal{M}^{\mathcal{E}}$, and
$\mathcal{M}^{s}$ are then optimized to selectively activate the network for
each subspace under the sparsity constraint. The cumulative loss for a sample
$x_{i}$ is a summation over the losses corresponding to each subspace:

$\ell(f_{\phi}(x_{i}),y_{i})=\sum_{k\in\\{\mathcal{I},\mathcal{E}\\}}\lambda^{k}\ell^{k}(f(x_{i};\phi^{ks}\odot\mathcal{M}^{ks}),y_{i}^{k}),$
(3)

where $\lambda^{k}$ denotes the regularization weight for the loss associated
with each subspace and $\phi^{ks}=\phi^{k}\cup\phi^{s}$. The sparsely
activated parameters for a task are thus $\phi^{ks}\odot\mathcal{M}^{ks}$.

###  II-B Proposed Framework

Our sparse multitask learning framework focuses on optimizing the binary masks
$\mathcal{M}^{\mathcal{I}}$ and $\mathcal{M}^{\mathcal{E}}$ for the imagery
and execution tasks, alongside the shared parameter mask $\mathcal{M}^{s}$, at
the beginning of training. Specifically, this initial phase of static sparse
training involves evaluating the importance or saliency of each network
parameter for the specific tasks of MI and ME classification and pruning the
network accordingly.

For computing the saliency scores, we measured the sensitivity of the loss
function to the removal of each parameter, following the sensitivity-based
one-shot evaluation of network parameters introduced in SNIP[27]. Sensitivity
is determined based on the gradient of the loss function with respect to each
parameter, indicating how much a small change in the weight would affect the
loss, and parameters with higher sensitivity scores are considered more
crucial for each task. Based on the computed scores, we can generate the task-
specific binary mask $\mathcal{M}^{k}$ for $k\in\\{\mathcal{I},\mathcal{E}\\}$
for selecting only the top-ranked parameters. Given the target sparsity level
$\sigma$, we want to retain the top $(1-\sigma)\cdot d^{ks}$ parameters, where
$d^{ks}$ is the number of task-specific network parameters. These masks are
used to prune the network, deactivating less critical parameters and enforcing
the target sparsity level.

The next crucial step is the integration of these task-specific masks to form
the final mask for the shared parameters, $\mathcal{M}^{s}$. The individual
masks are input into an arbiter function $\mathcal{A}$, or an element-wise
logical OR, which integrates the importance assessments from all tasks and
ensures that only universally non-critical parameters are pruned. The efficacy
of this approach hinges on the careful balance between task specificity and
commonality, enabling our multitask learning framework to specialize and
generalize where necessary. Finally, the network is initialized and trained
with these static masks in place, such that only the parameters indicated by
the masks are updated during training. The loss for each task is calculated as
previously defined. The resulting sparse multi-task CNN model was evaluated on
a validation set to assess its performance in both MI and ME classification
tasks.

##  III EXPERIMENTS

###  III-A Dataset Description

The study utilized a publicly available dataset[15] comprising EEG recordings
from ten participants (7 males and 3 females, right-handed, aged 24-31 years).
The experimental setup was designed to capture EEG signals across both ME and
MI sessions. Each trial began with a 4-second rest period, followed by a
3-second instruction phase, and a 4-second task performance period based on
visual cues indicating one of three upper limb movements: forearm extension,
hand grasp, or wrist supination. For the ME session, participants physically
performed the movement tasks, while in the MI session, they were instructed to
vividly imagine the movement without any physical execution. Each task was
randomized across 50 trials, culminating in a total of 150 trials per session.

###  III-B Evaluation Metrics

In assessing the performance of our multitask learning framework, we computed
the classification accuracy and the harmonic mean of precision and recall,
commonly known as the F1 score, for both MI and ME tasks. Table III-B shows
the performance evaluation results averaged across all participants. We
compared our proposed method against the unsparsified baseline model and two
established benchmarks for static sparse training, namely the Lottery Ticket
Hypothesis (LTH)[28] and Single-shot Network Pruning (SNIP)[27].

Method |  Sparsity (%) | Motor Imagery Task | Motor Execution Task
---|---|---|---
Acc. (%) | F1 | Acc. (%) | F1
| Baseline
---
0 | 60.0 $\pm$ 4.0 | 0.59 $\pm$ 0.04 | 70.0 $\pm$ 3.0 | 0.69 $\pm$ 0.03
| 20 | 61.0 $\pm$ 4.0 | 0.60 $\pm$ 0.04 | 71.0 $\pm$ 3.0 | 0.70 $\pm$ 0.03
LTH[28] | 40 | 62.5 $\pm$ 3.5 | 0.62 $\pm$ 0.03 | 72.5 $\pm$ 2.5 | 0.72 $\pm$ 0.02
| 80 | 58.0 $\pm$ 5.0 | 0.57 $\pm$ 0.05 | 68.0 $\pm$ 4.0 | 0.67 $\pm$ 0.04
| 20 | 61.5 $\pm$ 3.5 | 0.61 $\pm$ 0.03 | 71.5 $\pm$ 2.5 | 0.71 $\pm$ 0.02
SNIP[27] | 40 | 63.0 $\pm$ 3.0 | 0.62 $\pm$ 0.03 | 73.0 $\pm$ 2.0 | 0.72 $\pm$ 0.02
| 80 | 57.5 $\pm$ 4.5 | 0.57 $\pm$ 0.04 | 67.5 $\pm$ 3.5 | 0.67 $\pm$ 0.03
| 20 | 62.0 $\pm$ 3.0 | 0.61 $\pm$ 0.03 | 72.0 $\pm$ 2.0 | 0.71 $\pm$ 0.02
Ours | 40 | 64.5 $\pm$ 2.5 | 0.64 $\pm$ 0.02 | 74.5 $\pm$ 1.5 | 0.74 $\pm$ 0.01
| 80 | 60.5 $\pm$ 4.0 | 0.60 $\pm$ 0.04 | 70.5 $\pm$ 3.0 | 0.70 $\pm$ 0.03

∗Acc.: accuracy, F1: F1-score

##  IV RESULTS AND DISCUSSION

The observed results from the comparative study on sparsity methods suggest a
nuanced relationship between sparsity levels and performance metrics across MI
and ME tasks. Consistently, the methods displayed a trend where moderate
sparsity often improved performance over the unsparsified baseline, which may
be attributed to the regularization effect of pruning. This effect likely
helps in mitigating overfitting, thereby enhancing the model’s generalization
capability on unseen data. Interestingly, our sparse training method
demonstrated a robustness to increased sparsity levels, outperforming other
methods at higher sparsity percentages. This robustness can be linked to the
fact that it is precisely designed to preserve critical network connections
that are essential for the tasks at hand, thus maintaining high performance
even with a reduced number of parameters. It is also worth noting that the
performance in ME tasks consistently surpassed that in MI tasks. This could be
due to the more pronounced and consistent signal patterns inherent to executed
movements as opposed to imagined ones, which are known to be subtler and more
variable. The F1 scores, following a similar trend to accuracy, reinforce the
notion that precision and recall are not significantly compromised at moderate
levels of sparsity. This balance is crucial in applications where the trade-
off between false positives and false negatives is delicate, particularly for
BCI systems.

Figure 2: Convergence of training and validation losses across sparsity
levels. The proposed method demonstrates superior convergence rates and lower
loss values when compared with alternative methods. This suggests enhanced
learning efficiency and model generalization facilitated by our proposed
sparse training strategy.

However, as expected, an excessive sparsity level leads to a degradation in
performance, likely due to the loss of too many relevant connections, which
are critical for capturing the complex patterns required for accurate
classification. This performance drop at high sparsity levels serves as a
reminder of the delicate balance required in pruning methods to retain
essential network functionality. The observed trends also shed light on the
importance of task-specific pruning strategies in multitask learning
environments. The distinct performance profiles between MI and ME tasks
suggest that a one-size-fits-all approach to sparsity may not be optimal, and
task-specific considerations are imperative. In summary, the findings
underscore the potential of tailored sparsity methods in BCI systems, where
computational efficiency must be harmonized with robust performance across
varied tasks. Future work should explore the integration of these findings
into real-world applications and investigate the neurophysiological correlates
that underpin the differential performance between MI and ME tasks.

##  V CONCLUSION

This study presents a novel integration of multitask learning and neural
network pruning, specifically applied to motor imagery and execution tasks in
EEG-based BCIs. The proposed sparse multitask learning framework not only
mitigates the challenge of overfitting but also preserves—and in some
instances, enhances—the model’s performance even with a reduced parameter set.
The results indicate that carefully tailored sparsity can lead to more
efficient models without compromising the accuracy and robustness necessary
for BCI applications. Looking forward, the potential of this methodology
extends beyond the current dataset to a broader range of BCI paradigms
involving both imagined and executed tasks, such as imagined versus overt
speech. The adaptability and effectiveness of our approach in these varied
contexts could further validate its utility and robustness. Future work will
focus on applying our sparse multitask learning framework to diverse BCI
datasets and investigating the feasibility of deploying these models in real-
time BCI applications. Ultimately, our goal is to bridge the gap between
theoretical research and real-world scenarios, paving the way for more robust
and efficient BCI systems.

## References

  * [1] R. M. Hardwick, S. Caspers, S. B. Eickhoff, and S. P. Swinnen, “Neural correlates of action: Comparing meta-analyses of imagery, observation, and execution,” _Neurosci. Biobehav. Rev._ , vol. 94, pp. 31–44, 2018.
  * [2] X. Jiang, H. Saggar, S. I. Ryu, K. V. Shenoy, and J. C. Kao, “Structure in neural activity during observed and executed movements is shared at the neural population level, not in single neurons,” _Cell Rep._ , vol. 32, no. 6, 2020.
  * [3] J. A. Gallego, M. G. Perich, L. E. Miller, and S. A. Solla, “Neural manifolds for the control of movement,” _Neuron_ , vol. 94, no. 5, pp. 978–984, 2017.
  * [4] N. Padfield, J. Zabalza, H. Zhao, V. Masero, and J. Ren, “EEG-based brain-computer interfaces using motor-imagery: Techniques and challenges,” _Sensors_ , vol. 19, no. 6, p. 1423, Jan. 2019.
  * [5] D.-Y. Lee, J.-H. Jeong, B.-H. Lee, and S.-W. Lee, “Motor imagery classification using inter-task transfer learning via a channel-wise variational autoencoder-based convolutional neural network,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 30, pp. 226–237, 2022.
  * [6] R. Abiri, S. Borhani, E. W. Sellers, Y. Jiang, and X. Zhao, “A comprehensive review of EEG-based brain–computer interface paradigms,” _J. Neural Eng._ , vol. 16, no. 1, p. 011001, 2019.
  * [7] S.-B. Lee _et al._ , “Comparative analysis of features extracted from EEG spatial, spectral and temporal domains for binary and multiclass motor imagery classification,” _Inf. Sci._ , vol. 502, pp. 190–200, 2019.
  * [8] J. Kim _et al._ , “Abstract representations of associated emotions in the human brain,” _J. Neurosci._ , vol. 35, no. 14, pp. 5655–5663, 2015.
  * [9] K.-H. Thung _et al._ , “Conversion and time-to-conversion predictions of mild cognitive impairment using low-rank affinity pursuit denoising and matrix completion,” _Med. Image Anal._ , vol. 45, pp. 68–82, 2018.
  * [10] Y. Roy _et al._ , “Deep learning-based electroencephalography analysis: a systematic review,” _J. Neural Eng._ , vol. 16, no. 5, p. 051001, 2019.
  * [11] J.-S. Bang, M.-H. Lee, S. Fazli, C. Guan, and S.-W. Lee, “Spatio-spectral feature representation for motor imagery classification using convolutional neural networks,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 33, no. 7, pp. 3038–3049, 2021.
  * [12] D.-Y. Kim, D.-K. Han, J.-H. Jeong, and S.-W. Lee, “EEG-based driver drowsiness classification via calibration-free framework with domain generalization,” in _Proc. IEEE Int. Conf. Syst. Man, Cybern. (SMC)_ , 2022, pp. 2293–2298.
  * [13] K. Yin, E. Y. Lim, and S.-W. Lee, “GITGAN: Generative inter-subject transfer for EEG motor imagery analysis,” _Pattern Recognit._ , p. 110015, 2023.
  * [14] K.-T. Kim, C. Guan, and S.-W. Lee, “A subject-transfer framework based on single-trial EMG analysis using convolutional neural networks,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 1, pp. 94–103, 2019.
  * [15] J. H. Jeong, J. H. Cho, K. H. Shim, B. H. Kwon, B. H. Lee, D. Y. Lee, others, and S. W. Lee, “Multimodal signal dataset for 11 intuitive movement tasks from single upper extremity during multiple recording sessions,” _GigaScience_ , vol. 9, no. 10, p. giaa098, 2020.
  * [16] R. Mane _et al._ , “FBCNet: A multi-view convolutional neural network for brain-computer interface,” _arXiv preprint arXiv:2104.01233_ , 2021.
  * [17] V. Sanh _et al._ , “Multitask prompted training enables zero-shot task generalization,” _arXiv preprint arXiv:2110.08207_ , 2021.
  * [18] S.-H. Lee, M. Lee, and S.-W. Lee, “Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 12, pp. 2647–2659, 2020.
  * [19] S.-H. Lee, M. Lee, J.-H. Jeong, and S.-W. Lee, “Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery,” in _Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC)_ , 2019, pp. 4409–4414.
  * [20] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [21] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” _arXiv preprint arXiv:1608.08710_ , 2016.
  * [22] S. Srinivas, A. Subramanya, and R. Venkatesh Babu, “Training sparse neural networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , 2017, pp. 138–145.
  * [23] J. Wang, S. Zhao, C. Xu, J. Zhang, and R. Zhong, “Brain-inspired interpretable network pruning for smart vision-based defect detection equipment,” _IEEE Trans. Industr. Inform._ , vol. 19, no. 2, pp. 1666–1673, 2022.
  * [24] D.-Y. Kim, D.-K. Han, and H.-B. Shin, “Calibration-free driver drowsiness classification based on manifold-level augmentation,” in _Int. Winter Conf. Brain Comput. Interface (BCI)_ , 2023, pp. 1–4.
  * [25] M. Lee, C.-B. Song, G.-H. Shin, and S.-W. Lee, “Possible effect of binaural beat combined with autonomous sensory meridian response for inducing sleep,” _Front. Hum. Neurosci._ , vol. 13, pp. 425–440, 2019.
  * [26] E. Elsen, M. Dukhan, T. Gale, and K. Simonyan, “Fast sparse convnets,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , 2020, pp. 14 629–14 638.
  * [27] N. Lee, T. Ajanthan, and P. H. Torr, “SNIP: Single-shot network pruning based on connection sensitivity,” _arXiv preprint arXiv:1810.02340_ , 2018.
  * [28] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” _arXiv preprint arXiv:1803.03635_ , 2018.