# Comparison of convolutional neural networks for cloudy optical images
reconstruction from single or multitemporal joint SAR and optical images

Rémi Cresson
UMR TETIS, INRAE
500 rue jean-francois breton, 34090 Montpellier
<EMAIL_ADDRESS>
&Nicolas Narçon
UMR TETIS, INRAE
&Raffaele Gaetano
UMR TETIS, CIRAD
500 rue jean-francois breton, 34090 Montpellier
&Aurore Dupuis
CNES
18 Av. Edouard Belin, 31400 Toulouse
&Yannick Tanguy
CNES
&Stéphane May
CNES
&Benjamin Commandré
UMR TETIS, INRAE

###### Abstract

With the increasing availability of optical and synthetic aperture radar (SAR)
images thanks to the Sentinel constellation, and the explosion of deep
learning, new methods have emerged in recent years to tackle the
reconstruction of optical images that are impacted by clouds. In this paper,
we focus on the evaluation of convolutional neural networks that use jointly
SAR and optical images to retrieve the missing contents in one single polluted
optical image. We propose a simple framework that ease the creation of
datasets for the training of deep nets targeting optical image reconstruction,
and for the validation of machine learning based or deterministic approaches.
These methods are quite different in terms of input images constraints, and
comparing them is a problematic task not addressed in the literature. We show
how space partitioning data structures help to query samples in terms of cloud
coverage, relative acquisition date, pixel validity and relative proximity
between SAR and optical images. We generate several datasets to compare the
reconstructed images from networks that use a single pair of SAR and optical
image, versus networks that use multiple pairs, and a traditional
deterministic approach performing interpolation in temporal domain.

_Keywords_ Image reconstruction, clouds, atmospheric perturbation, SAR,
optical, convolutional neural networks, gap-filling

##  1 Introduction

###  1.1 Context

The Sentinel constellation is composed of different coupled SAR and optical
sensors with short revisit period (five to ten days). However, optical images
are frequently polluted by cloud cover. To leverage the problem of optical
image reconstruction, various approaches have been proposed over the years.
First, approaches based on mathematical, physical or statistical model, have
been extensively used to reconstruct the missing parts of the images. A review
of these traditional approaches are summarized in Shen et al. (2015). Among
them, we can distinguish multispectral based Hu et al. (2015) methods,
multitemporal Cheng et al. (2014) Li et al. (2014), and methods using optical
and SAR data fusion Eckardt et al. (2013). Lastly, machine learning and
particularly deep learning have become popular to achieve the task of cloudy
images reconstruction, thanks to the unprecedented ability to fuse images of
different modalities, and accompanied with state of the art results. In recent
years, deep neural networks have proven to be effective for image
reconstruction from time series of same modality Zhang et al. (2018) or from
timely available images at coarser spatial resolution Liu et al. (2019), or
from joint optical and SAR time series Scarpa et al. (2018)Cresson et al.
(2019). Sarukkai et al. (2020) have casted the problem of cloud removal as a
conditional image synthesis challenge, and have proposed a network to remove
clouds from a single optical image or from a triplet of optical images. In
Meraner et al. (2020), a cloudy optical image is reconstructed with the help
of a single SAR, using a convolutional neural network with a residual
correction performed on the input cloudy optical image. Conversely, Ebel et
al. (2020) have jointly modeled the cloud removal and the synthetic cloudy
optical images generation problems, concluding that the networks trained over
real data were performing the best.

###  1.2 Problematic

Our purpose is to lead the evaluation of several approaches based on
convolutional neural networks trained on real data, that reconstruct optical
images impacted by clouds. We compare these approaches with a popular
traditional deterministic approach, the gap-filling Inglada et al. (2015).
While the existing literature covers mostly the comparison of approaches that
consume the same inputs, comparing approaches consuming various forms of
inputs, e.g. single or multiple, optical and/or SAR images or pairs of images,
remains an interesting topic from an operational perspective. In this paper,
we address the comparison of the following kinds of methods, which consume one
or more input pair of optical and SAR images acquired in various conditions to
reconstruct or generate one single output optical image:

  1. 1.

Reconstruct a cloudy optical image using an additional SAR image acquired at
the same date,

  2. 2.

Reconstruct a cloudy optical image using an additional SAR image acquired at
the same date, and two other cloudy optical/SAR images pairs acquired before
and after,

  3. 3.

Generate an optical image at one given desired date using two clean optical
images acquired before and after.

###  1.3 Method

In this paper, we provide an insight into various optical image reconstruction
methods. In particular, we address the question of which approach to employ
for a specific availability of remote sensing products, in comparing a few
selected single date based and multiple dates based methods. Since these
methods employ inputs of different nature (single optical or SAR image, or
optical and SAR images pair), number (single or multiple image or pair or
images), and cloud coverage (clean or cloudy images), their comparison is not
straightforward. To leverage this, we introduce the acquisitions layout, a
descriptor of the available inputs and their properties, for a specific
approach. We then use space partitioning data structures to ease the
generation of various datasets from specific acquisitions layouts. These
datasets are then used to train networks, and also at inference time for the
comparison of the different methods when a common set of inputs can be shared
and matches the expected constraints, e.g. cloud coverage. This simple yet
generic framework allows to produce datasets tailored for a specific problem
and suited to the data availability, i.e. inputs and targets images. We carry
out the benchmarks of representative state of the art methods for optical
image reconstruction, namely the network presented in Meraner et al. (2020),
which uses a single pair of optical and SAR image acquired near the same date,
and a convolutional network that inputs three consecutive pairs to reconstruct
the central optical image Cresson et al. (2019)Scarpa et al. (2018). In Scarpa
et al. (2018), an additional DEM is used as input of the network, and we also
investigate the contribution of such ancillary data in the single date
network. To better disentangle the benefits of the different modalities, we
perform an ablation study removing the DEM, and the SAR inputs.

###  1.4 Overview

In section 2, we present the remote sensing data used in this study. In
section 3, we detail the implemented models. In section 4, we detail our
framework for the creation of datasets, which is a crucial aspect of our work.
In section 5 we detail the methodology used to train the models and carry out
the comparison of the different approaches. Finally we discuss the results in
section 6.

##  2 Data

###  2.1 Sentinel images

We use 10 tiles of Sentinel-2 images acquired over the Occitanie area in
France (figure 1), from january 2017 to january 2021, that represents a total
of $3593$ optical images. We also use every available Sentinel-1 images
acquired in ascending orbit over the Occitanie area during the same period,
that we superimpose over the Sentinel-2 images pixels grids (more details are
provided in section 2.1.1), which represents a total of $5136$ SAR images. We
believe that since a large part of the earth is covered only with single orbit
(i.e. ascending or descending), our study results would be more easily
reproducible with a single orbit for SAR images, hence we use only the
ascending orbit over our study area. Table 1 summarizes the number of Sentinel
images used for this study. The total area covered by the Sentinel tiles is
$106.7\times 10^{3}km^{2}$. The following sections details the Sentinel-1 and
Sentinel-2 products.

Figure 1: The region of interest, located in the Occitanie area (south of
France mainland). Sentinel-2 images envelopes are plotted in black. Map data ©
OpenStreetMap contributors, CC BY-SA

####  2.1.1 Sentinel-1 images

We have used the so-called S1Tiling tool111https://gitlab.orfeo-
toolbox.org/s1-tiling/s1tiling to automatically download and process the
Sentinel-1 images. The tool performs the orthorectification and the
calibration in sigma nought of the VV and VH SAR images channels. It also
projects and resamples the final images over the same coordinate reference
system and pixel grid as the Sentinel-2 images, at 10m physical spacing
resolution.

####  2.1.2 Sentinel-2 images

The Theia Land data center222https://www.theia-
land.fr/en/product/sentinel-2-surface-reflectance/ provides Sentinel-2 images
in surface reflectance. The products are computed using MACCS (Multi-sensor
Atmospheric Correction and Cloud Screening), a level 2A processor which
detects the clouds and their shadows, and estimates aerosol optical thickness,
water vapour and corrects for the atmospheric effects Hagolle et al. (2015).
While level 1C processing level could have been used in this study, we chose
level 2A products because they include cloud masks that are useful meta-data.
Indeed these information suffice to derive a cloud coverage percentage over
patches. Figure 2 shows the cloud coverage computed for each location from the
number of cloudy pixels among available ones in the temporal dimension. The
average cloud coverage over the area is $39.1\%$ and the standard deviation
$6.8\%$. An evaluation of the cloud masks is provided in Baetens et al.
(2019). To discharge storage and computational requirements, we used only the
10m spacing bands, i.e. spectral bands number 2, 3, 4 and 8.

Tile | S1 | S2
---|---|---
31TCH | 684 | 374
31TCJ | 456 | 380
31TCK | 454 | 273
31TDH | 462 | 370
31TDJ | 462 | 302
31TDK | 462 | 286
31TEJ | 672 | 512
31TEK | 595 | 311
31TFJ | 443 | 408
31TFK | 446 | 377
Table 1: Number of Sentinel-1 (S1) and Sentinel-2 (S2) images used over the
Occitanie area (France) Figure 2: The cloud coverage over the area. Left: the
percentage of cloud-free pixels in computed at each location. Right: histogram
of the values. Mean cloud coverage: $39.13\%$, standard deviation: $6.80\%$.
Map data © OpenStreetMap contributors, CC BY-SA

####  2.1.3 Digital Elevation Model

The Digital Elevation Model (DEM) from the Shuttle Radar Topography Mission
(SRTM)Farr and Kobrick (2000) is used to perform the processing of Sentinel-1
images. It is also used in our $SSOP_{unet+DEM}$ and $MSOP_{unet+DEM}$
networks, introduced in the following sections.

##  3 Architectures

The implemented deep learning based models are detailed in the subsections
below. The number of trainable parameters, batch size used per GPU, and
training time is summarized in table 2 for each model.

Model | Nb. params | Batch | Training time
---|---|---|---
$SSOP_{mer}$ | 18,905M | 8 | 3136h.GPU
$SSOP_{unet}$ | 16.652M | 64 | 104h.GPU
$MSOP_{unet}$ | 29.272M | 32 | 149h.GPU
Table 2: Number of parameters in the trained models, batch size used, and
training time (in hours.GPUs) for the training step on a NVIDIA V100 GPU with
32Gb RAM. The training time corresponds to the duration required to computed
the best model.

###  3.1 Single SAR/Optical pair (SSOP)

We denote SSOP, the approaches that input one SAR image and one optical image
polluted by cloud, and which reconstruct the missing parts of the optical
image. Introduced in Meraner et al. (2020), this kind of approach is trained
and evaluated from samples composed of $(S1_{t}$, $S2_{t}$, $S2_{t^{\prime}})$
triplets, where $S2_{t}$ is an optical image potentially polluted by clouds,
with $S1_{t}$ and $S2_{t}$ acquired close together and $S2_{t^{\prime}}$ a
cloud-free optical image acquired close to the ($S1_{t},S2_{t}$) pair. Figure
3 illustrates the architecture of this family of networks.

Figure 3: SSOP network. $S1_{t}$ and $S2_{t}$ denotes the input pair of
Optical and SAR images acquired at date $t$. The $S2_{t^{\prime}}$ denotes the
reference image used to compute the loss (during training) and the metrics (at
inference time from test data) from the reconstructed optical image
$\widehat{S2_{t}}$.

####  3.1.1 $SSOP_{mer}$

We implement the network described in Meraner et al. (2020), which uses a
residual correction from a ResNet backbone He et al. (2016) to reconstruct the
output optical image from a pair of one SAR image and one cloudy optical
image. However, the authors of Meraner et al. (2020) did not have
$S2_{t^{\prime}}$ acquisitions systematically close to $S2_{t}$, therefore
they use an additional loss based on cloud masks to encourage the identity
transformation of cloud-free pixels. Since our goal is to use a simple
information about the presence of clouds, namely an approximation of the cloud
cover percentage in a Sentinel-2 image or a set of patches, we don’t use such
pixel-wise cloud-mask based loss. Moreover, as we control the gathering of
samples matching the acquisitions layout described in table 4, a cloud-free
$S2_{t^{\prime}}$ image acquired close to $S2_{t}$ is always available. We
hypothesize that the contribution of such loss is likely marginal since none
or very little changes should happen between $S2_{t^{\prime}}$ and $S2_{t}$.
Hence we train the network using only the $l_{1}$ loss. We denote $SSOP_{mer}$
the implementation of this network.

####  3.1.2 $SSOP_{unet}$

We implement a modified version of the previously described architecture,
employing a U-Net backbone Ronneberger et al. (2015) instead of ResNet. Our
motivation behind this modification is that the ResNet backbone has two
disadvantages compared to U-Net: (i) convolutions are applied without any
stride, which consumes a lot more memory and requires much more operations
since all convolutions apply over the entire images at their native resolution
i.e. without any downsampling of the features, (ii) all inputs have to be
resampled at the smallest input images resolution, i.e. a physical spacing of
10 m, since all the network features are computed at the same resolution and
no downscaling is performed across the network, which is computationally
counterproductive. We denote $SSOP_{unet}$ our implementation of this modified
network. To illustrate the advantage of additional inputs at a lower
resolution, we use an additional Digital Elevation Model (DEM) as input,
resampled at 20m, as shown in figure 4. Our modified network is illustrated in
figure 5. We denote $SSOP_{unet+DEM}$ the model with the input DEM. It can be
noted that such a model could also generate outputs of different resolutions,
typically the Sentinel-2 spectral bands at 20m.

Figure 4: Proposed architecture for encoder ($E$) and decoder ($D$), enabling
the use of inputs of different resolutions. First and last convolutions use a
unitary stride and a kernel of size 5. Other convolutions use strides 2 and a
kernel of size 3. Skip connections between the encoder and the decoder perform
the concatenation of the features from the encoder with the decoder outputs.
All convolutions except the last k5s1 convolution are followed with a ReLU
activation function. No batch normalization is used. Figure 5: Our modified
SSOP network. $S1_{t}$ and $S2_{t}$ denotes the input pair of Optical and SAR
images. The $S2_{t^{\prime}}$ denotes the reference image used to compute the
loss (during training) and the metrics (at inference time from test data) from
the reconstructed optical image $\widehat{S2_{t}}$. (E) and (D) denotes
respectively the encoder and the decoder of the U-Net backbone.

###  3.2 Multiple SAR/Optical pairs (MSOP)

A number of approaches using multiple pairs of optical and SAR images have
been presented in the literature. For instance, Scarpa et al. (2018) use two
pairs of SAR/optical acquired before and after date $t$, a Digital Elevation
model (DEM), and an additional SAR image acquired at date $t$ to estimate
radiometric indices at date $t$. Conversely, Cresson et al. (2019) use
multiple optical and SAR images to generate a synthetic optical image at date
$t$. While these works were carried on input cloud-free images, a similar
network architecture can also be applied on cloudy input images to retrieve
the missing contents of the optical image at date $t$.

####  3.2.1 $MSOP_{unet}$

We build a multitemporal network inspired from the architectures presented in
Scarpa et al. (2018) and Cresson et al. (2019). We generalize to multitemporal
the approach of Meraner et al. (2020) with a new architecture that inputs
mutliple SAR/optical images pairs at $t-1$, $t$ and $t+1$ and a DEM, aiming to
reconstruct the potentially damaged optical image at $t$. We use a similar
architecture as the encoder/decoder U-Net backbone of the $SSOP_{unet}$ model,
except that encoder weights are shared for $t-1$, $t$ and $t+1$ inputs, and
features from three encoders ($E$) are concatenated before being processed
with the decoder ($D$), which outputs the reconstructed optical image at $t$.
Unlike Meraner et al. (2020), our model does not employ residual connections
to generate the reconstructed optical image (figure 6).

Figure 6: $MSOP_{unet}$ network. $(S1_{t-1},S2_{t-1})$, $(S1_{t},S2_{t})$ and
$(S1_{t+1},S2_{t+1})$ denotes the input pairs of optical and SAR images.
$S2_{t^{\prime}}$ denotes the reference image used to compute the loss (during
training) and the metrics (at inference time from test data) from the
reconstructed optical image $\widehat{S2_{t}}$.

We denote $MSOP_{unet+DEM}$ the MSOP model using the input DEM in (E) as shown
in figure 4.

###  3.3 Gap-filling

The Gap-filling consists in interpolating temporally close optical images to
approximate one target image Inglada et al. (2015). While gap-filling is not a
reconstruction method, i.e. the input image at $t$ is not used, it is commonly
used as such, in estimating the parts of the image that is polluted by clouds.
Gap-filling is restricted to cloud-free input images, and do not use SAR
images. In the case of a linear model, the generated output image
$\widehat{S2_{t}}$ can be written using the following formula:

$\widehat{S2_{t}}=S2_{t-1}+(S2_{t+1}-S2_{t-1})\times\frac{T_{t}-T_{t-1}}{T_{t+1}-T_{t-1}}$
(1)

Where $T$ is the timestamp, in seconds, of the dates.

##  4 Datasets

###  4.1 Acquisitions layouts

The so-called Acquisitions layout describes inputs and targets of a specific
use-case scenario. In the particular case of image reconstruction addressed in
this paper, the acquisitions layouts presented in the following sections have
one common item, namely $S2_{t^{\prime}}$ the target cloud-free optical image.
Depending on the approaches, the acquisitions layout can include additional
items, for instance:

  * •

A single SAR image

  * •

A single optical image

  * •

A pair of SAR + optical image

For each item, the acquisitions layout describes crucial properties:

  * •

For each optical image: a range of cloud coverage percentage (e.g. $[0,10]$),

  * •

For each (SAR, optical) pair: the maximum temporal gap between the two images
acquisition dates, in hours,

  * •

For each SAR or optical image: the acquisition date range, relative to a
reference item of the acquisitions layout (e.g. $[240h,360h]$).

This simple yet generic description formalizes of how the images are acquired
for a particular use-case scenario. We have carefully crafted acquisitions
layouts that represents the operational context of use of the approaches, i.e.
for which it is possible to use them on every available images. For instance,
to chose the maximum temporal gap between the SAR and the optical images
acquisition dates, we have analyzed the distribution of the temporal gap
between the closest (S1, S2) images (figure 7). Since more than $96\%$ of the
nearest (S1, S2) pairs are close to $72$ hours, we used this duration as the
maximum temporal gap in SAR-optical images pairs. We provide in section 4 all
acquisitions layouts suited for the training and testing of the involved
networks, and explain how the other parameters (i.e. time ranges for each
acquisition layout items) are chosen.

Figure 7: Distribution of the temporal gap between the closest (S1, S2) images
from the available images.

###  4.2 Patches indexation

Figure 8 gives an overview of the dataset creation step. In order to perform
the query of samples, i.e. the search of groups of patches that match the
properties defined in the acquisitions layout, we use an indexation structure.
An R-Tree indexes all available patches. The space partitioning data structure
describes the following dimensions:

  * •

Cloud coverage (ranging from $0\%$ to $100\%$)

  * •

Time from the reference image of the acquisitions layout

  * •

Duration to the closest valid SAR patch

  * •

Number of pixels different from the no-data value

A Kd-Tree is used to ease the computation of the duration to the closest valid
SAR patch. The R-Tree is built for each patches of Sentinel tiles using the
acquisition dates provided in the Sentinel images metadata, and the following
statistics collected on the Sentinel images patches:

  * •

For Sentinel-1 images: the number of valid pixels,

  * •

For Sentinel-2 images: the number of valid pixels, and the number of pixels
impacted by clouds. To compute this last, we use the cloud quality mask
provided in the Theia product.

We note that cloud masks are only used as a single value for each patch,
representing the proportion of cloud coverage. We have computed the R-Trees
from non-overlapping, $256\times 256$ sized patches in all Sentinel tiles.
This indexation structure is computed once. After that, any acquisitions
layout can be used to query all samples matching the defined properties of the
remote sensing acquisitions. One generated sample includes data arrays
containing pixels and ancillary data, e.g. acquisition date, for each items of
the acquisitions layout. Finally, the samples are restricted in the provided
region of interest, to allow the generation of mutually exclusive samples in
the geographical domain, i.e. in training, validation and test datasets.

Figure 8: Workflow for the creation of datasets. The sample query use a R-Tree
indexing the available Sentinel-1 and Sentinel-2 images patches.

###  4.3 Generation of samples

For each network, samples are first extracted in the images from the specific
acquisitions layout. We split the samples in three groups: training,
validation, and test. We ensure that these three groups are mutually exclusive
by randomly selecting their location in the geographical domain, without
overlapping patches of distinct groups. We have randomly selected $5\%$ and
$15\%$ of the area to form the region of interest for the validation and test
datasets, and the other $80\%$ has been used to form training datasets. Since
the swath of Sentinel-1 and 2 does overlap in some areas, the samples density
is heterogeneous in spatial domain. For this reason, we have limited the
number of samples per spatial location in the training and validation
datasets, to guarantee that models are trained with the same number of patches
at each location. Thus for the training and validation datasets, a maximum
amount of $50$ samples has been collected at each locations. For the test
dataset, all available samples are extracted. Table 3 summarize the number of
distinct samples for training, validation and test datasets. The differences
in samples number is due to the availability of images, or pairs of images,
depending on the properties defined in the acquisitions layout, i.e. number
and type of acquisitions, cloud coverage and temporal constraints. All
acquisitions layouts used to generate the datasets are detailed in the
following subsections.

Dataset | Training | Validation | Test
---|---|---|---
SSOP | $600.1k$ | $35.3k$ | $70.6k$
MSOP | $600.1k$ | $35.3k$ | $70.6k$
MSOPcld | / | / | $76k$
Table 3: Number of samples in each datasets. The SSOP, MSOP and MSOPcld
datasets are used respectively to train the mono-temporal networks, the multi-
temporal networks, and to compare all models with the gap-filling over an
acquisitions layout matching all approaches validity domains.

In the following sections, we detail the properties of each acquisitions
layout of the datasets.

####  4.3.1 SSOP (Single date inputs cloudy optical image)

The acquisitions layout for SSOP networks training is presented in table 4. We
chose the following parameters for the acquisitions layout: the maximum gap
between the acquisition dates of S1 and S2 images is set to $72$ hours and the
maximum spread between the cloud-free and the polluted optical images is set
to $10$ days. With this settings, we reach a total number of $600k$ samples
for training, which is approximately 4 times the amount of samples used in the
original paper of Meraner et al. (2020).

Name | S1 ($\pm\Delta t$) | S2 (% clouds) | Time-stamp
---|---|---|---
$t$ | Yes ($\pm 72$h) | $[0,100]$ | Reference
$t^{\prime}$ | / | $0$ | $[-10d,+10d]$
Table 4: Acquisitions layout for the SSOP networks, depicting the ($S1_{t}$,
$S2_{t}$) images pair acquired close to a single cloud-free optical image
($S2_{t^{\prime}}$).

####  4.3.2 MSOP (Multitemporal inputs, any optical images i.e. cloudy or not)

The acquisitions layout for the training of the $MSOP_{unet}$ network is
presented in table 5. It consists of three optical images at $t-1$, $t$ and
$t+1$ that can be polluted by clouds, and one cloud-free optical image at
$t^{\prime}$, used as the training target. As explained in section 4.1, we
used $72$ hours for the maximum gap between the acquisition dates of S1 and S2
images. The cloud-free optical image is acquired at most $10$ days from the
optical image at $t$, to roughly falls within the less frequent revisit cycle
of the Sentinel-2 constellation everywhere over our study area. Finally, we
have selected a temporal range for $t-1$ and $t+1$ dates that avoids the
cloud-free optical image acquisition date, and that also falls within the
revisit cycle of the Sentinel-2 constellation, i.e. $10$ to $18$ days.

Name | S1 ($\pm\Delta t$) | S2 (% clouds) | Time-stamp
---|---|---|---
$t-1$ | Yes ($\pm 72$h) | $[0,100]$ | $[-18d,-10d]$
$t$ | Yes ($\pm 72$h) | $[0,100]$ | reference
$t^{\prime}$ | / | $0$ | $[-10d,+10d]$
$t+1$ | Yes ($\pm 72$h) | $[0,100]$ | $[+10d,+18d]$
Table 5: Acquisitions layout used to train the $MSOP_{unet}$ network.

###  4.4 MSOPcld (Multitemporal inputs, with cloud-free optical images at
$t-1$ and $t+1$, and one cloudy image at $t$)

Table 6 shows one acquisitions layout enabling the comparison of the gap-
filling with the SSOP and MSOP networks, thanks to cloud-free optical images
available at $t-1$ and $t+1$, and one completely cloudy optical images at $t$,
which intends to make as fair as possible the comparison. We denote the
corresponding dataset MSOPcld. In this acquisitions layout, the cloud-free
optical image at $t^{\prime}$, acquired at most $5$ days from the date $t$, is
used to compute the metrics over the reconstructed image.

We note that our settings make possible the extraction of a sufficient number
of samples, thanks to the availability of Sentinel-1 and Sentinel-2 over our
study site, but this setting might be adjusted for other regions of the world
where the Sentinel coverage is less timely available.

Name | S1 ($\pm\Delta t$) | S2 (% clouds) | Time-stamp
---|---|---|---
$t-1$ | Yes ($\pm 72$h) | $0$ | $[-18d,-10d]$
$t$ | Yes ($\pm 72$h) | $100$ | reference
$t^{\prime}$ | / | $0$ | $[-5d,+5d]$
$t+1$ | Yes ($\pm 72$h) | $0$ | $[+10d,+18d]$
Table 6: Acquisitions layout enabling the comparison of the gapfilling and the
$MSOP_{unet}$ network. The cloud coverage at $t$ is $100\%$, all other optical
images remain cloud free.

##  5 Benchmarks

###  5.1 Protocol

We train all networks with their respective datasets presented in section 2.
The SSOP and MSOP models are trained over the training dataset detailed in
section 4.3.1, and evaluated over the test datasets detailed in sections 4.3.2
and 4.4. The MSOP models are trained over the dataset detailed in section
4.3.2, and evaluated on the test datasets detailed in sections 4.3.2 and 4.4.
We use the ADAM algorithm Kingma and Ba (2014) to train all networks to
minimize the $l_{1}$ loss. For $MSOP_{unet}$ and $SSOP_{unet}$, we use a
learning rate of $lr=0.00012$, $\beta_{0}=0.9$ and $\beta_{1}=0.999$, with a
batch of size 128 distributed across 4 GPUs. We train the $SSOP_{mer}$ network
using the same setup as described by the authors. All experiments are realized
on NVIDIA V100 GPUs with 32Gb RAM. We kept the trained models that reach the
smallest $l_{1}$ loss on the validation split. To assess the performance of
each approach, we compute the following metrics between the reconstructed
output optical images $\widehat{S2_{t}}$ and the reference cloud-free optical
image $S2_{t^{\prime}}$ over the test datasets:

  * •

Peak Signal to Noise Ratio (PSNR):

$PSNR=10\times log_{10}(\frac{d^{2}}{MSE})$ (2)

Where MSE is the Mean Squared Error computed over $n$ patches:

$MSE=\frac{1}{n}\sum^{n}{\lVert\widehat{S2_{t}}-S2_{t^{\prime}}\rVert^{2}}$
(3)

The higher is the PSNR, the closer are the values of the estimated image to
the target image.

  * •

Spectral angle (SAM)Kruse et al. (1993), representing the mean spectral angle
between the estimated image and the target image, ranging in $[0,\pi]$

  * •

The Structural Similarity Index (SSIM)Wang et al. (2004), measures the
similarity in terms of structure, between the estimated image and the target
image. The range is $[0,1]$, and values close to $1$ correspond to the best
structural similarity between the images.

###  5.2 Ablation study

In order to assess the SAR, DEM, and optical modalities benefits, we have
performed an ablation study. For the SSOP and MSOP networks, we have derived
two modified architectures, one without the SAR input, and the other without
DEM and without SAR. For a sake of computational budget, we only have studied
the ablation with the U-Net based networks.

###  5.3 Results

In the following section, we report the evaluation metrics computed on all
test datasets.

####  5.3.1 Comparison of SSOP networks

The comparison between SSOP networks is carried out on the test dataset
presented in section 4.3.1. We first report the metrics obtained with SSOP
networks in table 7. It can be noticed that the metrics from the modified
network ($SSOP_{unet}$) are close to the original $SSOP_{mer}$. SSIM and SAM
are slightly better for $SSOP_{mer}$ and MSE and PSNR a bit better for
$SSOP_{unet}$. While this result is not groundbreaking in terms of evaluation
metrics, we highlight the huge difference of required overall processing time:
to train both networks over the same dataset with the same setup,
$SSOP_{unet}$ needs $\approx 30$ hours and $SSOP_{mer}$ $\approx 35$ days. For
this particular reason, we have chosen to perform all other benchmarks only on
$SSOP_{unet}$, since the metrics are quite similar to $SSOP_{mer}$, but the
processing budget far lower and we could ran more experiment at lower cost. It
can be noticed that the $SSOP_{unet+DEM}$ model has the best PSNR, MSE and
SAM, but the $SSOP_{mer}$ still has a slightly better SSIM. One explanation
could be that in the $SSOP_{mer}$ model, all convolutions are performed in the
original resolution, and no downsampling is performed, which might preserve
the structural similarity, hence a better SSIM. The lowest metrics are
obtained with the $SSOP_{unet,w/oSAR}$ model, which does not use DEM and SAR
inputs, showing the benefits of these modalities for the reconstruction.

Model | MSE | SSIM | PSNR | SAM
---|---|---|---|---
$SSOP_{unet,w/oSAR}$ | 324508 | 0.8388 | 24.888 | 0.1595
$SSOP_{mer}$ | 277971 | 0.8656 | 25.560 | 0.1425
$SSOP_{unet}$ | 261223 | 0.8568 | 25.830 | 0.1448
$SSOP_{unet+DEM}$ | 234410 | 0.8645 | 26.300 | 0.1401
Table 7: Comparison of SSOP networks performed on the SSOP dataset

Figure 9 shows images from the test dataset, processed with the different
U-Net based SSOP networks. We can visually appreciate the contributions of the
input SAR and DEM. It can be noticed the limits of the method with thick
clouds in the optical image, especially for the $SSOP_{unet,w/oSAR}$ network
that only use the input optical image. Figure 10 show the limits of the
$SSOP_{unet+DEM}$ network with very thick atmospheric perturbation.

Figure 9: From left to right: input cloudy optical image $S2_{t}$ (1), input
SAR image $S1_{t}$ (2), output $\widehat{S2_{t}}$ from $SSOP_{unet,w/oSAR}$
(3), $SSOP_{unet}$ (4), and $SSOP_{unet+DEM}$ (5), (6) reference image

Figure 10: Limits of the SSOP networks with thick clouds in optical images.
Top left: input cloudy optical image $S2_{t}$, Top right: SSOP network output
$\widehat{S2_{t}}$. Bottom: the reference image $S2_{t^{\prime}}$.

###  5.4 Comparison of SSOP and MSOP networks

The comparison of approaches that input one or more cloudy images to
reconstruct the optical image at $t$, is carried out. We compare networks that
consume different kind of inputs, i.e. one single (S1, S2) pair for SSOP
networks versus three pairs of images for MSOP networks. We recall that,
unlike the MSOP networks, $t-1$ and $t+1$ images are not used by the SSOP
networks. We compare the networks on the test dataset detailed in section
4.3.2, since its acquisitions layout fulfills both MSOP and SSOP models
validity domains, in particular the maximum SAR-optical temporal gap at $t$.
Evaluation metrics are reported in table 8. While it can be observed the same
outcome in the comparison between SSOP networks, these quality metrics differ
a bit from the ones presented in table7, since the evaluated samples are just
a subset of this last dataset. The qualitative inspection of the reconstructed
images shows that the $MSOP_{unet+DEM}$ network produces images better
reconstructed than the $MSOP_{unet}$ and $MSOP_{unet,w/oSAR}$ networks,
especially under thick cloudy areas, highlighting the importance of the SAR
and DEM modalities (figure 11).

Model | MSE | SSIM | PSNR | SAM
---|---|---|---|---
$SSOP_{unet,w/oSAR}$ | 324099 | 0.8388 | 24.893 | 0.1595
$SSOP_{unet}$ | 260827 | 0.8567 | 25.836 | 0.1448
$SSOP_{unet+DEM}$ | 221909 | 0.8583 | 26.538 | 0.1390
$MSOP_{unet,w/oSAR}$ | 141283 | 0.9249 | 28.499 | 0.1128
$MSOP_{unet}$ | 138212 | 0.9267 | 28.594 | 0.1111
$MSOP_{unet+DEM}$ | 133061 | 0.9277 | 28.759 | 0.1095
Table 8: Comparison between SSOP and MSOP networks, over the MSOP test dataset
detailed in section 4.3.2. Figure 11: From left to right: input images
$S2_{t-1}$ (1), $S2_{t}$ (2), $S2_{t+1}$ (3), output reconstructed optical
images $\widehat{S2_{t}}$ from $MSOP_{unet,w/oSAR}$ (4), $MSOP_{unet}$ (5),
$MSOP_{unet+DEM}$ (6) and the reference image $S2_{t^{\prime}}$ (7).

####  5.4.1 Comparison of deep-learning based approaches and gap-filling

In this setup, optical images acquired at $t-1$ and $t+1$ are completely
cloud-free, which enables the use of the gap-filling. Also, the optical image
acquired at $t$ is completely covered by clouds or clouds shadows, according
to the cloud masks, helping toward a fair comparison between the approach that
consume the optical image at $t$ ($MSOP_{unet}$) and the gap-filling. We
perform the comparison of MSOP, SSOP models and the gap-filling using the
MSOPcld test dataset detailed in table 6, which matches the validity range of
all approaches. Table 9 reports the metrics obtained. We can observe that all
the metrics are in favor of the MSOP models. Also, in this particular use-
case, the gap-filling leads to superior results to the SSOP models. We can
notice that all metrics are largely in favor of the $MSOP_{unet+DEM}$ model,
showing the benefit of the multitemporality, the SAR and the DEM modalities.
Figure 12 shows reconstructed images from the test dataset. We can notice that
the gap-filling fails to retrieve various details in the reconstructed images,
like sudden crops changes. Also, the input images cloud masks are not always
exact, and the gap-filling might interpolates polluted images, unlike the MSOP
models which are capable of removing those clouds.

Model | MSE | SSIM | PSNR | SAM
---|---|---|---|---
$SSOP_{unet,w/oSAR}$ | 239238 | 0.7911 | 26.212 | 0.1847
$SSOP_{unet}$ | 178284 | 0.8187 | 27.489 | 0.1472
$SSOP_{unet+DEM}$ | 157663 | 0.8264 | 28.023 | 0.1422
gapfilling | 79904 | 0.9249 | 30.974 | 0.1021
$MSOP_{unet,w/oSAR}$ | 63097 | 0.9338 | 32.000 | 0.0952
$MSOP_{unet}$ | 61016 | 0.9345 | 32.146 | 0.0940
$MSOP_{unet+DEM}$ | 52814 | 0.9421 | 32.772 | 0.0901
Table 9: Comparison of the gap-filling and the SSOP and MSOP networks. All
approaches are compared on the MSOPcld test dataset detailed in table 6.
Figure 12: From left to right: input images $S2_{t-1}$ (1), $S2_{t}$ (2),
$S2_{t+1}$ (3), output reconstructed optical images $\widehat{S2_{t}}$ from
$MSOP_{unet+DEM}$ (4), the Gap-filling (5), and the reference image
$S2_{t^{\prime}}$ (6). In violet are circled details in the reconstructed
images that the gap-filling fails to retrieve, or artifacts caused by wrong
could masks in input images.

##  6 Discussion

We have compared various single date SAR/optical networks, with an ablation
study to analyze the contribution of the SAR, optical, and DEM inputs. We have
modified the original network from Meraner et al. (2020), which is
considerably greedy in term of computational resources, replacing the ResNet
backbone with a U-Net backbone. This has two advantages: first, it is less
computationally extensive since convolutions are performed on downsampled
features maps. The processing time is diminished with a factor greater than
30, and leads to similar image quality metrics, with a slightly higher PSNR,
but slightly lower SAM and SSIM. Secondly, using a U-Net backbone instead of a
ResNet backbone enables input images at lower resolution that the 10m bands of
Sentinel images: we have shown that a 20m spacing DEM can be injected after
the first downsampling of the network without prior spatial re-sampling,
improving the reconstruction of optical images. However, we only have trained
all single date based networks using the only the $l_{1}$ loss, and future
works could investigate other objective formulations. We have carried out the
comparison of single date networks and the multitemporal networks over the
MSOP test dataset, which represents the nominal operational context of both
networks, e.g. using every available input images, cloudy or not. Our results
shown that the multitemporal networks lead to superior image reconstruction.
We believe that more available input images improves the retrieval of the
missing contents of the cloudy optical image at $t$. The comparison between
the deep learning based networks and the gap-filling is performed over the
MSOPcld test dataset, which contains samples where the $t$ optical image
patches are covered by clouds at $100\%$. The gap-filling performs better than
the single date network with a significant margin in this particular setup.
However, even though the multitemporal network is not primarily designed for
this task, it has outperformed the gap-filling. Finally our ablation study
shows that the SAR and the SAR+DEM contribute in the optical image
reconstruction process in both single date based networks, and multitemporal
based networks. For future works, we believe that a further investigation of
the SAR signal contribution should be carried out. For instance, it could be
interesting to study if feeding geometrical information (e.g. local SAR
incidence angle) in networks would help, and if physical based SAR pre-
processing (i.e despeckeling or target decomposition) benefit the optical
image reconstruction task.

##  7 Summary and conclusion

In this paper, we sought to provide a comparison of single date based and
multitemporal convolutional networks with the traditional deterministic
temporal interpolation between two images. We have introduced a framework to
generate various datasets to train and evaluate various methods for cloudy
optical image reconstruction. Our simple yet convenient method relies on space
partitioning data structures indexing the crucial parameters of the remote
sensing acquisitions, i.e. how SAR and optical remote sensing images must be
acquired in the datasets, in term of cloud coverage, SAR/optical maximum gap,
number and type of acquisition, and relative acquisition time. We have built
several datasets to train single date based networks and multitemporal
networks, and to evaluate the different selected approaches, representing
various operational contexts. The studied single date based network take their
roots in an existing architecture that uses a ResNet backbone, and we have
shown how it could be improved using a U-Net backbone, increasing its training
and inference speed and enabling to input additional image of different scale.
We have built a multitemporal network that generalize the single date image
reconstruction from three input pairs of images, and which uses the same
backbone shared across the inputs. Our model inputs three cloudy optical and
SAR images pairs acquired at dates $t-1$, $t$ and $t+1$, and a DEM. We have
lead the comparison between the single date networks, the multitemporal
networks, and the gapfilling in various contexts, showing that the gapfilling
performs better than the single date based networks in the context of
Sentinel-2 time series. We have analyzed the contribution of the different
kind of inputs, namely optical, SAR and DEM with an ablation study, showing
how the reconstructed image benefits from these modalities. Also, we have
shown that, even if the primary design of the multitemporal convolutional
network is not focused on image interpolation in temporal domain, it leads to
similar even better results than the gap-filling. However, we should interpret
our results carefully regarding the ancillary data available for cloud
coverage characterization, since our cloud coverage information per patch
depends from it, and the bias it can introduce. Finally, we lead our study
over a small area that do not represents the various atmospheric conditions
all over the earth. With the continuous stream of synchronized SAR and optical
acquisitions thanks to the Sentinel constellation, it is expected that future
data driven multitemporal models will help to deliver more exploitable data.
Our dataset generation framework and our models are available as open-source
software 333http://github.com/cnes/decloud

## ACKNOWLEDGEMENTS

The authors would like to thank the reviewers for their valuable suggestions.
This work was granted access to the HPC resources of IDRIS under the
allocation AD011011608 made by GENCI.

## References

  * Shen et al. [2015] Huanfeng Shen, Xinghua Li, Qing Cheng, Chao Zeng, Gang Yang, Huifang Li, and Liangpei Zhang.  Missing information reconstruction of remote sensing data: A technical review.  _IEEE Geoscience and Remote Sensing Magazine_ , 3(3):61–85, 2015.
  * Hu et al. [2015] Gensheng Hu, Xiaoyi Li, and Dong Liang.  Thin cloud removal from remote sensing images using multidirectional dual tree complex wavelet transform and transfer least square support vector regression.  _Journal of Applied Remote Sensing_ , 9(1):095053, 2015.
  * Cheng et al. [2014] Qing Cheng, Huanfeng Shen, Liangpei Zhang, Qiangqiang Yuan, and Chao Zeng.  Cloud removal for remotely sensed images by similar pixel replacement guided with a spatio-temporal mrf model.  _ISPRS journal of photogrammetry and remote sensing_ , 92:54–68, 2014.
  * Li et al. [2014] Xinghua Li, Huanfeng Shen, Liangpei Zhang, Hongyan Zhang, Qiangqiang Yuan, and Gang Yang.  Recovering quantitative remote sensing products contaminated by thick clouds and shadows using multitemporal dictionary learning.  _IEEE Transactions on Geoscience and Remote Sensing_ , 52(11):7086–7098, 2014.
  * Eckardt et al. [2013] Robert Eckardt, Christian Berger, Christian Thiel, and Christiane Schmullius.  Removal of optically thick clouds from multi-spectral satellite images using multi-frequency sar data.  _Remote Sensing_ , 5(6):2973–3006, 2013.
  * Zhang et al. [2018] Qiang Zhang, Qiangqiang Yuan, Chao Zeng, Xinghua Li, and Yancong Wei.  Missing data reconstruction in remote sensing image with a unified spatial–temporal–spectral deep convolutional neural network.  _IEEE Transactions on Geoscience and Remote Sensing_ , 56(8):4274–4288, 2018.
  * Liu et al. [2019] Xun Liu, Chenwei Deng, Jocelyn Chanussot, Danfeng Hong, and Baojun Zhao.  Stfnet: A two-stream convolutional neural network for spatiotemporal image fusion.  _IEEE Transactions on Geoscience and Remote Sensing_ , 57(9):6552–6564, 2019.
  * Scarpa et al. [2018] G. Scarpa, M. Gargiulo, A. Mazza, and R. Gaetano.  A cnn-based fusion method for feature extraction from sentinel data.  _Remote Sensing_ , 10(2):236, 2018.
  * Cresson et al. [2019] Rémi Cresson, Dino Ienco, Raffaele Gaetano, Kenji Ose, and D Ho Tong Minh.  Optical image gap filling using deep convolutional autoencoder from optical and radar images.  In _IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium_ , pages 218–221. IEEE, 2019.
  * Sarukkai et al. [2020] Vishnu Sarukkai, Anirudh Jain, Burak Uzkent, and Stefano Ermon.  Cloud removal from satellite images using spatiotemporal generator networks.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1796–1805, 2020.
  * Meraner et al. [2020] Andrea Meraner, Patrick Ebel, Xiao Xiang Zhu, and Michael Schmitt.  Cloud removal in sentinel-2 imagery using a deep residual neural network and sar-optical data fusion.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 166:333–346, 2020.
  * Ebel et al. [2020] Patrick Ebel, Andrea Meraner, Michael Schmitt, and Xiao Xiang Zhu.  Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery.  _IEEE Transactions on Geoscience and Remote Sensing_ , 59(7):5866–5878, 2020.
  * Inglada et al. [2015] J. Inglada, M. Arias, B. Tardy, O. Hagolle, S. Valero, D. Morin, G. Dedieu, G. Sepulcre, S. Bontemps, P. Defourny, and B. Koetz.  Assessment of an operational system for crop type map production using high temporal and spatial resolution satellite optical imagery.  _Remote Sensing_ , 7(9):12356–12379, 2015.
  * Hagolle et al. [2015] Olivier Hagolle, Sylvia Sylvander, Mireille Huc, Martin Claverie, Dominique Clesse, Cécile Dechoz, Vincent Lonjou, and Vincent Poulain.  Spot-4 (take 5): simulation of sentinel-2 time series on 45 large sites.  _Remote sensing_ , 7(9):12242–12264, 2015.
  * Baetens et al. [2019] Louis Baetens, Camille Desjardins, and Olivier Hagolle.  Validation of copernicus sentinel-2 cloud masks obtained from maja, sen2cor, and fmask processors using reference cloud masks generated with a supervised active learning procedure.  _Remote Sensing_ , 11(4):433, 2019.
  * Farr and Kobrick [2000] Tom G Farr and Mike Kobrick.  Shuttle radar topography mission produces a wealth of data.  _Eos, Transactions American Geophysical Union_ , 81(48):583–585, 2000.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kruse et al. [1993] Fred A Kruse, AB Lefkoff, JW Boardman, KB Heidebrecht, AT Shapiro, PJ Barloon, and AFH Goetz.  The spectral image processing system (sips)-interactive visualization and analysis of imaging spectrometer data.  In _AIP Conference Proceedings_ , volume 283, pages 192–201. American Institute of Physics, 1993.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.


# Towards Voice Reconstruction from EEG during Imagined Speech

Young-Eun Lee1, Seo-Hyun Lee1, Sang-Ho Kim2, Seong-Whan Lee2 Corresponding
author

###### Abstract

Translating imagined speech from human brain activity into voice is a
challenging and absorbing research issue that can provide new means of human
communication via brain signals. Endeavors toward reconstructing speech from
brain activity have shown their potential using invasive measures of spoken
speech data, however, have faced challenges in reconstructing imagined speech.
In this paper, we propose NeuroTalk, which converts non-invasive brain signals
of imagined speech into the user’s own voice. Our model was trained with
spoken speech EEG which was generalized to adapt to the domain of imagined
speech, thus allowing natural correspondence between the imagined speech and
the voice as a ground truth. In our framework, automatic speech recognition
decoder contributed to decomposing the phonemes of generated speech, thereby
displaying the potential of voice reconstruction from unseen words. Our
results imply the potential of speech synthesis from human EEG signals, not
only from spoken speech but also from the brain signals of imagined speech.

## Introduction

Brain signals contain various information related to human action or imagery,
making them valuable materials for understanding human intentions. Brain-
computer interface (BCI) is a technology of analyzing user’s brain activity to
derive external commands to control the environment through brain signals,
therefore, can benefit paralyzed or locked-in patients (Chaudhary, Birbaumer,
and Ramos-Murguialday 2016). Brain-to-speech (BTS) is a novel research stream
in the field of BCI, which aims to directly synthesize audible speech from
brain signals (Lee, Lee, and Lee 2022, 2020). While current studies of
decoding speech from human brain signals mainly focus on using spoken speech
brain signals measured with invasive methods (Anumanchipalli, Chartier, and
Chang 2019; Angrick et al. 2019; Herff et al. 2019), reconstructing imagined
speech using non-invasive modalities is a fascinating issue that can convert
user’s imagined speech into real voice. However, due to the fundamental
constraint of the imagined speech lacking the ground truth (GT) voice, it is
challenging to synthesize user’s own voice from imagined speech brain signals.

Since reconstructing speech from brain signals of spoken speech has shown its
potential (Anumanchipalli, Chartier, and Chang 2019; Angrick et al. 2019;
Herff et al. 2019), we anticipate that there must be a relevant brain
activation that may encode significant features of the speech. Imagined speech
is known to resemble the neural activation path of spoken speech, which is
mainly located on the ventral sensorimotor cortex (vSMC)(Watanabe et al. 2020;
Si et al. 2021; Cooney, Folli, and Coyle 2018; Lee, Lee, and Lee 2019). If
imagined speech has similar features to spoken speech, it may be possible to
link the spoken speech brain signals, spoken speech audio, and imagined speech
brain signals. Furthermore, if we could train and infer phonemes from imagined
speech, several unseen words composed of already trained phonemes can also be
reconstructed from the trained word sets.

In this study, we proposed NeuroTalk framework that can correlate imagined
speech electroencephalography (EEG) with spoken speech EEG and their
corresponding audio, to reconstruct voice from imagined speech. The imagined
utterances were decoded from EEG signals to reconstruct voice at a word level.
Moreover, we estimated the possibility of reconstructing unseen words using
the pre-trained model trained with only few words, to potentially expand the
degree of freedom using the model trained with minimal words including various
phonemes. Based on our results, we aim to find the potential of speech
reconstruction from imagined speech brain signals to the user’s own voice. The
main contributions are as follows:

### Main Contribution

  * •

We propose a generative model based on multi-receptive residual modules with
recurrent neural networks that can extract frequency characteristics and
sequential information from neural signals, to generate speech from non-
invasive brain signals.

  * •

The fundamental constraint of the imagined speech-based BTS system lacking the
ground truth voice have been addressed with the domain adaptation method to
link the imagined speech EEG, spoken speech EEG, and the spoken speech audio.

  * •

Unseen words were able to be reconstructed from the pre-trained model by using
character-level loss to adapt various phonemes. This implies that the model
could learn the phoneme level information from the brain signal, which
displays the potential of robust speech generation by training only several
words or phrases.

## Background

### Speech-related Paradigms

Speech-related paradigms mainly used in the BTS studies can be largely divided
into three categories: spoken speech, mimed speech, and imagined speech
(Schultz et al. 2017). While spoken speech indicates the natural speech that
accompanies vocal output and movement of the articulators, mimed speech does
not produce vocal output but accompanies the movement of the mouth and tongue
as if speaking out loud (Schultz et al. 2017; Cooney, Folli, and Coyle 2018).
Imagined speech is the mode of internally imagining speech, accompanying both
the imagery of the mouth movement and the vocal sound, without producing
actual movement or voice (Cooney, Folli, and Coyle 2018).

### Invasive Approach

Invasive measurements involve surgical process of implementation inside the
skull to capture brain activation directly from the cortex. Therefore, medical
risks and difficulties exist to be applied for healthy users (Wang and Ji
2021). However, due to the high signal-to-noise ratio (SNR), many previous
studies focused primarily on synthesizing speech from invasive brain signals.
Studies using electrocorticography (Akbari et al. 2019; Anumanchipalli,
Chartier, and Chang 2019; Angrick et al. 2019; Herff et al. 2019) and attempts
to decode speech from deeper brain structures using stereotactic
electroencephalography depth electrodes (Angrick et al. 2021a, 2022, b; Herff,
Krusienski, and Kubben 2020) have reported the possibility of speech
reconstruction using spoken speech data.

### Non-invasive Approach

#### Electroencephalography

EEG is the most widely used non-invasive modality for practical use, since it
does not involve any surgical process and are relatively easy to access
(Krishna et al. 2020). However, non-invasive measures have relatively low SNR
and artifact problems compared to the invasive modalities, which makes it hard
to extract user’s intention from brain signals (Graimann, Allison, and
Pfurtscheller 2009).

#### Spoken speech based BTS

Speech reconstruction from spoken speech or mimed speech brain signals,
kinematic or EMG data have shown potential (Gaddy and Klein 2020, 2021;
Gonzalez et al. 2017). However, a spoken speech-based BTS system cannot be the
final solution for the essential goal of BCI, since it is not a silent
communication (if the user can speak out, there’s no need to reconstruct
speech from brain signals), and it cannot be used for patients who cannot
speak or move.

#### Decoding imagined speech

Current technologies of decoding imagined speech from EEG have shown promising
results in terms of classification problems (Nguyen, Karavas, and Artemiadis
2017; Saha, Abdul-Mageed, and Fels 2019; Wang et al. 2013; Lee, Lee, and Lee
2020; Krishna et al. 2020). Previous works about imagined speech mostly
targeted classification tasks, or text decoding from EEG signals (Makin,
Moses, and Chang 2020). However, it is challenging to expand the number of
classes for the classification scenario (Lee, Lee, and Lee 2020). Also, to
provide an intuitive system that can generate voice from the brain signals,
speech reconstruction from the imagined speech is crucial.

#### Imagined speech based BTS

The fundamental constraint of speech reconstruction from EEG of imagined
speech is the inferior SNR, and the absence of vocal ground truth
corresponding to the brain signals. Therefore, speech synthesis from imagined
speech with non-invasive measures has so far not led to convincing results
(Proix et al. 2022). Attempts to reconstruct speech from invasive data during
whispered and imagined speech have existed, however, yet have been reported
relatively inferior performance with even invasive measures (Angrick et al.
2021b). Speech synthesis from imagined speech may be the key to open a new era
of human communication from current voice or text-based to brain-based
communication. Also, this may be a technology that can help patients who are
unable to speak or those who might lose their voice in the future.

## Method

Figure 1: Overall frameworks in this study. Imagined speech EEG were given as
the input to reconstruct corresponding audio of the imagined word or phrase
with the user’s own voice. $G$ refers generator, which generate mel-
spectrogram from embedding vector. $D$ refers discriminator, which distinguish
the validity of input. On the bottom part, the two model, pretrained vocoder
$V$ and a pretrained ASR model $A$, generate text from mel-spectrogram. Figure
2: The architecture details in (a) generator, (b) discriminator, and (c) MRF
module. The MRF modules in both generator and discriminator were repeated
three times in our experiment. $k_{r}$ indicates the kernel size of residual
block and $D_{r}$ indicates the dilation rates of the residual block.

In this section, we describe the model frameworks used in this paper,
including generator, discriminator, vocoder and automatic speech recognition
(ASR), as well as losses including reconstruction loss, generative adversarial
network (GAN) loss, and connectionist temporal classification (CTC), as shown
in Figure 1. The collected brain signals of spoken speech and imagined speech
are represented as feature embeddings to extract the optimal features from
brain signals. The generator applying GAN (Goodfellow et al. 2014)
reconstructs a mel-spectrogram to match the target voice during spoken speech.
The reconstruction loss for the generator is determined as the difference
between the reconstructed mel-spectrogram from the EEG signals and the ground
truth mel-spectrogram during spoken speech. The discriminator classifies the
validity of whether the input samples of mel-spectrogram are real or fake, and
calculates an adversarial loss for the generator and discriminator. ASR model
is a speech-to-text model, which can represent the speech as a contextual
sequence of discrete units (Baevski et al. 2020). The pretrained vocoder
converts the mel-spectrogram to a reconstructed voice, which is then
transformed into characters by the pretrained ASR model. The pre-trained ASR
model transforms the voice into text, and calculates the CTC loss for the
generator.

Since the voices were not recorded during the imagined speech, voices during
the spoken speech were used as the ground truth. To match the EEG to the voice
of spoken speech, dynamic time warping (DTW) was applied between the
reconstructed mel-spectrogram from EEG and the mel-spectrogram of voice during
spoken speech. Furthermore, domain adaptation (DA) was conducted to transfer
the architecture of spoken speech to that of imagined speech.

### Architectures

#### Embedding vector

It is known that spatial, temporal, and spectral information are all important
for speech-related brain signals, and vector-based brain embedding features
can represent the contextual meaning in brain signals (Goldstein et al. 2022;
Lee, Lee, and Lee 2020). The embedding vector was generated using common
spatial pattern (CSP) to maximize spatial patterns and log-variance to extract
temporal oscillation patterns. CSP finds the optimal spatial filters using
covariance matrices (Devlaminck et al. 2011), and helps to decode the brain
signals related to speech (Lee, Lee, and Lee 2020; Nguyen, Karavas, and
Artemiadis 2017).

To reduce the difference between the data distribution of spoken EEG and
imagined EEG, CSP filters were shared with both EEG signals. CSP filters were
trained with imagined EEG, which contains pure brain signals only, rather than
spoken EEG which may contains some noise. By sharing the CSP filters, spoken
EEG domain was adapted to the subspace of imagined EEG.

The CSP filters were trained with eight CSP features and sixteen segments
without overlap using training dataset. Each trial of EEG signals has a size
of time point $\times$ channels (5000 $\times$ 64). After applying CSP, the
embedding vector, transformed from EEG signals, has 104 features $\times$ 16
time segments, where the features consist of 13 classes $\times$ 8 CSP
features.

#### Generator

The main architecture of the proposed generator consists of gated recurrent
unit (GRU) (Cho et al. 2014) to capture the sequence information and several
residual blocks to capture the temporal and spatial information with
preventing vanishing gradient issue. Figure 2a describe the generator in
detail. The input of the generator is given as the embedding vector of EEG
signals and the output is generated as mel-spectrogram. The embedding vector
goes through pre-convolution layer consisting 1d convolution and concatenates
the features from bi-directional GRU extract the sequence features. To match
the output size of the mel-spectrogram, 1d convoltion layer was applied. After
that, the generator upsamples it using transposed convolution with stride of
two or three, and multi-receptive field fusion (MRF) module, sum of outputs of
multiple residual blocks with different kernel size, follows.

#### Discriminator

The discriminator is similarly composed in the opposite direction to the
generator, described in Figure 2b. The input of the discriminator is the mel-
spectrogram and the output is the validity of real/fake voice. Moreoever, the
discriminator was trained with their class only using mel-spectrogram from
voice. The input goes through pre-convolution layer consisting 1d convolution.
And then, the upsampling layer using transposed convolution and MRF module are
conducted. After that, the bi-directional GRU extracts the sequence features,
and the validity is estimated with the classifier.

#### Vocoder and ASR

Vocoder and ASR model are used to clarify the reconstructed voice from the
brain signal by translating to text. To adjust our framework for a real-time
BTS system, we applied a pretrained HiFi-GAN (Kong, Kim, and Bae 2020) which
is a high-quality vocoder with fast inference speed. The same architecture and
hyperparameters were applied with the pretrained model ‘Universal ver.1’,
trained with Universal dataset.

The ASR is composed of pretrained HuBERT (Hsu et al. 2021) with a large
configuration, which is a self-supervised learning model of speech
representations trained with Libri-Light dataset and fine-tuned with the
LibriSpeech dataset.

### Training Loss Term

This section describes losses for training, including reconstruction loss, GAN
loss, and CTC loss. The generator uses reconstruction loss $L_{rec}$,
adversarial loss $L_{adv}$, and CTC loss $L_{ctc}$, while the discriminator
uses adversarial loss $L_{adv}$.

$L(G)=\lambda_{g1}L_{rec}(G)+\lambda_{g2}L_{adv}(D;G)+\lambda_{g3}L_{ctc}(G)$
(1) $L(D)=\lambda_{d}L_{adv}(D;G)$ (2)

, where loss coefficients are referred to $\lambda_{g1-3}$ for the generator
and $\lambda_{d}$ for the discriminator.

#### Reconstruction loss

To reinforce the guideline for reconstructing the mel-spectrogram of target,
reconstruction loss was applied. Reconstruction loss have been verified in
many studies (Kong, Kim, and Bae 2020; Isola et al. 2017), which can help
improve the efficiency of generator and the fidelity of reconstructed data.
Since imagined speech has no reference speech to compare the reconstructed
performance, spoken speech audio collected at the same sequence of imagined
speech was used as the target audio to compute reconstruction loss. DTW was
applied to match the alignment of EEG during spoken/imagined speech and the
target spoken voice.

$L_{rec}(G)=E_{s}[(G(s)-x)^{2}]$ (3)

, where $s$ refers to the input of the generator such as an embedding vector
from EEG signals, and $x$ refers to the input of the discriminator such as a
mel-spectrogram.

#### GAN loss

To reconstruct the mel-spectrogram to follow the real one, adversarial GAN
loss $L_{adv}$ was conducted on the generator $G$ and discriminator $D$ as
follows.

$L_{adv}(D;G)=E_{(x,s)}[log(1-D(x))+log(D(G(s)))]$ (4)
$L_{adv}(G;D)=E_{s}[log(1-D(G(s)))]$ (5)

, where $x$ refers the input of discriminator such as mel-spectrogram and $s$
refers the input of generator such as embedding vector from EEG signals.

#### CTC loss

CTC loss is a common metric of the performance for automatic speech
recognition systems (Graves et al. 2006). CTC loss $L_{ctc}$ allows to train
the model using sequential data without the alignment information. CTC loss
was primarily given to guide the prediction of character and phonemes, to
enhance the performance of unseen classes.

### Domain Adaptation

The DA strategy was employed to resolve the fundamental constraint of speech
reconstruction from imagined speech. Since imagined speech does not accompany
the movement of the articulators, it is relatively reliable in terms of
movement artifacts accompanied by the mouth movement and the vibration.
However, since the ground truth audio for imagined speech does not exist, we
designed an adaptation framework that adapts the domain of imagined speech
from spoken speech, in order to exploit the natural correspondence of imagined
EEG and the voice of spoken speech. The DA process was performed in two steps;
1) sharing the covariance matrix between imagined EEG and spoken EEG by
applying the CSP filter of imagined speech and 2) applying transfer learning
for the generator and discriminator from the trained model of spoken EEG.

#### Sharing subspace

The CSP weights, trained with a training set (60%) of imagined EEG, were
shared to generate embedding vectors. Sharing the CSP filters computed from
imagined EEG allows the latent space of spoken EEG to be shifted into a
comparable feature space of imagined EEG. Unlike most transfer learning
approaches that involve applying the weak domain to the well-trained
classifier, we elected to the contrary, bringing the spoken speech feature
space to that of the imagined speech. In that case, we could achieve a clear
pattern more from the brain signal of speech rather than the movement
artifacts or vibration artifacts.

#### Transfer learning

The model was trained with a training set of spoken EEG, and then fine-tuned
with a training set of imagined EEG at a smaller learning rate than the case
of spoken EEG. This was to connect with the voice recordings of spoken speech,
which acts as the ground truth of imagined speech. The trained model from
spoken EEG can assist training the models of imagined EEG that has
insufficient information, therefore, the spoken EEG could guide learning from
the weak features of imagined EEG.

## Experimental Setup

### Dataset

#### Participants

Six participants volunteered in the study. The study was conducted in
accordance with the Declaration of Helsinki, approved by Korea University
Institutional Review Board [KUIRB-2019-0143-01]. Informed consent was obtained
from all subjects.

#### Paradigms

For the spoken speech session, participants were instructed to naturally
pronounce the randomly given thirteen phases, provided as an auditory cue of
twelve words/phrases (ambulance, clock, hello, help me, light, pain, stop,
thank you, toilet, TV, water, and yes) and a silent phase. Speech data were
recorded in a rhythmic manner to avoid any visual or auditory disruptions. The
imagined speech data was collected in the exactly same manner as the spoken
speech, following the previous study (Lee, Lee, and Lee 2020). 100 trials of
both spoken speech and imagined speech per class were collected for each
participant. Therefore, each participant had 1300 trials for the spoken and
imagined speech paradigm.

#### Recording

The dataset used in this study consists of scalp EEG recordings of
spoken/imagined speech and voice recordings of spoken speech. During the
experiment, EEG signals were recorded in the sampling rate of 2500Hz via Brain
Vision/Recorder (BrainProduct GmbH, Germany), and the corresponding audio of
spoken speech was simultaneously recorded in the sampling rate of 8000Hz.
Brain signals were recorded with 64-channel EEG cap with active Ag/AgCl
electrode placement following the international 10-10 system.

### Pre-processing

EEG signals were extracted in 2 second intervals for each trial. The data was
filtered with a 5th order Butterworth bandpass filter in the high-frequency
range of 30–120 Hz which is well-known to contain speech-related information
(Lachaux et al. 2012; Lee, Lee, and Lee 2020). Notch filter was used to remove
the line noise at 60 Hz with harmonics of 120 Hz. The electroocoulography
(EOG) and electromyography(EMG) of spoken speech were removed using blind
source separation referencing from EOG and EMG (Gómez-Herrero et al. 2006).
Baseline was corrected by subtracting the average value of 500 ms before each
trial. Pre-processing procedures were performed in Python and Matlab using
OpenBMI Toolbox (Lee et al. 2019), BBCI Toolbox (Krepki et al. 2007), and
EEGLAB (Delorme and Makeig 2004). For the voice data, we resampled the voice
signals to 22050 Hz, and reduced the noise using noisereduce library(Sainburg
2019; Sainburg, Thielk, and Gentner 2020).

### Dataset Composition and Training Procedure

By definition, imagined speech does not have reference voice to train the
model. However, spoken speech accompanies vocal output, therefore, both audio
and the EEG data of each spoken speech utterance was collected in perfectly
time-aligned pair. Since the experimental design of imagined speech and spoken
speech was completely identical, the voice recording of the identical sequence
of spoken speech for each subject was used as the reference voice for the
imagined speech evaluation. Also, due to the lack of the reference voice for
the imagined speech brain signals, transfer learning was applied with the
model trained on spoken speech EEG and spoken speech audio to imagined speech
EEG.

The dataset was divided in 5-fold into training, validation, and test dataset
according to the random selection with random seed. One unseen word, ‘stop’
was separated from the dataset, and wasn’t included in the training set. It
was chosen to test unseen case, since every phoneme composing the word ‘stop’
was covered with the remaining 11 words used for the training. That is, we
trained 11 words/phrases and a silent phase as training dataset, and validated
12 words/phrases and a silent phase in validation and test dataset including
the unseen word.

Generator and discriminator were first trained using spoken EEG data with
ground truth of voice for each trial. As the imagined speech does not have
each trial of voice, the voice during spoken speech was used as ground truth.
To match the time points between EEG and voice of spoken speech, DTW was
applied to the synthesized mel-spectrogram of EEG with mel-spectrogram of
voice. Moreover, the generator and discriminator executed transfer learning
from the trained model of spoken EEG to connect naturally between imagined EEG
and the voice during spoken speech.

Figure 3: Mel-spectrogram and the audio wave of original voice, reconstructed
voice from EEG. Three examples of reconstruction include ‘Hello’, ‘Water’, and
‘Help me’. Silent phases for both spoken and imagined EEG were successfully
decoded. Unseen cases were also reconstructed despite their inferior
performance.

### Model Implementation Details

The generator had three residual block with the kernel size of 3, 7, and 11,
each dilation of 1, 3, and 5, and upsampling rate of 3, 2, and 2 with twice
upsample kernel size. The number of initial channel was 1024, the directional
GRU dimension was the half of initial channel. The discriminator had the same
residual block as generator, but downsampling rate of 3, 3, and 3 with twice
kernel size. The number of final channel was 64, the directional GRU dimension
was the half of final channel. The mel-spectrogram was managed in sampling
rate of 22050 Hz and the STFT and mel function was conducted with nFFT of
1024, window of 1024, hop size of 256, and 80 bands of mel-spectrograms.
Initial training was conducted with an initial learning rate of $10^{-4}$, and
the fine-tuning was conducted a lower learning rate such as $10^{-5}$ in
maximum epoch of 500 and a batch size of 10. We trained the model on a NVIDIA
GeForce RTX 3090 GPU. We used AdamW optimizer (Loshchilov and Hutter 2017)
with searched parameters of $\beta_{1}$=0.8, $\beta_{2}$=0.99, and weight
decay $\lambda$=0.01, which was scheduled by 0.999 factor in every epoch. We
released the source code and sample data on Github at:
https://github.com/youngeun1209/NeuroTalk

### Evaluation Metrics

For the evaluation metrics, we used root mean square error (RMSE), character
error rate (CER), and a subjective mean opinion score (MOS) test. To evaluate
the accurate reconstructing performace of the generator, we computed the RMSE
between the target and reconstructed mel-spectrogram. To evaluate the clarity
quantitatively, we conducted CER after going through the ASR model. For the
subjective evaluation, MOS test was conducted to evaluate the quality of the
reconstructed speech. We randomly selected 125 samples of voice from a test
dataset. The samples were evaluated by more than 20 raters on a scale of 1-5
with 0.5 point increments. We compared EEG dataset with GT and converted GT in
form of mel-spectrogram, waveform, and character. Moreover, to demonstrate the
extension of NeuroTalk, we evaluated the generation performance of unseen word
which is composed of phonemes that were contained in the trained word classes.

## Results and Discussion

### Voice Reconstruction from EEG

The audio samples are included in the demo page at:
https://neurotalk.github.io/demo/neurotalk.html. Figure 3 displays the mel-
spectrogram and the audio wave of original voice, reconstructed voice from
spoken speech EEG, and reconstructed voice from imagined speech EEG. As shown
in the figure, successfully reconstructed cases display similar patterns of
mel-spectrogram and the audio waveform. Table 1 shows the evaluation results
of reconstructed voice from brain signal compared with GT. Objective measures
of RMSE and CER have shown inferior performance on the case of imagined speech
EEG compared to that of the spoken speech EEG. MOS of spoken speech cases was
high with no large difference from GT, which means the model can generate
natural speech from spoken EEG. Interestingly, the unseen spoken EEG shows
higher MOS than imagined EEG though objective evaluation of RMSE and CER
indicates inferior, which implies that reconstructing spoken EEG that
simultaneously produce voice enables to generate natural voice.

As shown in Figure 3, test samples for the silent phases were successfully
reconstructed with no activation. Silent cases for both spoken and imagined
EEG was successfully decoded except for only one case of imagined speech.
According to this result, we can infer that our NeuroTalk model accurately
learned the silence interval and can detect the precise onset from both spoken
speech and imagined speech EEG. Although imagined speech doesn’t have ground
truth voice, the results show that the proposed NeuroTalk framework
effectively adapts the spoken speech based model to the imagined speech EEG to
decode user’s intention from brain signals and generate voice.

There were some instances of failure in the imagined speech case (Figure 4).
The significant difference between the success and failure cases was whether
it detects silence intervals. As shown in the Figure 4, a failure case with
CER of 50% displays few silence interval between ’thank’ and ’you’. Moreover,
the failure case with CER of 100% produces only a few words that cannot
represent any characters from ground truth.

Figure 4: Success and failure cases. Mel-spectrogram and waveform were
displayed for original voice and reconstructed voices.

### Ablation Study

The results of ablation study are demonstrated in Table 2. We performed an
ablation study of GRU in generator and discriminator to clarify to add the
module in the model, and losses of GAN, reconstruction, and classification to
verify the effect of each loss on the performance of generator. In the
objective evaluation, as the performance without reconstruction loss is the
worst and then CTC loss, reconstruction loss followed by CTC loss has the
greatest impact on the framework. Furthermore, the results of all cases were
much worse than the baseline, that indicates that all cases are performing
their roles, in particular, reconstruction loss have the largest impact on
training. In the subjective evaluation, naturalness shows different tendency
to a certain extent, and the results without GRU show the worst with inferior
naturalness, which shows that sequential features are important for natural
speech synthesis.

### Voice Reconstruction of Unseen Words

The unseen and untrained word could be reconstructed from EEG, with much lower
CER than chance level, and produce quite high quality of audio with MOS over
3. The gap between the CER of spoken and imagined EEG was relatively small in
the unseen case, compared to the trained words. These results imply that while
the robust performance of spoken EEG with trained words may be affected by
movement artifacts to be overfitted to the trained classes, imagined speech
may be relatively effective in capturing phonemes from the trained words to
generate unseen word.

Although it still could be further improved, our result demonstrates that the
NeuroTalk model has the potential to extend the degree of freedom of decodable
words or sentences by training on several word level dataset. We expect that
CTC loss could learn the character or phoneme information of words even from
brain signals, which contains human intention. Since we trained the model with
limited words/phrases, it may be simply classifying the EEG as one of the
training classes. However, the model have shown the potential to generate
unseen word outside of the training set indicates the possibility of the model
to be generalized and expanded to the classes outside of the training set.

Table 1: Results of subjective and quantitative tests Model | RMSE | CER (%) | MOS
---|---|---|---
GT | - | 18.35 ($\pm$11.45) | 3.67 ($\pm$0.97)
GT $\rightarrow$ Mel ($\rightarrow$ ASR) | - | 23.35 ($\pm$10.85) | 3.68 ($\pm$0.88)
Spoken EEG | 0.166 ($\pm$0.022) | 40.21 ($\pm$13.49) | 3.34 ($\pm$0.95)
Imagined EEG | 0.175 ($\pm$0.029) | 68.26 ($\pm$2.47) | 2.78 ($\pm$1.11)
Unseen Spoken EEG | 0.185 ($\pm$0.029) | 78.89 ($\pm$7.43) | 2.87 ($\pm$1.12)
Unseen Imagined EEG | 0.187 ($\pm$0.026) | 83.06 ($\pm$14.54) | 2.57 ($\pm$1.18)
Table 2: Results of ablation study Input | RMSE | CER (%) | MOS
---|---|---|---
Baseline | 0.175 ($\pm$0.029) | 68.26 ($\pm$2.47) | 2.78 ($\pm$1.11)
w/o GRU | 0.185 ($\pm$0.027) | 76.05 ($\pm$3.25) | 2.18 ($\pm$1.24)
w/o GAN loss | 0.180 ($\pm$0.022) | 76.05 ($\pm$2.33) | 2.86 ($\pm$1.21)
w/o reconstruction loss | 0.620 ($\pm$0.121) | 80.16 ($\pm$7.98) | 2.50 ($\pm$1.25)
w/o CTC loss | 0.387 ($\pm$0.069) | 76.90 ($\pm$0.25) | 2.52 ($\pm$1.21)
w/o DA | 0.175 ($\pm$0.025) | 72.30 ($\pm$1.71) | 2.66 ($\pm$1.24)

### Domain Adaptation

DA was performed by sharing the CSP subspaces and transferring the spoken
speech-based trained model to imagined speech EEG. As shown in Table 2, the
result with DA has shown superior performance compared to the baseline. This
implies that spoken speech EEG was useful to train imagined speech EEG, which
means the neural substrates of imagined and spoken speech has common features
that can be represented in our embedding vector. Speech production and
articulation is mainly known to be associated with the inferior frontal gyrus,
so called Broca’s area. Angular gyrus functions to associate various language-
related activation from auditory, motor, sensory, and also visual cortex,
therefore, not only the left temporal lobe but the whole brain may function in
the speech process (Watanabe et al. 2020). Our embedding vector which was
generated from the whole channel EEG, may contain both articulatory
information and the speech intention. Therefore, we demonstrates the potential
of generating speech by extracting informative speech-related features, which
refer to the similarity of spoken speech EEG and imagined speech EEG.

### Leave-one-out Scenario

In order to apply our NeuroTalk system to locked-in patients who can only use
imagined speech, we conducted an additional experiment of leave-one-out (LOO)
approach to apply to an entirely new data from an unseen person. The model was
trained with the spoken EEG of entire subject excluding one subject and was
fine-tuned with the imagined EEG of the excluded subject. As a result,
comparable performance was obtained, inferior to the baseline but better than
without DA. Based on the LOO approach, we have found potential to expand our
framework to entirely new person, which could further help people who have
lost their own voice.

## Conclusion

We presented NeuroTalk, which reconstructs user’s own voice from EEG during
imagined speech. DA approach was conducted by sharing feature embedding and
training the models of imagined speech EEG, using the trained models of spoken
speech EEG. Our results demonstrate the feasibility of reconstructing voice
from non-invasive brain signals of imagined speech in word-level. Furthermore,
unseen word can be generated with several characters although the performance
was not high, which means we can expand our study to a larger dataset and to
sentence-level speech synthesis in the future. We hope our study can
contribute to expanding the means of human communication and further benefit
patients or disabled people to gain freedom for their communication. We look
forward to a world where we can communicate without saying anything.

## Acknowledgement

This work was supported by Institute for Information & Communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government
(MSIT) (No.2021-0-02068, Artificial Intelligence Innovation Hub; No.
2017-0-00451, Development of BCI based Brain and Cognitive Computing
Technology for Recognizing User’s Intentions using Deep Learning; No.
2019-0-00079, Artificial Intelligence Graduate School Program(Korea
University)).

## References

  * Akbari et al. (2019) Akbari, H.; Khalighinejad, B.; Herrero, J. L.; Mehta, A. D.; and Mesgarani, N. 2019\.  Towards reconstructing intelligible speech from the human auditory cortex.  _Scientific Reports_ , 9(1): 1–12.
  * Angrick et al. (2019) Angrick, M.; Herff, C.; Mugler, E.; Tate, M. C.; Slutzky, M. W.; Krusienski, D. J.; and Schultz, T. 2019.  Speech synthesis from ECoG using densely connected 3D convolutional neural networks.  _Journal of Neural Engineering_ , 16(3): 036019.
  * Angrick et al. (2022) Angrick, M.; Ottenhoff, M.; Diener, L.; Ivucic, D.; Ivucic, G.; Goulis, S.; Colon, A. J.; Wagner, L.; Krusienski, D. J.; Kubben, P. L.; et al. 2022.  Towards closed-Loop speech synthesis from stereotactic EEG: A unit delection approach.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1296–1300. IEEE.
  * Angrick et al. (2021a) Angrick, M.; Ottenhoff, M.; Goulis, S.; Colon, A. J.; Wagner, L.; Krusienski, D. J.; Kubben, P. L.; Schultz, T.; and Herff, C. 2021a.  Speech synthesis from stereotactic EEG using an electrode shaft dependent multi-input convolutional neural network approach.  In _2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, 6045–6048. IEEE.
  * Angrick et al. (2021b) Angrick, M.; Ottenhoff, M. C.; Diener, L.; Ivucic, D.; Ivucic, G.; Goulis, S.; Saal, J.; Colon, A. J.; Wagner, L.; Krusienski, D. J.; et al. 2021b.  Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity.  _Communications Biology_ , 4(1): 1–10.
  * Anumanchipalli, Chartier, and Chang (2019) Anumanchipalli, G. K.; Chartier, J.; and Chang, E. F. 2019.  Speech synthesis from neural decoding of spoken sentences.  _Nature_ , 568(7753): 493–498.
  * Baevski et al. (2020) Baevski, A.; Zhou, Y.; Mohamed, A.; and Auli, M. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33: 12449–12460.
  * Chaudhary, Birbaumer, and Ramos-Murguialday (2016) Chaudhary, U.; Birbaumer, N.; and Ramos-Murguialday, A. 2016.  Brain–computer interfaces for communication and rehabilitation.  _Nature Reviews Neurology_ , 12(9): 513–525.
  * Cho et al. (2014) Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Cooney, Folli, and Coyle (2018) Cooney, C.; Folli, R.; and Coyle, D. 2018.  Neurolinguistics research advancing development of a direct-speech brain-computer interface.  _IScience_ , 8: 103–125.
  * Delorme and Makeig (2004) Delorme, A.; and Makeig, S. 2004.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.  _Journal of Neuroscience Methods_ , 134(1): 9–21.
  * Devlaminck et al. (2011) Devlaminck, D.; Wyns, B.; Grosse-Wentrup, M.; Otte, G.; and Santens, P. 2011.  Multisubject learning for common spatial patterns in motor-imagery BCI.  _Computational Intelligence and Neuroscience_ , 2011.
  * Gaddy and Klein (2020) Gaddy, D.; and Klein, D. 2020.  Digital voicing of silent speech.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 5521–5530.
  * Gaddy and Klein (2021) Gaddy, D.; and Klein, D. 2021.  An improved model for voicing silent speech.  _arXiv preprint arXiv:2106.01933_.
  * Goldstein et al. (2022) Goldstein, A.; Zada, Z.; Buchnik, E.; Schain, M.; Price, A.; Aubrey, B.; Nastase, S. A.; Feder, A.; Emanuel, D.; Cohen, A.; et al. 2022.  Shared computational principles for language processing in humans and deep language models.  _Nature neuroscience_ , 25(3): 369–380.
  * Gómez-Herrero et al. (2006) Gómez-Herrero, G.; De Clercq, W.; Anwar, H.; Kara, O.; Egiazarian, K.; Van Huffel, S.; and Van Paesschen, W. 2006.  Automatic removal of ocular artifacts in the EEG without an EOG reference channel.  In _Proceedings of the 7th Nordic signal processing symposium-NORSIG 2006_ , 130–133. IEEE.
  * Gonzalez et al. (2017) Gonzalez, J. A.; Cheah, L. A.; Gomez, A. M.; Green, P. D.; Gilbert, J. M.; Ell, S. R.; Moore, R. K.; and Holdsworth, E. 2017.  Direct speech reconstruction from articulatory sensor data by machine learning.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 25(12): 2362–2374.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27.
  * Graimann, Allison, and Pfurtscheller (2009) Graimann, B.; Allison, B.; and Pfurtscheller, G. 2009.  Brain–computer interfaces: A gentle introduction.  In _Brain-Computer Interfaces_ , 1–27. Springer.
  * Graves et al. (2006) Graves, A.; Fernández, S.; Gomez, F.; and Schmidhuber, J. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In _Proceedings of the 23rd international conference on Machine learning_ , 369–376.
  * Herff et al. (2019) Herff, C.; Diener, L.; Angrick, M.; Mugler, E.; Tate, M. C.; Goldrick, M. A.; Krusienski, D. J.; Slutzky, M. W.; and Schultz, T. 2019.  Generating natural, intelligible speech from brain activity in motor, premotor, and inferior frontal cortices.  _Frontiers in Neuroscience_ , 13: 1267.
  * Herff, Krusienski, and Kubben (2020) Herff, C.; Krusienski, D. J.; and Kubben, P. 2020.  The potential of stereotactic-EEG for brain-computer interfaces: current progress and future directions.  _Frontiers in Neuroscience_ , 14: 123.
  * Hsu et al. (2021) Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021.  HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29: 3451–3460.
  * Isola et al. (2017) Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1125–1134.
  * Kong, Kim, and Bae (2020) Kong, J.; Kim, J.; and Bae, J. 2020.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33: 17022–17033.
  * Krepki et al. (2007) Krepki, R.; Blankertz, B.; Curio, G.; and Müller, K.-R. 2007.  The Berlin Brain-Computer Interface (BBCI)–towards a new communication channel for online control in gaming applications.  _Multimedia Tools and Applications_ , 33(1): 73–90.
  * Krishna et al. (2020) Krishna, G.; Tran, C.; Han, Y.; Carnahan, M.; and Tewfik, A. H. 2020.  Speech synthesis using EEG.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1235–1238. IEEE.
  * Lachaux et al. (2012) Lachaux, J.-P.; Axmacher, N.; Mormann, F.; Halgren, E.; and Crone, N. E. 2012.  High-frequency neural activity and human cognition: past, present and possible future of intracranial EEG research.  _Progress in Neurobiology_ , 98(3): 279–301.
  * Lee et al. (2019) Lee, M.-H.; Kwon, O.-Y.; Kim, Y.-J.; Kim, H.-K.; Lee, Y.-E.; Williamson, J.; Fazli, S.; and Lee, S.-W. 2019.  EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy.  _GigaScience_ , 8(5): giz002.
  * Lee, Lee, and Lee (2019) Lee, S.-H.; Lee, M.; and Lee, S.-W. 2019.  EEG representations of spatial and temporal features in imagined speech and overt speech.  In _Asian Conference on Pattern Recognition_ , 387–400.
  * Lee, Lee, and Lee (2020) Lee, S.-H.; Lee, M.; and Lee, S.-W. 2020.  Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication.  _IEEE Transactions on Neural Systems and Rehabilitation Engineering_.
  * Lee, Lee, and Lee (2022) Lee, S.-H.; Lee, Y.-E.; and Lee, S.-W. 2022.  Toward imagined speech based smart communication system: potential applications on metaverse conditions.  In _2022 10th International Winter Conference on Brain-Computer Interface (BCI)_ , 1–4. IEEE.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Makin, Moses, and Chang (2020) Makin, J. G.; Moses, D. A.; and Chang, E. F. 2020.  Machine translation of cortical activity to text with an encoder–decoder framework.  _Nature neuroscience_ , 23(4): 575–582.
  * Nguyen, Karavas, and Artemiadis (2017) Nguyen, C. H.; Karavas, G. K.; and Artemiadis, P. 2017.  Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features.  _Journal of neural engineering_ , 15(1): 016002.
  * Proix et al. (2022) Proix, T.; Delgado Saa, J.; Christen, A.; Martin, S.; Pasley, B. N.; Knight, R. T.; Tian, X.; Poeppel, D.; Doyle, W. K.; Devinsky, O.; et al. 2022.  Imagined speech can be decoded from low-and cross-frequency intracranial EEG features.  _Nature Communications_ , 13(1): 1–14.
  * Saha, Abdul-Mageed, and Fels (2019) Saha, P.; Abdul-Mageed, M.; and Fels, S. 2019.  SPEAK YOUR MIND! Towards imagined speech recognition with hierarchical deep learning.  _Proc. Interspeech 2019_ , 141–145.
  * Sainburg (2019) Sainburg, T. 2019.  timsainb/noisereduce: v1.0.
  * Sainburg, Thielk, and Gentner (2020) Sainburg, T.; Thielk, M.; and Gentner, T. Q. 2020.  Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires.  _PLoS Computational Biology_ , 16(10): e1008228.
  * Schultz et al. (2017) Schultz, T.; Wand, M.; Hueber, T.; Krusienski, D. J.; Herff, C.; and Brumberg, J. S. 2017.  Biosignal-based spoken communication: A survey.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 25(12): 2257–2271.
  * Si et al. (2021) Si, X.; Li, S.; Xiang, S.; Yu, J.; and Ming, D. 2021.  Imagined speech increases the hemodynamic response and functional connectivity of the dorsal motor cortex.  _Journal of Neural Engineering_ , 18(5): 056048.
  * Wang et al. (2013) Wang, L.; Zhang, X.; Zhong, X.; and Zhang, Y. 2013.  Analysis and classification of speech imagery EEG for BCI.  _Biomedical signal processing and control_ , 8(6): 901–908.
  * Wang and Ji (2021) Wang, Z.; and Ji, H. 2021.  Open vocabulary Electroencephalography-To-Text decoding and zero-shot sentiment classification.  _arXiv preprint arXiv:2112.02690_.
  * Watanabe et al. (2020) Watanabe, H.; Tanaka, H.; Sakti, S.; and Nakamura, S. 2020.  Synchronization between overt speech envelope and EEG oscillations during imagined speech.  _Neuroscience research_ , 153: 48–55.