# FingerFlex: Inferring Finger Trajectories from ECoG signals

Vladislav Lomtev,1 Alexander Kovalev, 2 Alexey Timchenko 34

###### Abstract

Motor brain-computer interface (BCI) development relies critically on neural
time series decoding algorithms. Recent advances in deep learning
architectures allow for automatic feature selection to approximate higher-
order dependencies in data. This article presents the FingerFlex model - a
convolutional encoder-decoder architecture adapted for finger movement
regression on electrocorticographic (ECoG) brain data. State-of-the-art
performance was achieved on a publicly available BCI competition IV dataset 4
with a correlation coefficient between true and predicted trajectories up to
0.74. The presented method provides the opportunity for developing fully-
functional high-precision cortical motor brain-computer interfaces.

††Preprint. Under review

## Introduction

### Brain-computer interface

One of the promising rehabilitation tools for people with disabilities is the
brain-computer interface (BCI) (Lebedev and Nicolelis 2006; Bamdad, Zarshenas,
and Auais 2015; Yuan and He 2014). In general, developing a functioning BCI
involves interpreting neural signals to extract relevant information from the
brain. The extracted features, i.e. control parameters can be used in the
restoration of the lost functions and ability augmentation via controlling
external devices. Specifically, motor BCIs aim to decode movement-related
information in the sensory-motor cortex into meaningful behavioral patterns,
such as joint movement or finger flexion (Tam et al. 2019; Volkova et al.
2019).

### Recording of the neural activity

Brain activity is analyzed using electrophysiological methods measuring single
neuron or population level electric potentials. Most methods allowing to
capture spiking activity, such as optogenetics, single-unit and multi-unit
recordings are highly-invasive. At the same time, population level recording
tools, such as electroencephalogram (EEG) and magnetoencephalogram (MEG)
require no surgical intervention, albeit lack spacial resolution due to a
voltage leakage in the human scalp and movement-related artifacts (Neil Cuffin
and Cohen 1979; Hämäläinen et al. 1993; Jackson and Bolger 2014). The
electrocorticogram (ECoG) used in this study lies in between these methods,
estimated to register localized activity of $10^{5}$ neurons on the cortical
surface (Engel et al. 2005). ECoG electrodes are placed over the surface of
the cortex with approximately $1$ cm inter-electrode distance.
Electrocorticogram provides higher spatial resolution and superior signal
quality compared to EEG and MEG with a tradeoff of portability and
invasiveness.

### Movement correlates in the human brain

The primary brain area involved in the movement processing is primary motor
cortex (PMC) (Kakei, Hoffman, and Strick 1999). The adjacent brain areas -
supplementary motor area (SMA) and somatosensory cortex take part into motor
planning and receive peripheral inputs on the body position and environment
(Tam et al. 2019). Planning and executing motor programs change the activity
of single neurons in the cortical sheet of the motor areas (Kakei, Hoffman,
and Strick 1999). Spiking activity in a synchronized neuronal population, in
turn, contributes to the change in local field potentials registered via
surface electrodes (Buzsáki, Anastassiou, and Koch 2012). Despite the lack of
an exact area-to-muscle correspondence in the motor cortex, overlapping
regions associated with different muscle groups and even fingers can be
distinguished (Schieber 2001; Marjaninejad, Taherian, and Valero-Cuevas 2017;
Georgopoulos and Carpenter 2015).

Another feature of population activity in the cortex is that its local field
potential comprises a set of independent oscillatory processes (Buzsáki and
Draguhn 2004). It was shown that changes in alpha band (8-13 Hz) power, also
known as mu-rhythm, beta (13-35 Hz), gamma (35-100 Hz) and high-gamma (100-250
Hz) correlate significantly with ongoing movement patterns (Tam et al. 2019;
Marjaninejad, Taherian, and Valero-Cuevas 2017). It is thus possible to use
frequency-decomposed or time-frequency represented signal instead of raw time
series data.

The primary purpose of this study is to improve existing solutions for
decoding a continuous finger trajectory based solely on ECoG-registered brain
oscillatory patterns in the sensorimotor cortex.

### Related work

While some studies concentrated on classifying finger activations (Hotson et
al. 2016; Onaran, Ince, and Cetin 2011; Yao and Shoaran 2019), the others
predicted hand translations (Pistohl et al. 2008; Nakanishi et al. 2013; Bundy
et al. 2016; Śliwowski et al. 2022) and gestures (Pan et al. 2018; Branco et
al. 2017), numerous approaches were presented for solving finger trajectory
prediction task.

Liang and Bougrain (2012) and Flamary (2012) presented the first attempts to
decode one-dimensional finger flexions on the publicly available BCI
Competition dataset IV (Schalk et al. 2007). The former study employed
switching linear models, while the latter - linear regression with rigorous
feature selection, which allowed to win the competition. Nakanishi et al.
(2014) used sparsed linear regression on frequency band decomposed signal
achieving comparable results both on the BCI competition dataset and data
gathered by their team. With the advancement of deep learning (LeCun, Bengio,
and Hinton 2015) techniques over the following years, neural network
approaches were more frequently used in consequent studies. For example, Xie,
Schwartz, and Prasad (2018) used long short-term memory based (Hochreiter and
Schmidhuber 1997) architecture to decode finger trajectory. Jubien et al.
(2019) compared different conventional machine learning methods with
artificial neural network approaches on the same task to conclude that deep
learning solves the task with higher accuracy. The method developed by Frey et
al. (2021) presented the opportunity for developing a generalized decoder for
neural data yielding high performance over a variety of tasks including finger
trajectory regression over the ECoG data. The attempt to build an
interpretable convolutional neural network was made by Petrosyan et al.
(2021), which is highly relevant for neuroscience research on details of motor
encoding in the human brain. Lastly, a new type of features based on Riemann
geometry (Congedo, Barachant, and Bhatia 2017) was introduced in the finger
movement decoding task by Yao, Zhu, and Shoaran (2022) used consequently with
linear discriminators and support vector machine (SVM) regressors.

The novelty of this work is primarily addressed by the use of different
decoding paradigm based on the encoder-decoder architecture, which transforms
feature time series into a target time series through a set of convolutions,
deconvolutions, and skip-connections. Our research aims to prove this approach
can outperform existing conventional machine learning solutions based on
thorough feature extraction, as well as deep learning models implemented by
the other research teams in a finger flexion decoding task.

## Methods

### Dataset

The primary dataset for training the model was the BCI competition IV dataset
4 (Schalk et al. 2007). The dataset contains electrocorticographic recordings
of three anonymous patients with corresponding finger movement information,
represented as a one-dimensional coordinate denoting the finger fold. The
recording grids consist of 48-64 electrodes evenly distributed in grids of
6x8, 8x8, and other shapes. The authors of a dataset imposed a limitation on
ECoG electrode distribution information. The channels were shuffled beforehand
and the electrode positions relative to the brain were not provided either.
ECoG was recorded at a sampling frequency of 1000 Hertz, while finger
movements were sampled at 25 Hz with a data glove.

The subjects were instructed to move a particular finger after a cue, which
was presented as a corresponding word (e.g., ”thumb”) on a computer monitor.
Each cue lasted for two seconds and was followed by a two-second resting
period during which the screen was blank. More details on the experimental
procedure and recording setup can be found in the dataset description (Miller
and Schalk 2008).

### Data preprocessing and feature extraction

Data preprocessing can be divided into two parts: preprocessing of ECoG
signals and preprocessing of the finger movement data (Figure 1). The ECoG
preprocessing pipeline consisted of the following steps. Firstly, data
standardization and subtraction of the median from each channel was performed.
Then, band-pass and band-stop filters were used to filter frequencies in the
range of 40 to 300 Hz and remove the power grid frequency of 50 Hz and its
harmonics. The frequency range from 40 to 300 Hz constitutes gamma and high-
gamma components of the local field potential signal, which were chosen as the
main predictors for the task.

Figure 1: ECoG and finger motion data processing pipeline

As a method of explicit extraction of time-frequency representation of the
signal, the convolution with complex Morlet wavelets was chosen. Wavelet
decomposition allows capturing specific frequency components of a signal
change in time. Forty wavelet kernels were created with base frequencies
ranging from 40 to 300 Hz evenly spaced on a logarithmic scale. The
convolution operation of selected wavelets with each channel was performed and
the amplitudes of resulting analytical signals were extracted as an absolute
value of a complex-valued vector. As a result of this operation for each ECoG
channel, a spectrogram is obtained of shape: (number of wavelets, number of
time points in a time window). Then, the sampling rate was then lowered from
1000 Hz to 100 Hz by decimation for the correspondence to a motion capture
time series. Next, the obtained time-frequency envelopes were standardized
with respect to $0.1$ and $0.9$ quantiles, and the values not falling within
this range were then taken equal to the corresponding scaler boundary values
to combat outliers. This operation was done using the RobustScaler class from
the scikit-learn library, which was fitted on the training set and applied to
both the training and validation sets. Lastly, to account for the delay
between brain activity and actual movements, the forward shift of ECoG data
with respect to the finger movement time series was introduced. The exact
value of the shift can be regarded as a tunable hyperparameter within the
relevant physiological range: $0-200$ms.

The preprocessing of finger movement data consisted of two steps. First, to
keep the sampling frequencies of finger movements and time-frequency time
series matched, upsampling from $25$ Hz to $100$Hz using bicubic interpolation
was conducted. Second, MinMax scaling was applied.

### The model architecture

The final model is designed in the same way as the convolutional autoencoder
with a set of additions and improvements. The architecture of the model is
depicted on Figure 2

Figure 2: FingerFlex model architecture

As the input, the model takes a window of ECoG features of arbitrary length
and returns the corresponding window of predicted movements; however, during
the training process, the length of such a window is fixed and equal to $256$
time points ($2.56$ secs). Initially, the model performs dimensionality
reduction using a regular convolutional layer. Next comes a set of
convolutional encoder layers. Each encoder layer consists of a 1D convolution
layer, layer normalization, GeLU activation function, dropout layer, and
maximum pooling with stride equals $2$. This part of the model is responsible
for encoding the features of the time window fragments. Encoder layer
architecture is inspired by wav2vec architecture (Schneider et al. 2019).

This is followed by a symmetrical set of convolutional decoder layers, each of
which uses for further prediction not only information from the previous
layer, but also from the symmetrical encoder layer. In other words, skip
connections like those of the UNet model (Ronneberger, Fischer, and Brox 2015)
were added. Skip connections make it possible to use for prediction not only
the generalized representation in the latent space obtained by the encoder,
but also the original local features, which ultimately increases the accuracy
of the final model. The model is completed by a $1\times 1$ convolution layer
that uses the resulting set of features from each point in time to predict the
coordinates of each of the five fingers at that point.

### Training details

The BCI competition IV dataset 4 was already divided into training and
validation sets, the recording of each of the three patients was divided into
$2$ parts, where the first part of $6.5$ minutes duration constitutes training
subset and the second part of $3.5$ minutes long constitutes validation data
subset. The model was trained and optimized using only the training subset,
while the validation set was used for the performance measurement and
hyperparameter adjustment. Due to lack of alignment of electrode grid shapes
between subjects and intentional mixing of electrodes done by the dataset
creators, the model was trained for each subject individually. Note that
overfitting due to a hyperparameter search using the validation set was
addressed by performing ablation study using only one subject provided in the
dataset. The same model was trained on the other patients without any
additional hyperparameter adjustment.

##### Loss function

The loss function was chosen as the half sum of the mean squared error (MSE)
and the mean cosine distance. Since the metric of the final score is
correlation, it is essential to include it in the loss for training process.
However, the correlation itself is not an optimal optimization criterion when
working with small windows, since it is too sensitive to insignificant noise
within this window in those cases when there is no movenet. Therefore, instead
of a correlation loss, it was decided to use the cosine distance. The ablation
study was conducted to test whether the combination of MSE and cosine
similarity metrics performed better than any of those metrics alone.
Nevertheless, due to the evaluation criteria at BCI competition IV,
correlation coefficient (CC) metric was used to evaluate and compare model
performance.

##### Hyperparameters

The number of features on each of the layers of the encoder as a result of the
experiments was taken equal to $(32,32,64,64,128,128)$. The number of features
in the decoder layers is symmetrical with respect to the bottleneck. The
stride size was taken equal to $2$ to reduce the sampling rate of the original
window as gradually as possible, which allows better extraction of common
features.

The learning rate was picked according to the Pytorch Lightning library’s
automatic learning rate finder recommendation and was set fixed to $8.4\cdot
10^{-5}$.

As was previously mentioned, wavelet number, loss function combination
coefficients and the delay between ECoG and hand movement time series can be
considered as the tunable hyperparameters.

## Results

### Model development

Every subsequently improved model was trained for $30$ epochs. Model weights
trained during the epoch preceding overfitting of the model were chosen for
inference. The model refinement took several important steps to achieve the
final performance.

#### Architecture refinement

Baseline encoder-decoder architecture, trained on unscaled absolute values of
wavelet-convoluted signals, yielded average correlation value of $0.2$ on the
first subject (Table 1). However, adding data scaling both for ECoG features
and finger movements improved results using the same architecture by more than
two-fold, to $0.45$ on average on the first subject. It can be speculated that
this win in performance is due to highly variable absolute values of the time-
frequency representation of the signal. The electrophysiological recordings
consistently show a $1/f$ relationship between signal spectral power and a
corresponding frequency (Buzsáki and Draguhn 2004), leading to a multiple-fold
difference between signal power in highly distinct frequency bands. A
modification of the encoder-decoder architecture parameters, which are mainly
a number of features in each layer and the number of layers allowed to
increase the average correlation coefficient further to $0.55$. The final
model architecture is shown in Figure 2. Then the use of wavelets only from
the gamma frequency band and an increase in their number up to $40$ also led
to an improvement in the results to an average correlation value of $0.6$.
Finally, skip connections were added, which increased the mean correlation
coefficient to $0.66$.

Table 1:  Model development steps with corresponding performance obtained on the validation set. Model improvement step | Mean correlation
---|---
Baseline model | 0.2
\+ Feature scaling | 0.45
\+ Architecture improvement | 0.55
\+ Frequency range adjustment | 0.59
\+ Optimizing number of wavelets | 0.6
\+ Adding skip-connections | 0.66

#### Hyperparameter tuning

Then an ablation study was performed to calculate the optimal value of the
delay hyperparameter between the ECoG signal and the predicted motion. The
baseline delay chosen as the initial one was $20$ ms, and as the consequent
experiments showed that for the built model the value of delay is not crucial,
as long as it is within $0-200$ms range. Therefore, the final shift value
between ECoG and finger motion time series was kept at $20$ ms.

Another ablation study was conducted to test whether the use of a combined
loss function - the sum of cosine distance and MSE is empirically justified.
Cosine distance, when used alone yielded a correlation value of $0.44$, which
is a worse result than that of MSE ($0.64)$. However, using a combination of
cosine distance and MSE increases the correlation by an average of $0.02$
compared to the regular MSE, summing up to 0.66.

### Decoding performance

Finger movement inference, carried out on three subjects presented in the BCI
Competition IV dataset 4 showed the following results: $0.66$ average
correlation on the first subject, $0.62$ on the second and $0.74$ on the third
(see Table 3). Recall that each patient had its own model trained, however,
hyperparameter tuning was carried out only using the first subject. The mean
correlation coefficient across all subjects is $0.67$. Correlation values for
different fingers are in the range of $0.6-0.75$ for the first subject,
$0.53-0.73$ for the second and $0.68-0.78$ for the third.

Table 2:  Performance comparison on works solving BCI Competition IV dataset 4 finger trajectory decoding task Mean CC by subject |
---|---
Method | S1 | S2 | S3 | Average
Flamary (2012) - BCI Competition 2nd place | 0.48 | 0.24 | 0.56 | 0.43
Liang and Bougrain (2012) - BCI Competition winner | 0.45 | 0.39 | 0.59 | 0.48
Xie et al. (2018) - LSTM | 0.56 | 0.41 | 0.58 | 0.52
Frey et al. (2021) - Multi-purpose CNN | N/A | N/A | N/A | 0.52
Petrosyan et al. (2021) - Interpretable CNN | 0.45 | 0.34 | 0.56 | 0.45
Yao et al. (2022) - lightGBM | 0.52 | 0.47 | 0.61 | 0.53
Our solution (FingerFlex) | 0.66 | 0.62 | 0.74 | 0.67
Table 3:  FingerFlex model performance on the BCI competition IV dataset 4.
Correlation coefficients are calculated separately for each subject and finger
using the provided validation set.  Correlation coefficient
---
Subject |  Thumb |  Index |  Middle |  Ring |  Little |  Avg.
S1 |  0.66 |  0.75 |  0.64 |  0.66 |  0.6 |  0.66
S2 |  0.66 |  0.73 |  0.53 |  0.63 |  0.54 |  0.62
S3 |  0.78 |  0.68 |  0.68 |  0.78 |  0.76 |  0.74
Average |  0.7 |  0.72 |  0.62 |  0.69 |  0.63 |  0.67

### Model verification on a different dataset

To check model stability across datasets, the built model was also tested on
Stanford dataset (Miller 2019). Since the data representation in this dataset
was the same as in BCI Competition IV dataset 4, the entire data processing
pipeline (Figure 1) remained unchanged.

Table 4:  FingerFlex model performance (correlation coefficients) on the ECoG
finger motion dataset provided by Miller (2019).  Correlation coefficient
---
Subject |  Thumb |  Index |  Middle |  Ring |  Little |  Avg.
bs |  0.65 |  0.66 |  0.45 |  0.53 |  0.44 |  0.55
cc |  0.76 |  0.7 |  0.61 |  0.76 |  0.73 |  0.71
zt |  0.59 |  0.73 |  0.68 |  0.62 |  0.53 |  0.63
jp |  0.68 |  0.67 |  0.31 |  0.65 |  0.53 |  0.57
ht |  0.31 |  0.32 |  0.34 |  0.36 |  0.36 |  0.34
mv |  0.42 |  0.85 |  0.42 |  0.40 |  0.71 |  0.56
wc |  0.45 |  0.63 |  0.42 |  0.27 |  0.31 |  0.42
wm |  0.33 |  0.08 |  0.25 |  0.32 |  0.06 |  0.22
jc |  0.65 |  0.37 |  0.32 |  0.43 |  0.37 |  0.43

The feature processing, hyperparameters and model architecture were fixed. The
obtained results are represented in the Table 4. While there are subjects,
which the model fails to recognize finger movements with correlations of
$0.2-0.4$, there are patients, who have mean correlation coefficients between
predicted and actual finger movements in the range of $0.5-0.7$. The low
performance on some patients could be attributed to irrelevant positioning of
the ECoG grid with regards to brain areas mostly involved during movement
planning and execution.

## Discussion

### Contribution

Figure 3: An example of true and decoded finger trajectories time series. The
data segment for visualization was taken from the validation set

Our approach shows great promise in developing high-precision decoders of
neural signals related to movements for use in the brain-computer interface.
The proposed model significantly outperforms the winner of the BCI competition
IV with a mean correlation value of approximately $0.67$. It also improves
greatly the current state-of-the-art results of other research groups solving
the finger trajectory decoding task (Table 2). Moreover, FingerFlex
successfully decoded finger trajectories using the other dataset without any
additional hyperparameter search (Table 4)

The method developed in this study stands on several important ideas and
improvements. First of all, the time-frequency feature selection choice was
crucial to capture fine changes in different frequency components of the brain
signals responsible for finger movements. Secondly, the hybrid encoder-decoder
architecture with skip connections was employed for the first time for the
ECoG decoding task. In addition, the introduction of a new loss function to a
conventional MSE further improved regression results. Lastly, the
physiologically relevant hyperparameters, such as time delay between finger
trajectory and ECoG time series and frequency range for wavelet convolution
alongside with conventional hyperparameters, such as learning rate and model
depth, are stable across participants and datasets. These hyperparameters can
be fixed once providing sufficient performance for the task.

There are also several advantages inherent in the suggested approach. Firstly,
our method does not require training separate decoders for resting and moving
states, as that was done in Elgharabawy and Wahed (2016) and Flamary (2012).
It captures both changes in finger coordinate during movement periods and
during resting periods without switching and adding any additional
information. Secondly, the model is relatively lightweight, having only about
$600000$ parameters, which potentially allows to use it in real-time inference
applications. Furthermore, the method does not rely on ECoG data, which
follows the movement in time, i.e. it makes predictions based only on a
current time window, not taking into account future and preceding ones. This
feature further facilitates real-time deployment. Lastly, the length of the
time window can be arbitrary due to the nature of convolution operation, thus
providing further flexibility for other decoding solutions.

### Limitations and potential improvements

The unequal electrode placement between subjects and uncertainty about
electrode positioning relative to the brain limit the possible generalization
approaches. However, the proposed method is highly suitable for individualized
training. One of the strongest arguments for that is that only 6.5 minutes of
recording is sufficient to achieve a significant performance on decoding fine
movements.

One of the further research directions is implementing transfer learning
across participants (Wan et al. 2021). If electrode positions and subjects’
magnetic-resonance imaging scans are available, it becomes possible to use
source reconstruction approaches (Hämäläinen et al. 1993; Chen, Yao, and
Hudson 2002), i.e. solving the ”inverse problem”. This allows making a
transition from individualized brain surface spatial patterns (electrode
space) to standardized cortical activations (source space). This is essential
for developing cross-subjects solutions and training a network on the larger
amounts of data gathered in different experimental settings and across
participants. Another possible way of solving the lack of correspondence of
electrode location between subjects is to use separate feature extractors for
each subject inferring generalized features suitable for transfer learning
like was done in the work of Peterson et al. (2021).

Besides transfer learning, a possible advancement could be adding
interpretable blocks to enhance understanding of the data. An important step
in that direction was made by Petrosyan et al. (2021) and Sturm et al. (2016),
and while adding interpretable blocks might worsen model performance, research
in this direction could provide useful insights on motor encoding in the human
brain and what features of ECoG signals really matter for the movement
prediction.

Finally, an important step would be to omit any manual feature extraction of
the time series data allowing the neural network to learn to extract relevant
features for the task itself. Time-frequency methods have their limitations
and their use imply increasing a number hyperparameters, which is not always
favorable. Temporal convolutions are by far the most relevant way to implement
automatic feature extraction and this approach also works well, as was shown
by Li et al. (2021b), Li et al. (2021a) and Peterson et al. (2021). The
possible downside of this approach would be the increased demand for lengthy
data due to the increased number of parameters in the network, and gathering
more data is often impossible in clinical settings of neurophysiological
research.

## Conclusion

The present study developed a novel deep learning approach for decoding finger
flexions based on oscillatory activity on the ECoG channels. With the use of
convolutional encoder-decoder architecture with skip-connections, the
performance on the trajectory regression task was significantly improved
compared to the existing solutions using the BCI competition IV dataset 4. The
proposed model can potentially be used for real-time inference of finger
flexions. This research shows promise for developing fast, functional, and
precise neuroprosthetic devices.

## References

  * Bamdad, Zarshenas, and Auais (2015) Bamdad, M.; Zarshenas, H.; and Auais, M. A. 2015.  Application of BCI Systems in Neurorehabilitation: A Scoping Review.  _Disability and Rehabilitation: Assistive Technology_ , 10(5): 355–364.
  * Branco et al. (2017) Branco, M. P.; Freudenburg, Z. V.; Aarnoutse, E. J.; Bleichner, M. G.; Vansteensel, M. J.; and Ramsey, N. F. 2017.  Decoding Hand Gestures from Primary Somatosensory Cortex Using High-Density ECoG.  _NeuroImage_ , 147: 130–142.
  * Bundy et al. (2016) Bundy, D. T.; Pahwa, M.; Szrama, N.; and Leuthardt, E. C. 2016.  Decoding Three-Dimensional Reaching Movements Using Electrocorticographic Signals in Humans.  _Journal of Neural Engineering_ , 13(2): 026021.
  * Buzsáki, Anastassiou, and Koch (2012) Buzsáki, G.; Anastassiou, C. A.; and Koch, C. 2012.  The Origin of Extracellular Fields and Currents — EEG, ECoG, LFP and Spikes.  _Nature Reviews Neuroscience_ , 13(6): 407–420.
  * Buzsáki and Draguhn (2004) Buzsáki, G.; and Draguhn, A. 2004.  Neuronal Oscillations in Cortical Networks.  _Science_ , 304(5679): 1926–1929.
  * Chen, Yao, and Hudson (2002) Chen, J.; Yao, K.; and Hudson, R. 2002.  Source Localization and Beamforming.  _IEEE Signal Processing Magazine_ , 19(2): 30–39.
  * Congedo, Barachant, and Bhatia (2017) Congedo, M.; Barachant, A.; and Bhatia, R. 2017.  Riemannian Geometry for EEG-based Brain-Computer Interfaces; a Primer and a Review.  _Brain-Computer Interfaces_ , 4(3): 155–174.
  * Elgharabawy and Wahed (2016) Elgharabawy, A.; and Wahed, M. A. 2016.  Decoding of Finger Movement Using Kinematic Model Classification and Regression Model Switching.  In _2016 8th Cairo International Biomedical Engineering Conference (CIBEC)_ , 84–89.
  * Engel et al. (2005) Engel, A. K.; Moll, C. K. E.; Fried, I.; and Ojemann, G. A. 2005.  Invasive Recordings from the Human Brain: Clinical Insights and Beyond.  _Nature Reviews Neuroscience_ , 6(1): 35–47.
  * Flamary (2012) Flamary, R. 2012.  Decoding Finger Movements from ECoG Signals Using Switching Linear Models.  _Frontiers in Neuroscience_ , 9.
  * Frey et al. (2021) Frey, M.; Tanni, S.; Perrodin, C.; O’Leary, A.; Nau, M.; Kelly, J.; Banino, A.; Bendor, D.; Lefort, J.; Doeller, C. F.; and Barry, C. 2021.  Interpreting Wide-Band Neural Activity Using Convolutional Neural Networks.  _eLife_ , 10: e66551.
  * Georgopoulos and Carpenter (2015) Georgopoulos, A. P.; and Carpenter, A. F. 2015.  Coding of Movements in the Motor Cortex.  _Current Opinion in Neurobiology_ , 33: 34–39.
  * Hämäläinen et al. (1993) Hämäläinen, M.; Hari, R.; Ilmoniemi, R. J.; Knuutila, J.; and Lounasmaa, O. V. 1993.  Magnetoencephalography—Theory, Instrumentation, and Applications to Noninvasive Studies of the Working Human Brain.  _Reviews of Modern Physics_ , 65(2): 413–497.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8): 1735–1780.
  * Hotson et al. (2016) Hotson, G.; McMullen, D. P.; Fifer, M. S.; Johannes, M. S.; Katyal, K. D.; Para, M. P.; Armiger, R.; Anderson, W. S.; Thakor, N. V.; Wester, B. A.; and Crone, N. E. 2016.  Individual Finger Control of a Modular Prosthetic Limb Using High-Density Electrocorticography in a Human Subject.  _Journal of Neural Engineering_ , 13(2): 026017.
  * Jackson and Bolger (2014) Jackson, A. F.; and Bolger, D. J. 2014.  The Neurophysiological Bases of EEG and EEG Measurement: A Review for the Rest of Us.  _Psychophysiology_ , 51(11): 1061–1071.
  * Jubien et al. (2019) Jubien, G.; Schaeffer, M.-C.; Bonnet, S.; and Aksenova, T. 2019.  Decoding of Finger Activation from ECoG Data: A Comparative Study.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , 1–8.
  * Kakei, Hoffman, and Strick (1999) Kakei, S.; Hoffman, D. S.; and Strick, P. L. 1999.  Muscle and Movement Representations in the Primary Motor Cortex.  _Science_ , 285(5436): 2136–2139.
  * Lebedev and Nicolelis (2006) Lebedev, M. A.; and Nicolelis, M. A. L. 2006.  Brain–Machine Interfaces: Past, Present and Future.  _Trends in Neurosciences_ , 29(9): 536–546.
  * LeCun, Bengio, and Hinton (2015) LeCun, Y.; Bengio, Y.; and Hinton, G. 2015.  Deep Learning.  _Nature_ , 521(7553): 436–444.
  * Li et al. (2021a) Li, F.; Chao, W.; Li, Y.; Fu, B.; Ji, Y.; Wu, H.; and Shi, G. 2021a.  Decoding Imagined Speech from EEG Signals Using Hybrid-Scale Spatial-Temporal Dilated Convolution Network.  _Journal of Neural Engineering_ , 18(4): 0460c4.
  * Li et al. (2021b) Li, Y.; Guo, L.; Liu, Y.; Liu, J.; and Meng, F. 2021b.  A Temporal-Spectral-Based Squeeze-and- Excitation Feature Fusion Network for Motor Imagery EEG Decoding.  _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , 29: 1534–1545.
  * Liang and Bougrain (2012) Liang, N.; and Bougrain, L. 2012.  Decoding Finger Flexion from Band-Specific ECoG Signals in Humans.  _Frontiers in Neuroscience_ , 6.
  * Marjaninejad, Taherian, and Valero-Cuevas (2017) Marjaninejad, A.; Taherian, B.; and Valero-Cuevas, F. J. 2017.  Finger Movements Are Mainly Represented by a Linear Transformation of Energy in Band-Specific ECoG Signals.  In _2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_ , 986–989.
  * Miller (2019) Miller, K. J. 2019.  A Library of Human Electrocorticographic Data and Analyses.  _Nature Human Behaviour_ , 3(11): 1225–1235.
  * Miller and Schalk (2008) Miller, K. J.; and Schalk, G. 2008.  Prediction of Finger Flexion 4th Brain-Computer Interface Data Competition.  3\.
  * Nakanishi et al. (2014) Nakanishi, Y.; Yanagisawa, T.; Shin, D.; Chen, C.; Kambara, H.; Yoshimura, N.; Fukuma, R.; Kishima, H.; Hirata, M.; and Koike, Y. 2014.  Decoding Fingertip Trajectory from Electrocorticographic Signals in Humans.  _Neuroscience Research_ , 85: 20–27.
  * Nakanishi et al. (2013) Nakanishi, Y.; Yanagisawa, T.; Shin, D.; Fukuma, R.; Chen, C.; Kambara, H.; Yoshimura, N.; Hirata, M.; Yoshimine, T.; and Koike, Y. 2013.  Prediction of Three-Dimensional Arm Trajectories Based on ECoG Signals Recorded from Human Sensorimotor Cortex.  _PLOS ONE_ , 8(8): e72085.
  * Neil Cuffin and Cohen (1979) Neil Cuffin, B.; and Cohen, D. 1979.  Comparison of the Magnetoencephalogram and Electroencephalogram.  _Electroencephalography and Clinical Neurophysiology_ , 47(2): 132–146.
  * Onaran, Ince, and Cetin (2011) Onaran, I.; Ince, N. F.; and Cetin, A. E. 2011.  Classification of Multichannel ECoG Related to Individual Finger Movements with Redundant Spatial Projections.  In _2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society_ , 5424–5427.
  * Pan et al. (2018) Pan, G.; Li, J.-J.; Qi, Y.; Yu, H.; Zhu, J.-M.; Zheng, X.-X.; Wang, Y.-M.; and Zhang, S.-M. 2018.  Rapid Decoding of Hand Gestures in Electrocorticography Using Recurrent Neural Networks.  _Frontiers in Neuroscience_ , 12.
  * Peterson et al. (2021) Peterson, S. M.; Steine-Hanson, Z.; Davis, N.; Rao, R. P. N.; and Brunton, B. W. 2021.  Generalized Neural Decoders for Transfer Learning across Participants and Recording Modalities.  _Journal of Neural Engineering_ , 18(2): 026014.
  * Petrosyan et al. (2021) Petrosyan, A.; Sinkin, M.; Lebedev, M.; and Ossadtchi, A. 2021.  Decoding and Interpreting Cortical Signals with a Compact Convolutional Neural Network.  _Journal of Neural Engineering_ , 18(2): 026019.
  * Pistohl et al. (2008) Pistohl, T.; Ball, T.; Schulze-Bonhage, A.; Aertsen, A.; and Mehring, C. 2008\.  Prediction of Arm Movement Trajectories from ECoG-recordings in Humans.  _Journal of Neuroscience Methods_ , 167(1): 105–114.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  arXiv:1505.04597.
  * Schalk et al. (2007) Schalk, G.; Kubanek, J.; Miller, K. J.; Anderson, N. R.; Leuthardt, E. C.; Ojemann, J. G.; Limbrick, D.; Moran, D.; Gerhardt, L. A.; and Wolpaw, J. R. 2007\.  Decoding Two-Dimensional Movement Trajectories Using Electrocorticographic Signals in Humans.  13\.
  * Schieber (2001) Schieber, M. H. 2001.  Constraints on Somatotopic Organization in the Primary Motor Cortex.  _Journal of Neurophysiology_ , 86(5): 2125–2143.
  * Schneider et al. (2019) Schneider, S.; Baevski, A.; Collobert, R.; and Auli, M. 2019.  Wav2vec: Unsupervised Pre-training for Speech Recognition.  arXiv:1904.05862.
  * Śliwowski et al. (2022) Śliwowski, M.; Martin, M.; Souloumiac, A.; Blanchart, P.; and Aksenova, T. 2022\.  Decoding ECoG Signal into 3D Hand Translation Using Deep Learning.  _Journal of Neural Engineering_ , 19(2): 026023.
  * Sturm et al. (2016) Sturm, I.; Lapuschkin, S.; Samek, W.; and Müller, K.-R. 2016.  Interpretable Deep Neural Networks for Single-Trial EEG Classification.  _Journal of Neuroscience Methods_ , 274: 141–145.
  * Tam et al. (2019) Tam, W.-k.; Wu, T.; Zhao, Q.; Keefer, E.; and Yang, Z. 2019.  Human Motor Decoding from Neural Signals: A Review.  _BMC Biomedical Engineering_ , 1(1): 22.
  * Volkova et al. (2019) Volkova, K.; Lebedev, M. A.; Kaplan, A.; and Ossadtchi, A. 2019.  Decoding Movement From Electrocorticographic Activity: A Review.  _Frontiers in Neuroinformatics_ , 13: 74.
  * Wan et al. (2021) Wan, Z.; Yang, R.; Huang, M.; Zeng, N.; and Liu, X. 2021.  A Review on Transfer Learning in EEG Signal Analysis.  _Neurocomputing_ , 421: 1–14.
  * Xie, Schwartz, and Prasad (2018) Xie, Z.; Schwartz, O.; and Prasad, A. 2018.  Decoding of Finger Trajectory from ECoG Using Deep Learning.  _Journal of Neural Engineering_ , 15(3): 036009.
  * Yao and Shoaran (2019) Yao, L.; and Shoaran, M. 2019.  Enhanced Classification of Individual Finger Movements with ECoG.  In _2019 53rd Asilomar Conference on Signals, Systems, and Computers_ , 2063–2066.
  * Yao, Zhu, and Shoaran (2022) Yao, L.; Zhu, B.; and Shoaran, M. 2022.  Fast and Accurate Decoding of Finger Movements from ECoG through Riemannian Features and Modern Machine Learning Techniques.  _Journal of Neural Engineering_ , 19(1): 016037.
  * Yuan and He (2014) Yuan, H.; and He, B. 2014.  Brain–Computer Interfaces Using Sensorimotor Rhythms: Current State and Future Perspectives.  _IEEE Transactions on Biomedical Engineering_ , 61(5): 1425–1435.

bibliography{refs.bib}