Spiking-Fer: Spiking Neural Network for Facial Expression Recognition
  With Event Cameras
Facial Expression Recognition (FER) is an active research domain that has
shown great progress recently, notably thanks to the use of large deep learning
models. However, such approaches are particularly energy intensive, which makes
their deployment difficult for edge devices. To address this issue, Spiking
Neural Networks (SNNs) coupled with event cameras are a promising alternative,
capable of processing sparse and asynchronous events with lower energy
consumption. In this paper, we establish the first use of event cameras for
FER, named "Event-based FER", and propose the first related benchmarks by
converting popular video FER datasets to event streams. To deal with this new
task, we propose "Spiking-FER", a deep convolutional SNN model, and compare it
against a similar Artificial Neural Network (ANN). Experiments show that the
proposed approach achieves comparable performance to the ANN architecture,
while consuming less energy by orders of magnitude (up to 65.39x). In addition,
an experimental study of various event-based data augmentation techniques is
performed to provide insights into the efficient transformations specific to
event-based FER.
Sami Barchid, Benjamin Allaert, Amel Aissaoui, José Mennesson, Chaabane Djéraba
# Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Cameras

###### Abstract.

Facial Expression Recognition (FER) is an active research domain that has shown great progress recently, notably thanks to the use of large deep learning models. However, such approaches are particularly energy intensive, which makes their deployment difficult for edge devices. To address this issue, Spiking Neural Networks (SNNs) coupled with event cameras are a promising alternative, capable of processing sparse and asynchronous events with lower energy consumption. In this paper, we establish the first use of event cameras for FER, named "Event-based FER", and propose the first related benchmarks by converting popular video FER datasets
to event streams. To deal with this new task, we propose "Spiking-FER", a deep convolutional SNN model, and compare it against a similar Artificial Neural Network (ANN). Experiments show that the proposed approach achieves comparable performance to the ANN architecture, while consuming less energy by orders of magnitude (up to 65.39x). In addition, an experimental study of various event-based data augmentation techniques is performed to provide insights into the efficient transformations specific to event-based FER.

## 1. Introduction

Facial Expression Recognition (FER) has received particular interest in recent years given its diverse and practical applications in computer vision, e.g., security, health, and communication. So far, efficient learning-based models have been proposed for FER (Krizhevsky et al., 2014). However, these methods often overcome the energy consumption constraint (Krizhevsky et al., 2014). The convergence between the need to reinforce the complexity of learning models and the requirement of a physical platform at the cutting edge of technology induces heavy consequences in terms of energy consumption and has repercussions on the environment of the planet.

Spiking Neural Networks (SNNs) have become popular due to practical interest in terms of addressable complexity and energy efficiency than artificial neural networks (ANN). The neurons in SNNs asynchronously transmit information through sparse and binary spikes, enabling event-driven computing. Unlike ANN requiring dense matrix operations, energy consumption occurs only when a spike is generated on a neuromorphic chip. Hence, SNNs can significantly improve the energy efficiency of artificial intelligence systems.

Although SNNs are less energy-consuming, they do not match the performance of ANNs in computer vision. Inspired by the advances in ANNs, many recent methods are proposed to improve the performance of SNNs, e.g., ANN to SNN conversion (Shi et al., 2017), innovative architectures (Shi et al., 2017), new features encoding methods (Beng et al., 2017), and data augmentation (Shi et al., 2017)). However, very few studies have focused on FER, mainly due to the lack of training data.

In this paper, we propose an innovative framework for FER using SNNs, as illustrated in Fig. 1. First, a conversion framework using the V2E converter (Vaswani et al., 2017) is proposed in order to preprocess the well-known FER video datasets and generate event-based streams which are a suitable format for SNN architecture. Then, the Spiking-FER model is trained using Surrogate Gradient Learning (Srivastava et al., 2014), which enables the applicability of the backpropagation algorithm for deep SNN architectures. Finally, several experiments are conducted in order to evaluate the proposed model and compare it to a conventional ANN in terms of recognition accuracy and energy consumption.

Our proposal brings three novelties:

* **Event-based FER Benchmark**: we provide a reproducible protocol to generate event-based FER benchmarks from the most popular video FER datasets which is, to the best of our knowledge, the first event-based benchmark proposed for FER.
* **New SNN Model Architecture**: we propose a new end-to-end deep convolutional SNN method, called "Spiking-FER" that encodes the event streams into spiking features that are then used by the output accumulator in order to predict the facial expression.
* **Event Data Augmentation for FER**: we analyze the performance of Spiking-FER and its corresponding ANN architecture depending on popular Event Data Augmentations (EDAs) to investigate their impacts on event-based FER.

The paper is structured as follows. In Section 2, we review relevant recent methods for FER and on the evolution of the SNN for event-based vision. Section 3 presents our SNN model architecture and the training process on event-based FER data. Section 4 introduces the experimental setup including datasets and evaluation protocols. The experimental results are provided in Section 5. Finally, we discuss the results and future work in Section 6.

## 2. Related Works

### Facial Expression Recognition

FER methods could be classified into two categories: static (frame-based) and dynamic (sequence-based) methods. Frame-based methods extract spatial features from still images. They rely on hand-crafted features (Shi et al., 2017) or learned features (Shi et al., 2017) using mainly CNNs (Shi et al., 2017), but recently transformer architecture has also come into play (Krizhevsky et al., 2014). Sequence-based methods are performed in different ways: either by aggregating frames, e.g., onset-apex-offset (Shi et al., 2017) or onset-apex (Shi et al., 2017), or by using consecutive frames in order to encode the temporal information. These methods use mainly deep architectures such as 3D CNN (Shi et al., 2017), Recurrent Neural Networks (RNN) (Shi et al., 2017), and Transformers (Shi et al., 2017).

Recently, motion has come into play, and has proven to be effective in sequence-based FER (Shi et al., 2017). Moreover, it has been proposed to address different challenges for FER (occlusions (Shi et al., 2017), intensity (Shi et al., 2017)), taking advantage of the fact that inter-individual variation in motion is better suited for FER than appearance features (Beng et al., 2017). Performance improvement is achieved by increasing the complexity of learning approaches, especially by taking into account spatio-temporal encoding. However, this improvement comes usually with the expense of energy consumption.
### Event-based Vision paired with Spiking Neural Networks

Recently, learning algorithms adapted from backpropagation such as surrogate gradient learning (Srivastava et al., 2014) enable the training of deep SNN architecture by solving the non-differentiability issue of spiking neurons (Goodfellow et al., 2016). Such directly trained deep architectures are the first of several attempts capable of tackling event-based vision problems of similar complexity to those addressed by ANNs currently, such as object detection (Goodfellow et al., 2016), semantic segmentation (Goodfellow et al., 2016), and object tracking (Wang et al., 2017). Furthermore, these architectures start to adapt state-of-the-art techniques from ANNs (Vision Transformer (Vaswani et al., 2017), spatio-temporal attention (Srivastava et al., 2014),...) to operate with spiking neurons thanks to this new ability of gradient-based optimization. This recent direction of directly trained SNNs coupled with event cameras demonstrates impressive efficiency by being able to outperform ANNs (Goodfellow et al., 2016) and showing reduced energy consumption by orders of magnitude (Goodfellow et al., 2016).

## 3. Methodology

### Problem Formulation

During a time interval \(\Delta_{\mathcal{T}}\), an event camera with a \(H\times W\) resolution produces a set of \(N\) asynchronous events \(\mathcal{E}=\{e_{i}\}_{i=1}^{N}\). Each event \(e_{i}\) of the sequence can be formulated as a tuple of 4 values: \(e_{i}=\{x_{i},y_{i},t_{i},p_{i}\}\), where \((x_{i},y_{i})\) correspond to the pixel coordinates, \(t_{i}\) is the timestamp, and \(p_{i}\in\{1,-1\}\) is the sign of the polarity change.

As the asynchronous nature of events is not appropriate for many computer vision approaches (Garfani et al., 2017), a popular event representation method (Garfani et al., 2017) is to discretize a stream of events \(\mathcal{E}\) into a sequence of \(T\) binary event frames \(\mathbf{X}_{T}\in\mathbb{B}^{T\times 2\times H\times W}=\{X_{t}\}_{t=1}^{T}\). In this work, it is done by accumulating events during \(T\) subsequent time intervals \(\frac{\Delta_{\mathcal{T}}}{T}\) to create the sequence of binary frames and thus the final spike tensor \(\mathbf{X}_{T}\).

Event-based FER can be defined as follows: given an event sequence \(\mathcal{E}\) obtained from capturing a subject that performs a facial expression, the objective is to recognize this expression as the appropriate label \(c\) among \(\mathcal{C}\) different classes. To do so, a model \(f_{\alpha}(\cdot)\) with a set of learnable parameters \(\alpha\) is trained such that: \(c=f_{\alpha}(\mathbf{X}_{T})\). The top of Fig. 2 illustrates the formulation of event-based FER with the related notations.

### Spiking-FER

Spiking-FER is represented by the model \(f_{\alpha}(\cdot)\), where \(\alpha\) denotes its synaptic weights. The bottom of Fig. 2 illustrates an overview of the proposed Spiking-FER architecture.

#### 3.2.1. Spiking Neuron Model

The proposed convolutional SNN architecture uses the Integrate-and-Fire (IF) neuron (Srivastava et al., 2014) as the spiking neuron model. It accumulates input spikes weighted by the synaptic weights into a'membrane potential'. When this membrane potential exceeds a certain threshold value, the neuron emits an output spike and is reset to zero. The discretized dynamics of a layer \(l\) of IF neurons from Spiking-FER at a certain time-step \(1\leq t\leq T\) is described as follows:

\[U_{t}^{l}=U_{t-1}^{l}+\mathcal{W}^{J}X_{t-1}^{l-1}-\theta X_{t}^{l} \tag{2}\] \[X_{t}^{l}=\Theta(U_{t}^{l}-\theta) \tag{1}\]

where \(U_{t}^{l}\) denotes the membrane potentials of the IF neurons, \(\mathcal{W}^{l}\) is the set of synaptic weights, \(X_{t}^{l}\in\mathbb{B}\) denotes the output spike tensor. \(X_{t}^{l}\) consists of 1's when the related element of \(U_{t}^{l}\) exceeds the threshold value \(\theta\), and 0's otherwise. For simplicity, the threshold is set to 1 for all layers (i.e., \(\theta=1\)). This mechanism, formulated in Eq. 2 is known as the Heaviside step function (\(\Theta(\cdot)\)).

#### 3.2.2. Direct Training via Surrogate Gradient

Spiking-FER is trained using Surrogate Gradient Learning (Goodfellow et al., 2016; Srivastava et al., 2014), a popular and effective training approach for deep SNN models. An SNN can be expressed as a Recurrent Neural Network where the membrane potentials are internal states. Consequently, the synaptic weights can be trained using Backpropagation Through Time (Srivastava et al., 2014). The main issue is related to the backward pass, where \(\Theta(\cdot)\) is not differentiable - i.e., its derivative is 0 almost everywhere, and +\(\infty\) at 0 - causing the gradient chain to break ("dead neuron problem" (Goodfellow et al., 2016)). Therefore, surrogate gradient learning solves this problem by employing the derivative of a continuous surrogate function \(\sigma(\cdot)\) on the backward pass as an approximation of the derivative of \(\Theta(\cdot)\). In Spiking-FER, we define \(\sigma(x)=\frac{1}{\pi}\arctan(\pi x)+\frac{1}{2}\).

#### 3.2.3. Model Architecture

Strongly related to (Goodfellow et al., 2016), Spiking-FER consists of two modules: **(1)** a deep convolutional SNN encoder that encodes the event streams into spiking features; and **(2)** an output accumulator module (Goodfellow et al., 2016) that predicts the emotion of the sample from the encoded spiking features.

The encoder is a SEW-ResNet-18 (Goodfellow et al., 2016) architecture that outputs spiking feature vectors \(F_{t}\in\mathbb{B}^{d}\), where \(d\) is the number of output channels (in SEW-ResNet-18, \(d=512\)). At each time-step, these extracted spiking features are fed into the output accumulator module responsible for making the final prediction.

As shown in the rightmost part of Fig. 2, the output accumulator module is composed of one fully connected layer of artificial neurons and one linear classifier. Firstly, it accumulates the spiking features from all time-steps to obtain a single feature vector \(\mathcal{F}\in\mathbb{R}^{d}\) such that:

\[\mathcal{F}=\sum_{t=1}^{T}\mathcal{W}\times F_{t} \tag{3}\]

, where \(\mathcal{W}\in\mathbb{R}^{d\times d}\) is the set of trainable weights in the fully connected layer. Then, the features \(\mathcal{F}\) are fed into the linear classifier to obtain the final classification prediction.

The whole network is trained end-to-end using the cross-entropy loss.

## 4. Experimental Setup

In this section, we present the experimental setup including datasets, evaluation protocols and the models configurations.

### Video-to-Events Conversion

To validate the applicability of event-based data and SNNs to FER applications, while being comparable to standard FER baselines (Kal
we convert some of the most popular video FER datasets: ADFES (Srivastava et al., 2017), CASIA (Srivastava et al., 2017), CK+ (Kang et al., 2017), and MMI (Mori et al., 2017) to an event-based format. Each video of a given FER dataset is processed by two successive steps. The first step is a standardization of all frames (Bahdan et al., 2016): the face of the represented subject is cropped and rotated based on 68 facial landmarks and converted to grayscale. Then, the resulting frame is resized to a resolution of \((200\times 200)\). The second step corresponds to the conversion of the standardized video into events using v2e (Srivastava et al., 2017), a video-to-event converter for realistic events simulation, as illustrated in Fig. 3. The code and parameters to reproduce the benchmark are available1.

Footnote 1: The code will be released upon acceptance

### Evaluation Protocol

Models that are evaluated on an event-based FER dataset follow a 10-fold cross-validation configuration: the given dataset is randomly split into 10 folds of equal size. The employed metric for every iteration is the top-1 classification accuracy. Finally, we report the mean accuracy score of the 10 folds.

### Implementation Details

The experiments are implemented in PyTorch, Tonic (Tonic, 2017) and Spikingfelly (Spikingfelly, 2017) as our SNN simulator, and run on one NVIDIA A40 GPU. We train our models (Spiking-FER and ResNet-18) during 500 epochs, using an SGD optimizer, with a learning rate of 0.01 and a cosine annealing scheduler (Toledo et al., 2017), and keep the best performance on the validation set. A low-latency regime is adopted for Spiking-FER, with \(T=6\).

### Comparison with ANN

Since the convolutional SNN encoder of Spiking-FER is a SEW-ResNet-18, we choose a ResNet-18 (He et al., 2018) model as the corresponding ANN. Similarly to the ANN model defined in (Bahdan et al., 2016), the spike tensor \(\mathbf{X}_{T}\) is fed into the 2D-CNN by concatenating all binary event frames together along the time axis.

## 5. Experiments

### Study on Event Data Augmentation

To investigate the impacts of popular event data augmentations (EDAs)(Kang et al., 2017), given in Table 1, on the model performance, the experiments are conducted in 2 successive parts: **(1)** an analysis of common EDAs; **(2)** an analysis on specific EDAs for either regularization of training with scarce datasets, or FER-specific transformation.

#### 5.1.1. Common EDAs

Since EDAs can be applied in combination, the main objective of this part is to assess which EDA has the best impact when they are combined with each other. Therefore, we run all possible combinations of common EDAs, which gives a total of 32 experiments for a given dataset, as illustrated in Fig. 4.

Figure 2. Overview of the proposed framework. _Top_) Formulation of Event-based Facial Expression Recognition. _Bottom_) The Spiking-FER architecture where the convolutional SNN encoder is expressed as a recurrent neural network.
The baseline results show that the SNN model performs better than the ANN model without augmentation - i.e., using only the original data from the event stream. Often observed in neural network training, data augmentation tends to significantly improve performance. This is especially true for FER, where databases are scarce. On ANNs, we observe that all the EDAs combinations have a positive or null impact, unlike the SNNs, where some EDAs combinations tend to decrease the performances. Among the EDA methods, the combination {_Crop_, _H Flip_ and _Noise_} significantly improves the performance on both ANNs and SNNs, except for the MMI dataset, where the improvement is less significant. This can be explained by the greater complexity of the data, where greater head pose variations and variety of facial movement patterns appear.

Then, we evaluate the accuracy scores of all folds for all experiments, which gives 320 scores. We perform a multivariate regression analysis on this population of 320 scores by considering the applied EDAs as categorical independent variables. For a given EDA, the regression analysis gives an approximation of the expected benefit in performance. Fig. 5 shows the results of the regression analysis for each dataset. According to the regression coefficients, \(Crop\) and \(HFlip\) have generally a positive impact, which suggest that they are well adapted for event-based FER. These methods cover well the small variations, e.g., face translation or image resolution changes, on the data observed in the different databases that compose the benchmark. However, _Reverse_ that reports either non-significant results or negative impacts in all cases. This can be explained by the fact that the activation of a facial expression follows a temporal sequence induced by the facial muscles. In this case, the reversal of the event flow is not consistent, especially since in this benchmark, where the sequences only go from the neutral state to the apex. _PolFlip_ highlights the differences between Spiking-FER and the ANN: while Spiking-FER constantly reports negative effects, the ANN model obtains a positive impact. This suggests that SNNs do not benefit from _PolFlip_ for event-based FER.

#### 5.1.2. Specific EDAs

We keep the best-performing combinations of common EDAs and evaluate the specific ones. For a given dataset, the best combination of common EDAs is defined as the highest mean accuracy score obtained on the 10-fold cross-validation. Fig. 6 reports the results obtained with and without these specific EDAs, adapted to the event flows for FER. Considering the performances, we note that the combination of _EventDrop_, which regularizes the training of neural networks on limited datasets, and _Mirror_, which transforms the visual aspect of a subject's face, is perfectly adapted to augment facial expressions for both ANNs and SNNs. In addition, to improve the performance of the models, the performance gap between the ANNs and SNNs models is significantly reduced, especially for the ADFES dataset. Both EDAs have been designed to adapt to inter-individual variation, e.g., face symmetry and expression activation time.

### Estimation of Energy Consumption

Similarly to (Beng et al., 2017; Wang et al., 2018), we compare the energy efficiency of Spiking-FER and a similar ANN when simulated on a 45nm CMOS chip (Wang et al., 2018).

The estimation methodology is described as follows: firstly, we quantify the spiking rate of each layer, as spiking neurons consume energy only when generating a spike. The spiking rate of a given layer \(l\) is calculated as follows:

\[Rs(l)=\frac{\text{\# spikes of $l$ over all time-steps}}{\text{\# neurons of $l$}} \tag{4}\]

Secondly, we compute the total floating-point operations (FLOPs) of a layer of spiking neurons (\(FLOPs_{\text{\small SNN}}\)) by using the FLOPs of the same layer in a non-spiking neural network (\(FLOPs_{\text{\small ANN}}\)) and the spike rate of the spiking neuron layer:

\begin{table}
\begin{tabular}{l l} \hline \hline
**EDA** & **Description** \\ \hline \(Crop\) & Spatial crop of the whole sequence with a random scale \\ \(HFlip\) & Horizontal flip of the whole sequence \\ \(Noise\) (\(BA\)) & Noisy events due to corrupted pixels in event cameras (Garfani et al., 2016). \\ \(PolFlip\) & Flip of polarity (i.e., \(p_{i}=-p_{i}\) for all events) \\ \(Reverse\) & Reverse the orders of events. \\
**EventDrop**(Wang et al., 2018) & Randomly drops events spatially, temporally or globally \\
**Mirror** & Mirrors the left or right half of the sequence \\ \hline \hline \end{tabular}
\end{table}
Table 1. Summary of EDAs. Common EDAs and Specific EDAs are respectively in italic and in bold.

Figure 3. Illustration of the proposed event-based FER benchmark. The video sequences are converted into events corresponding to the output of event cameras.
\[FLOPS_{SSNN}(I) =FLOPS_{SANN}(I)\times Rs(I) \tag{5}\] \[FLOPS_{ANN}(I) =\begin{cases}k^{2}\times O^{2}\times C_{in}\times C_{out}&\text{if $l$ is Conv.}\\ C_{in}\times C_{out}&\text{if $l$ is Linear.}\end{cases} \tag{6}\]

In Equation 6, \(k\) represents the kernel size, \(O\) represents the size of output feature maps, \(C_{in}\) represents the number of input channels, and \(C_{out}\) represents the number of output channels.

Finally, the total energy consumption of a model can be estimated on CMOS technology (Kang et al., 2017) by using the total FLOPs across all layers. Table 2 presents the energy cost of relevant operations in a 45nm CMOS process. MAC operation in ANNs requires one addition (32bit FP ADD) and one FP multiplication (32bit FP MULT) (Shen et al., 2016), whereas SNNs require only one FP addition per MAC operation due to binary spike processing. The total energy consumption of ANNs and SNNs are represented by \(E_{ANN}\) and \(E_{SSNN}\), respectively.

\[E_{ANN} =\sum_{I}FLOPS_{SANN}(I)\times E_{MAC} \tag{7}\] \[E_{SSNN} =\sum_{I}FLOPS_{SSNN}(I)\times E_{AC} \tag{8}\]

Table 3 reports the mean inference energy estimation for each dataset. Similarly to previous works (Chen et al., 2017), Spiking-FER shows better energy efficiency by orders of magnitude (from 47.42\(\times\) to 65.39\(\times\) more efficient), which proves the applicability of SNNs for low-power FER application on edge devices.

## 6. Conclusion

In this work, we introduced _event-based benchmarks for Facial Expression Recognition_ (FER) and proposed a new SNN architecture named _Spiking-FER_. We applied traditional augmentation techniques adapted to event streams, along with two specific techniques - _EventDrop_(Kang et al., 2017) and _Mirror_ - that led to significant improvements in our model's performance. Our proposed approach achieved similar performance to a traditional Artificial Neural Network (ANN) while consuming much less energy (up to 65.39\(\times\)). Our future work will extend this study to other applications such as gesture or action analysis.

\begin{table}
\begin{tabular}{|c|c|c|} \multicolumn{2}{c|}{**Operation**} & \multicolumn{1}{c|}{**Energy (p)**} \\ \hline
32bit FP MULT (\(E_{MULT}\)) & 3.7 \\
32bit FP ADD (\(E_{ADD}\)) & 0.9 \\
32bit FP MAC (\(E_{MAC}\)) & 4.6 (\(=E_{MULT}+E_{ADD}\)) \\
32bit FP AC (\(E_{AC}\)) & 0.9 \\ \end{tabular}
\end{table}
Table 2. Energy table for a 45nm CMOS process (from (Kang et al., 2017)).

Figure 4. Acc. obtained according to combinations of common EDA; (A) H Flip; (B) Noise; (C) Reverse; (D) Pol Flip; (E) Crop.

Figure 5. Significance regression coefficients (p-value \(<0.05\)) calculated on the 320 scores, corresponding to each common EDA for the different datasets (higher is better).

\begin{table}
\begin{tabular}{|c|