Evaluating Spiking Neural Network On Neuromorphic Platform For Human
  Activity Recognition
Energy efficiency and low latency are crucial requirements for designing
wearable AI-empowered human activity recognition systems, due to the hard
constraints of battery operations and closed-loop feedback. While neural
network models have been extensively compressed to match the stringent edge
requirements, spiking neural networks and event-based sensing are recently
emerging as promising solutions to further improve performance due to their
inherent energy efficiency and capacity to process spatiotemporal data in very
low latency. This work aims to evaluate the effectiveness of spiking neural
networks on neuromorphic processors in human activity recognition for wearable
applications. The case of workout recognition with wrist-worn wearable motion
sensors is used as a study. A multi-threshold delta modulation approach is
utilized for encoding the input sensor data into spike trains to move the
pipeline into the event-based approach. The spikes trains are then fed to a
spiking neural network with direct-event training, and the trained model is
deployed on the research neuromorphic platform from Intel, Loihi, to evaluate
energy and latency efficiency. Test results show that the spike-based workouts
recognition system can achieve a comparable accuracy (87.5\%) comparable to the
popular milliwatt RISC-V bases multi-core processor GAP8 with a traditional
neural network ( 88.1\%) while achieving two times better energy-delay product
(0.66 \si{\micro\joule\second} vs. 1.32 \si{\micro\joule\second}).
Sizhen Bian, Michele Magno
# Evaluating Spiking Neural Network On Neuromorphic Platform For Human Activity Recognition

###### Abstract.

Energy efficiency and low latency are crucial requirements for designing wearable AI-empowered human activity recognition systems, due to the hard constraints of battery operations and closed-loop feedback. While neural network models have been extensively compressed to match the stringent edge requirements, spiking neural networks and event-based sensing are recently emerging as promising solutions to further improve performance due to their inherent energy efficiency and capacity to process spatiotemporal data in very low latency. This work aims to evaluate the effectiveness of spiking neural networks on neuromorphic processors in human activity recognition for wearable applications. The case of workout recognition with wrist-worn wearable motion sensors is used as a study. A multi-threshold delta modulation approach is utilized for encoding the input sensor data into spike trains to move the pipeline into the event-based approach. The spikes trains are then fed to a spiking neural network with direct-event training, and the trained model is deployed on the research neuromorphic platform from Intel, Loihi, to evaluate energy and latency efficiency. Test results show that the spike-based workouts recognition system can achieve a comparable accuracy (87.5%) comparable to the popular milliwatt RISC-V bases multi-core processor GAP8 with a traditional neural network ( 88.1%) while achieving two times better energy-delay product (0.66 jJs vs. 1.32 jJs).

neuroomorphic computing, human activity recognition, spiking neural networks, workouts recognition, Loihi +
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: Computer Vision and Pattern Recognition

+
Footnote †: journal: journal: Computer Vision and Pattern Recognition
data like audio and sensor signals is relatively new and much less explored (Kyle et al., 2013). However, recent studies have shown promising results in using SNNs for ubiquitous computing with low-dimensional signal sensors, which can provide essential insights into related topics like HAR. Table 1 lists several recent studies that explore SNNs for ubiquitous computing with low dimensional signals and their resulting performance in different applications. Kyle et al. (Kyle et al., 2013) and Federico et al. (Fedico et al., 2014) explored the heartbeat classification with SNN with different training strategies and validated it on two neuromorphic processors with competitive accuracy. The energy-delay product (EDP) on Loihi shows over twenty-eight times more efficiency than the inference on a CPU. Enea et al. (Enea et al., 2015) run a direct-trained SNN on Loihi with customized EMG and DVS data set for hand gesture recognition. Similar to (Kyle et al., 2013), the EMG results on Loihi outperform in EDP compared with the results on GPU by ninety-seven times more efficiently. Besides the biological signals, audio signals were also explored with SNN (Kyle et al., 2013). In (Kyle et al., 2013), a fresh edge neuromorphic processor, Xylo, was used to classify ambient audios. An impressive inference energy was reported on Xylo with only 9.3uJ, over twenty-six times less energy than the edge IoT processor MAX78000 owning a convolutional hardware accelerator. One common result of those SNNs on low-dimensional signals is that SNN supplies state-of-the-art inference energy and impressive EDP compared with ANN on CPU and GPU. Besides this, the ANN-to-SNN training approach often results in competitive accuracy while the direct trained SNN shows incompetence in accuracy compared with the ANN result(Enea et al., 2015). The reason is that while encoding the signal to spikes, for example, the delta modulation, information loss is happening, especially for fast and huge signal variations (Kyle et al., 2013; Federico et al., 2014; Fedico et al., 2014).

In this work, we bring the following contributions:

1. We demonstrated the feasibility of using SNN for sensor-based HAR tasks pursuing latency and energy efficiency with a direct-trained SNN on the neuromorphic platform Loihi. The first Spiking-IMU dataset and the corresponding direct-trained SNN are released for benchmarking of HAR with the neuromorphic solution 1. Footnote 1: [https://github.com/zhaxidele/HAR-with-SNN](https://github.com/zhaxidele/HAR-with-SNN)
2. With spike trains generated by a multi-threshold delta modulation approach, a comparable accuracy (87.5%) is achieved compared with the ANN approach on the novel IoT processor GAP8 (88.1%), which has a dedicated RISC-V cluster for hardware acceleration and presented the state of the art edge AI performance in a rich of applications. Footnote 1: [https://github.com/zhaxidele/HAR-with-SNN](https://github.com/zhaxidele/HAR-with-SNN)
3. The latency and energy efficiency of the neuromorphic approach HAR and the mainstream approach HAR were compared in this case study, and it showed that the neuromorphic approach of HAR using SNNs on Loihi outperforms the ANN method in terms of inference energy on GAP8 while falling behind lightly in latency. However, the neuromorphic approach shows nearly two times the energy-delay product (0.66 uJ s vs. 1.31 uJ s).

## 3. System Architecture

Figure 1 depicts the pipeline of the proposed SNN for HAR applications, including three key steps: spike encoding from sensor data, off-line SNN training, and on line SNN inference on neuromorphic processor. To have a fair comparison with neural networks and low power digital processors, in this work we use a public data

\begin{table}
\begin{tabular}{p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt}} \hline Authors/Application Year & \begin{tabular}{c} \begin{tabular}{c} Sensor/Dataset \\ \end{tabular} & \begin{tabular}{c} Neuromor- \\ phic \\ platform \\ \end{tabular} & \begin{tabular}{c} Encoding \\ (vs \\ ANN) \\ \end{tabular} & \begin{tabular}{c} Training \\ Inference \\ (ms) \\ \end{tabular} & \begin{tabular}{c} Inference \\ energy \\ (m) \\ \end{tabular} & 
\begin{tabular}{c} Energy-Delay \\ Product \\ (\(\mu\)J s) \\ \end{tabular} \\ \hline
[15]-2019 & Heartbeat Classification & ECG MIT-BIH & DYNAP & Delta Modulation & SVM+ rSNN & 95.6\% (94.2\% ) & NA & NA \\ \hline
[32]-2021 & Oscillation Detection & EEG- iEEG- iEEG- iFOFOO: 1: 1: Time elapsed between the end of the input and the classification. 2: Only dynamic energy is considered in Loihi. 2: Not Available. 4: HFO was detected with morphology detector (Kyle et al., 2013).[ENDFOOTNOTE] & \begin{tabular}{c} EEG \\ Customized \\ \end{tabular} & \begin{tabular}{c} DYNAP- \\ SE \\ \end{tabular} & \begin{tabular}{c} Delta Modulation \\ \end{tabular} & \begin{tabular}{c} Direct \\ SNN \\ \end{tabular} & 
\begin{tabular}{c} 78.0\% \\ (67.0\%)d  & NA & NA \\ \end{tabular} & NA & NA \\ \hline
[14]-2020 & Hand Gesture Recognition & EMG Customized & Loihi & Delta Modulation & Direct SNN & 55.7\% (68.1\%) & 5.9 (3.8 on GPU) & 0.173 (25.5 on GPU) & 1.0 (97.3 on GPU) \\ \hline
[11]-2019 & Key Word Spotting & Audio Customized & Loihi & Rate Encoding & ANN-to- SNN & 97.9\% (97.9\%)e & 3.38 (97.9\%)e & 3.38 (1.30 on GPU, 2.4 GPU, 5.6 on a Jetson) & 0.27 (29.8 on Jetson) & 0.91 (38.7 on GPU, 13.44 on Jetson) \\ \hline
[12]-2023 & Ambient Audio Classification & Audio Xylo QUT- NOISE & 
\begin{tabular}{c} Power band \\ bin to spike \\ \end{tabular} & Direct SNN & 98.0\% (97.9\%) & 100\({}^{\dagger}\) & 0.0093 (0.25: MAX78000, 11.2: Cortex) & 0.93 \\ \hline
**Ours** & \begin{tabular}{c} **Human Activity Recognition** \\ \end{tabular} & 
\begin{tabular}{c} **1MU, Ca- \\ **1**-**1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** **1**-** 1**-** **1**-** **1**-** 1**-** **1**-** 1**-** **1**-** 1**-** 1**-** 1**-** 1**-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-** 1-1-** 1-** 1-1-** 1-1-** 1-** 1-** 1-** 1-1-** 1-** 1-1-** 1-** 1-** 1-1-** 1-** 1-** 1-** 1-1-** 1-** 1-1** 1-** 1-1-** 1-1-** 1-** 1-1-** 1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-** 1-1-1-** 1-1-** 1-1-1-** 1-1-1-** 1-1-1-** 1-1-1-** 1-1-1-** 1-1-1-1-** 1-1-1-1-** 1-1-1-1-** 1-1-1-1-** 1-1-1-1-1-1-**
set, RecGym (Beng et al., 2017), as a case study. However, it is important to notice that the proposed approach can be used with other HAR-related data sets from various sensing modalities. The data set records ten volunteers' gym sessions with a sensing unit composed of an IMU sensor and a Body Capacitance sensor(Beng et al., 2017; Chen et al., 2018). The sensing units were worn at three positions: on the wrist, pocket, and calf. Twelve gym activities are recorded, including eleven workouts like ArmCurl, LegPress, and StairsClimber, and a "Null" activity when the volunteer hangs around between different work sessions. Each participant performed the selected workouts for five sessions in five days. Altogether, fifty sessions of gym workout data are presented in this data set. In this study, we only focus on the motion signals with the sensing unit worn on the wrist.

### Spike encoding

To directly train an SNN, traditional numerical sequential value needs to be transformed into spike streams that carry both temporal and spatial knowledge of the original signals. Different encoding approaches have been explored mostly for vision data transformation (Zhou et al., 2017), like latency encoding, rate encoding, delta modulation, etc. Each has advantages and limitations and has been adopted in different works (Beng et al., 2017). For example, latency encoding normally achieves the best processing latency and energy consumption with fewer synaptic operations while being more susceptible to noise. Rate coding is demonstrated to exist in sensory systems like the visual cortex and motor cortex (Srivastava et al., 2017), showing the best resilience to input noise while limited by a lengthy processing period. In this work, we used the delta modulation approach due to the optimal trade of complexity and latency from both the firmware and hardware implementation. Moreover, the analog sensory information can be directly encoded to the spike train at the front end. To address the accuracy degradation caused by information loss during the encoding, we gave multiple thresholds for spike train generation. The relationship between the continuous signal \(s(t)\) and its spiking counterpart \(\hat{s}(t)\) is given by Equation 1.

\[\hat{s_{i}}(t)=\begin{cases}1,\text{ if }s(t)-s(t-1)>\epsilon_{i}\\ -1,\text{ if }s(t)-s(t-1)<-\epsilon_{i}\end{cases} \tag{1}\]

where \(\epsilon_{i}\) is the threshold empirically chosen for spike encoding, and \(i\) (\(0\sim 4\))represents the index of applied thresholds list. Figure 2 depicts the five spike train channels encoded from the X-axis of the accelerometer, where the fast and large signal trend gives spikes in more spike train channels. For the inertial data, the thresholds were empirically set to 0.00005 x (\(i+1\)), while 0.0000125 x (\(i+1\)) for the capacitance data, and the threshold element number was empirically set to five. In future work, a systematical exploration of choosing the best threshold value and element number will be explored. As seven continuous signals were collected in the data set, we got thirty-five (7x5) spike trains for SNN training and inferring. With two seconds time window as a classification instance, we got 81291 spiking samples for building the SNN model.

### Spiking neural network and Loihi

One of main contribution of this paper is the design of a SNN model and its evaluation on the Intel Loihi research platform. Loihi (2017) is an asynchronous neuromorphic digital processor mainly for research. The processor consists of a many-core mesh of 128 neuromorphic cores for spike processing and three synchronous Lakemont x86 cores to monitor and configure the network and assist with the injection and recording of input/output spikes. Each neuro core in Loihi can access its local memories independently without needing to share a global memory bus and can implement up to 1024 current-based leaky integrate and fire neurons. Among other research platforms, Loihi has been selected because it includes a software SDK to design and profile proposed SNN.

Our proposed SNN is composed of two convolutional and two dense layers (32C64C128D12D) with a kernel size of three, as Table 2 lists. The threshold for neuron spiking was empirically selected. The current and voltage decay constants for the leaky integrate and fire neurons were set to 1024 (32 ms) and 128 (4 ms), respectively. Before spike encoding, the data set was interpolated to 1 kHz using

Figure 1. Human activity recognition using the spiking neural network where the network is processed on neuromorphic platforms pursuing energy and latency efficiency

Figure 2. Encoded spike trains of signal Acc_x from the workout of BenchPress with thresholds of [0.00005, 0.0001, 0.0002, 0.0004, 0.0008]
the univariate spline method, aiming to maximally approach the biological behaviors of the brain regarding information feeding. Each sample contains two seconds length of spike trains. The model was trained offline on GPU with weighted classes and leave-one-user-out cross-validation, and the trained weights and delays were then used to configure the network on Loihi hardware for inference purposes.

To fully make use of the biological plausibility of SNN, we used the framework SLAYER (Schaver, 2017) for direct training, aiming to pursue the envelope of energy and latency efficiency of SNN. SLAYER evaluates the gradient of the convolutional and linear layers in an SNN by a temporal credit assignment policy, which distributes the error credit of an error back both through layers and in time, as a spiking neuron's current state depends on its previous states. Then a probability density function is used to estimate the change in the neuron state, thus approximating the derivative of the spike function. With SLAYER, the synaptic weights and axonal delays can be trained, and some state-of-the-art performances have been achieved on neuromorphic datasets like the NMNIST and ibmDVS-Gesture (Schaver, 2017). SLAYER supports Loihi-specific implementation for neuron model and weight quantization.

## 4. Experimental Evaluation

Table 3 lists the workouts classification performance with the trained SNN on Loihi. In comparison, we selected an ANN model using the same data set and being deployed on two different IoT processors, presented in (Louhihi et al., 2017). Such a comparison has seldom been made, as previous SNN evaluations mostly used ANN deployed on GPU/CPU as the baselines. The result will be meaningful for developing ubiquitous neuromorphic edge computing by supplying a straightforward comparison with the state-of-the-art using mainstream solutions. The multi-threshold spike encoding approach results in an accuracy of 87.5% with the directly trained SNN, which is much better than the single-threshold encoding result (below 60%) and acceptable compared with the accuracy from the ANN approach considering that the accuracy of direct-trained SNN on spike streams degrades in most cases. The inference latency of SNN on Loihi implies the time elapsed between the end of the input and the classification output and is reported as 4.4 ms, which is also much better than the latency on general IoT processors like STM32 with Cortex-M7 core but falls behind slightly to the GAP8, which features 8 RISC-V cores for dedicated hardware acceleration. However, the neuromorphic pipeline outperforms in dynamic energy consumption (0.15 mJ), benefitting from the sparsity of the spike trains and the in-memory computing of Loihi, which results in an EDP of 0.66 \(\upmu\)s, while the EDP on GAP8 and STM32 are almost two times and over two hundred times higher, respectively. The energy reported here is the dynamic energy on Loihi, which is measured by enabling the energy probe during inference as the difference between the total energy consumed by the network and the static energy when the chip is idle. We have to acknowledge that Loihi is not for edge computing specifically. Instead, it is designed more for general-purpose neuromorphic research. Thus there is still space for raising the neuromorphic performance, for example, the spike injection speed (the primary x86 core always waits 1ms before allowing the system to continue to the next timestep). To have a more fair comparison, end-to-end solutions of neuromorphic and traditional approaches should be developed, adopting the newly released edge neuromorphic processors (Schaver, 2017; Schaver, 2017).

## 5. Conclusion

This work explored the neuromorphic solution of human activity recognition with a typical case study of workout recognition. Neuromorphic solutions, mainly inferring the SNN associated with the neuromorphic processor, have been emerging benefiting from its latency and energy efficiency. We started with a multi-threshold delta modulation to encode the raw motion sensor signal into multiple spike trains, aiming to reduce the information loss during spike generation. A shallow SNN model was then used to train the spike-form workouts signal with the SLAYER framework. The model runs on Loihi showed a comparable accuracy of 87.5% and an impressive energy-delay-product of 0.66 \(\upmu\)s, compared with the state-of-the-art ANN solution on GAP8. This work demonstrates the efficiency of neuromorphic solutions in ubiquitous computing that pursues latency and energy efficiency. For future work, we will focus on new features in neuromorphic solutions that exceed the traditional edge solutions, for example, learn on the fly that can adapt the SNN models for specific subjects and environments, boosting the inference accuracy. We will also explore the newly released edge neuromorphic platforms and Loihi2, which has redesigned asynchronous circuits supplying faster speed and enhanced learning capabilities, featuring multiple times performance boosting compared with its predecessor.

###### Acknowledgements.

 This work was supported by the CHIST-ERA project ReHab(20CH21-203783).

\begin{table}
\begin{tabular}{c|c|c|r|r|r}  & Type & Size & Feature Size & Features & Stride \\ \hline
0 & Input & 7x5x2 & - & - & - \\
1 & Conv & 7x5x32 & 3x3 & 32 & 1 \\
2 & Conv & 7x5x64 & 3x3 & 64 & 1 \\
3 & Dense & 2240 & - & 128 & - \\
4 & Dense & 128 & - & 12 & - \\ \end{tabular}
\end{table}
Table 2. SNN model for the spiking RecGym dataset

\begin{table}
\begin{tabular}{l c c c c} \hline Hardware & Model & Accuracy & Latency & Energy & Energy-Delay \\  & & & (ms) & (m) & Product (\(\upmu\)s) \\ \hline Loihi (Neuromorphic) & SNN & 87.5\% & 4.4 & 0.15 & 0.66 \\ \hline GAP8 (RISC-V) & ANN & 88.1\% & 3.2 & 0.41 & 1.31 \\ \hline STM32 (Cortex-M7) & ANN & 89.3\% & 20.88 & 8.07 & 168.5 \\ \hline \end{tabular}
\end{table}
Table 3. Classification profiling vs. general edge solutions