capbtabboxtable[][0.45]

# Comparison of end-to-end neural network architectures and data augmentation
methods for automatic infant motility assessment using wearable sensors

Manu Airaksinen1,2, Sampsa Vanhatalo2,4,5, and Okko Räsänen1,3

###### Abstract

Infant motility assessment using intelligent wearables is a promising new
approach for assessment of infant neurophysiological development, and where
efficient signal analysis plays a central role. This study investigates the
use of different end-to-end neural network architectures for processing infant
motility data from wearable sensors. We focus on the performance and
computational burden of alternative sensor encoder and time-series modelling
modules and their combinations. In addition, we explore the benefits of data
augmentation methods in ideal and non-ideal recording conditions. The
experiments are conducted using a data-set of multi-sensor movement recordings
from 7-month-old infants, as captured by a recently proposed smart jumpsuit
for infant motility assessment. Our results indicate that the choice of the
encoder module has a major impact on classifier performance. For sensor
encoders, the best performance was obtained with parallel 2-dimensional
convolutions for intra-sensor channel fusion with shared weights for all
sensors. The results also indicate that a relatively compact feature
representation is obtainable for within-sensor feature extraction without a
drastic loss to classifier performance. Comparison of time-series models
revealed that feed-forward dilated convolutions with residual and skip
connections outperformed all RNN-based models in performance, training time,
and training stability. The experiments also indicate that data augmentation
improves model robustness in simulated packet loss or sensor dropout
scenarios. In particular, signal- and sensor-dropout -based augmentation
strategies provided considerable boosts to performance without negatively
affecting the baseline performance. Overall the results provide tangible
suggestions on how to optimize end-to-end neural network training for multi-
channel movement sensor data.

1Department of Signal Processing and Acoustics, Aalto University, Finland
2BABA center, Pediatric Research Center, Children’s Hospital, Helsinki
University Hospital and University of Helsinki, Finland
3Unit of Computing Sciences, Tampere University, Finland
4Department of Clinical Neurophysiology, HUS Medical Imaging Center,
University of Helsinki, Helsinki University Hospital and University of
Helsinki, Finland
5Neuroscience Center, University of Helsinki, Helsinki, Finland

##  1 Introduction

Developments in wearable electronics and especially in integrated sensors with
inertial measurement units (IMUs) have made automatic human activity
recognition (HAR) feasible in various use scenarios. Typical HAR systems aim
at detecting various everyday activity patterns of a person (e.g., sitting,
walking, running). These data can then be used for various purposes ranging
from personal activity logs [15] to medical analyses [5] (see [16] for recent
review). One particularly promising example of the latter HAR application area
involves automatic monitoring of infant movements using wearable intelligent
garments. As the observed motor behaviour of a child is closely coupled to the
development and integrity of the child’s central nervous system, abnormal
movement patterns or lack of certain movement skills at a given age can be
predictive of deviant neurocognitive development such as cerebral palsy [10].
In the same manner, automatic at-home monitoring of an infant undergoing a
therapeutic intervention can produce data on intervention efficacy that is
difficult to obtain during time-limited (and often uncomfortable) hospital
visits [9].

In our previous work [2], we developed a comfortable-to-wear infant overalls
(or jumpsuit) called “Maiju” (Fig. 1; Motor Assessment of Infants with a
JUmpsuit) with integrated IMUs (inertial measurement units) in each limb for
automatic assessment of infant movement repertoire for out-of-the-lab
settings. We also collected a unique data-set consisting of visually annotated
posture and movement data from several infants. By measuring movements of a
child using Maiju over some tens of minutes of independent movement, followed
by neural network-based signal analysis pipeline, our setup was able to
recognize different postures and movement patterns of the child with human
equivalent accuracy. However, the signal classification pipeline and its
training protocol used in the initial study were not results of systematic
optimization, but rather an attempt to construct a functional solution for the
task. Despite the apparent success, many technical details remained to be
defined more systematically to meet our ultimate goal to develop a widely
scalable system for out-of-hospital movement analysis in infants.

In this work, we take a more principled approach to the technical development
of a infant movement classifier from wearable IMUs. More specifically, we
investigate different end-to-end neural network architectures for multisensor
IMU data processing in infant movement analysis. We also analyze the utility
of different training data augmentation strategies for classifier performance
and robustness. Different system variants are tested in conditions that
simulate issues encountered in practical out-of-the-lab use scenarios, such as
sensor dropout or packet loss due to real-time wireless data transfer issues.
We also compare performance of different system architectures to their memory
footprint and computational complexity. As a result, we find many such
strategies in sensor signal encoding and time-series modeling that may benefit
also many other HAR applications that are based on multi-sensor setups.

Figure 1: Illustration of the recording set-up and data pre-processing steps.

##  2 Background

The Maiju jumpsuit reported in [2] records posture and movement patterns of
infants by measuring their movements with a set of limb-mounted IMUs (see Fig.
1) and by transforming the resulting signals into posture/movement categories
using an end-to-end neural network processing pipeline. Our initial study
showcasing the smart jumpsuit contained a broad scope, detailing the process
of textile design, recording set-up, data-set design including annotation and
its analysis, and finally, the evaluation of automatic classification
performance. As such, a broad comparison and optimization of varying
classifier architectures was beyond the scope of the study. Given the real-
world relevance of the analysis task and uniqueness of the collected data-set,
the question on the effects of varying end-to-end classifier architectures is
poorly understood and requires proper investigation.

The general neural network architecture presented in [2] can be thought of
consisting of an _encoder module_ that performs intra- and inter-sensor
channel fusion to obtain a unified feature representation from the raw tri-
axial accelerometer and gyroscope signals of each IMU, and a _time-series
modeling module_ for modeling the temporal context in the activity recognition
process (see Fig. 2). During training, both the encoder and time-series module
are jointly optimized for the classification task in an end-to-end manner.

HAR literature contains multiple proposals to perform raw-input feature
extraction for the encoder model (e.g., [4, 16]), and there is a growing
number of popular time-series modeling methods making use of deep learning [7,
3, 13]. The first goal of the present study is to find the best overall
combination of the two modules and also to understand the performance of
different modules in movement classification using the previously collected
data-set. Varying classifier architectures can have varying levels of
complexity in terms of model size (number of trainable parameters, memory
footprint) and computational complexity (model training time, generation
time). The second objective of the present study is to map the tested
classifiers within this three-dimensional (performance vs. size vs. speed)
space.

Finally, the recording setup of Maiju performs real-time streaming of the raw
data to the controlling device (a mobile phone) over Bluetooth. This means
that the real-life recordings might suffer from packet loss and/or sensor
drop-out. It has been well documented that data augmentation methods can
improve model robustness in image and audio processing applications [11, 1] as
well as in HAR applications [12]. Thus the final goal of the present study is
to explore the effect of data augmentation methods for the Maiju jumpsuit
movement classification task in optimal conditions and with varying levels of
packet loss or sensor drop-out.

Table 1: Details of the utilized infant motility dataset. | Overall statistics
---|---
Number of recordings | 22
Participant age (months) | Avg. 6.6 $\pm 0.84$ (min 4.5, max 7.7)
Recording length (min) | Avg. 28.9 $\pm 7.7$; min 8.6, max 40.4
Total length | 10h 35min (33021 frames)
Movement categories | Total length (min, frames, % present) /
Still | 376 min, 19547 frames, 63.5%
Proto movement | 182 min, 9483 frames, 28.7%
Turn L | 9 min, 485 frames, 1.5%
Turn R | 9 min, 466 frames, 1.5%
Pivot L | 18 min, 978 frames, 3.0%
Pivot R | 20 min, 1030 frames, 3.1%
Crawl commando | 20 min, 1032 frames, 3.1%

##  3 Data collection method

The “Maiju” smart jumpsuit is a full body garment that allows spontaneous
unrestricted movements of approx. 4–16 month old babies. The jumpsuit performs
tri-axial accelerometer and gyroscope recordings at a 52 Hz sampling rate from
four Suunto Movesense sensors with proximal limb placements. The sensors
stream the raw data via Bluetooth 4.0 Low Energy to a controlling smart phone
(iPhone 8); see Fig. 1).

The data-set collected in our previous study [2] is used for the experiments
within the present study. The data-set contains recordings of independent non-
assisted movement of 22 infants from approximately 30-minute play sessions
facilitated and monitored by a physiotherapist. The infants were approximately
7 months old and normally developing, meaning that their approximate motility
skills range from turning to crawling (see Table 1 for further details). Each
recording of the data-set was video recorded and annotated by three
independent experts based on a coding scheme reported in [2]. The annotations
contain two parallel tracks: one for posture (5 categories) and one for
movement (7 categories; see Table 1 for a list). In the previous study,
classification in the movement track turned out to be more challenging
compared to the posture track ($\approx$94% unweighted average F1-score for
posture, $\approx$73% for movement). Furthermore, similar posture
classification results were obtained with a traditional feature-based support
vector machine (SVM) classifier compared to the end-to-end convolutional
neural network (CNN) classifier, but the CNN produced statistically
significant improvements for movement classification. Based on this, only the
movement track-based data-set was selected for the experiments within the
present study.

In order to derive training class labels for the data-set, the previous study
used the three parallel annotations for each recording within the so-called
_iterative annotation refinement_ (IAR) process to obtain the most likely one-
hot labels for the movement categories [2]. In IAR, the probabilistic
interpretation of the human annotations is iteratively weighted with the
softmax output of the automatic classifier judgements, which results in more
consistent decision boundaries between categories. In this study, we will use
the previously-derived IAR-based labels for the movement classes of the data-
set. For further details, see [2] and its supplementary material.

Figure 2: Overall schematic of the compared classifier architectures.

##  4 Methods

###  4.1 Data pre-processing and windowing

The data pre-processing and windowing process is illustrated in Fig. 1. First,
the raw data from the jumpsuit sensors (4 IMUs, 3 accelerometer + 3 gyroscope
channels each) is streamed into the recording device in packets of four
subsequent time samples (4 timestamps + 4 x 6 channels). Due to sampling
jitter, the timestamps and signals from each sensor are first linearly
interpolated from the recorded time-base into a synthesized ideal time-base
with a 52-Hz sampling rate that is uniform for each sensor. The interpolated
signals are organized into a matrix with columns corresponding to different
recording channels. The gyroscope bias is estimated (and subtracted) from each
individual channel by computing the mean of the 64-sample segment with the
smallest variance within the recording. Next, excess noise is removed from the
recordings by median filtering each channel separately (5-sample median).
Finally, the recording matrix of shape (24, $N_{rec}$), where $N_{rec}$ is the
length of the recording in samples, is windowed into 120-sample long (2.3 sec)
frames with 50% overlap using a rectangular window and stored into a tensor
with shape ($N_{frames}$, 24, 120) to be used as an input to the classifier
stage.

###  4.2 Overall classifier design

The overall schematic of the processing pipeline is presented in Fig. 2. For
each recording, the pre-processed input tensor is fed into an encoder module
that produces a feature representation of 160 bottleneck features from each
frame. The bottleneck features are fed into the time-series modeling module
that tracks the frame-to-frame behavior of the bottleneck features, and
finally produces the output movement classification for each frame. In our
experiments, the encoder modules (Section 4.3) and time-series modeling
modules (Section 4.4) can consist of 5 distinct architectures, resulting in a
total of 25 combinations to be investigated.

(a) Dense encoder.

(b) 1-D CNN (Conv1D) encoder.

(c) 2-D CNN with individual paths (Conv2D-I) encoder.

(d) 2-D CNN with individual & shared paths (Conv2D-IS) encoder.

(e) 2-D CNN with sensor-independent modules (Conv2D-SI) encoder.

Figure 3: Compared encoder module architectures.

###  4.3 Compared sensor encoder modules

####  4.3.1 Dense network

The “Dense” encoder structure, presented in Fig. 3a, is the most simple
approach to learn an end-to-end feature representation. Each input frame is
flattened across all the channels from all the IMUs into a vector of size 1 x
2880, and a fully-connected transformation with tanh activation is applied to
it, reducing the dimensionality to 256. Three additional fully connected
transformations follow, and the output is the 160-dimensional bottleneck
feature vector for each frame. The inclusion of the “dense” encoder into the
experiments is mainly motivated by the assumption that it acts as an anchor
point for low-end performance compared to the more complex CNN architectures,
as the number of trainable parameters is higher and the module does not
utilize weight sharing.

####  4.3.2 1-D CNN

The 1-dimensional CNN system (Conv1D; Fig. 3b) is a simple CNN system that
utilizes spatial weight sharing over channels of filters to perform
dimensionality reduction of the sensory input. The input tensor of the 1-D CNN
system is treated so that the 24 channels are fully connected into the first
convolution layer’s filter channels, and the convolution filters operate over
the time axis of the windowed samples. Each convolutional layer has a filter
size of 5 and stride of 2, effectually halving the temporal dimension of the
input within each layer. The final layer of the 1-D CNN concatenates the
obtained feature channels into a flat vector and performs a fully-connected
transformation to obtain the output size of 160 features per frame. The
activation function for each layer is a leaky rectified linear unit (LReLU).

####  4.3.3 2-D CNN with individual paths

The previous encoder systems (dense and 1-D CNN) work under the assumption
that the same learned features can be extracted from the parallel
accelerometer and gyroscope signals. In reality, the accelerometer and
gyroscope sense complementary inertial information that potentially benefit
from individualized features. The “2-D CNN with individual paths” (Conv2D-I;
Fig. 3c) splits the input paths of the encoder modules to perform feature
extraction from the accelerometer and gyroscope separately. In addition, 2-D
convolutions are utilized in the first two layers to perform spatially
relevant channel fusion: The first layer fuses each sensors’ tri-axial (xyz)
channels into shared sensor-level channels, and the following layer fuses the
four sensor-level channels of the measurement modality (i.e., acc or gyro)
with the 2-D convolution. The accelerometer and gyroscope-specific features
are concatenated, and two fully connected operations are performed to mix the
domain-specific information to the final output representation. The activation
functions for each layer are leaky ReLUs.

(a) Dense time-series model.

(b) LSTM time-series model.

(c) GRU time-series model.

(d) BGRU time-series model.

(e) WaveNet time-series model.

Figure 4: Compared time-series module architectures.

####  4.3.4 2-D CNN with individual & shared paths

The 2-D CNN with individual & shared paths (Conv2D-IS; Fig. 3d) is identical
to the 2-D CNN with individual paths, except that the encoder input has also
an additional shared path, where shared filters are learned for the
accelerometer and gyroscope alongside the individual paths. This encoder is
equivalent to the encoder utilized in our initial study [2]. The addition of
the shared path was reported to increase performance in a previous study
regarding multi-IMU sensor HAR [4], and the comparison between the 2DCNN-I and
2DCNN-IS systems is interesting regarding the generalizability of the
aforementioned findings.

####  4.3.5 2-D CNN with sensor-independent modules

The 2-D CNN with sensor-independent modules (Conv2D-SI; Fig. 3e) encoder aims
to leverage a higher order of weight-sharing compared to the previous CNN
systems: The encoder is set to produce independent feature representations for
each IMU sensor using a shared sensor model, whose outputs are finally
concatenated. The sensor module utilizes similar architecture to the Conv2D-IS
system (three total paths), with the exception that no inter-sensor mixing of
channels is applied in any of the layers. As such, the encoder is forced to
learn a representation that is generalizable for each sensor, regardless of
its location in the recording setup.

###  4.4 Compared time series modeling modules

####  4.4.1 Dense network

In analog to its encoder counterpart, the “Dense” time-series modeling module
(Fig. 4a) acts as a baseline anchor point for the time-series modules: The
module has one hidden fully connected layer, and in fact does not perform any
time-series modeling but simply transforms the output of the sensor module
into a movement classification decision.

####  4.4.2 LSTM

Long-short term memory (LSTM [7]; Fig. 4b) networks have become one of the
most common recurrent neural network (RNN) architecture choices. LSTMs
circumvent the vanishing gradient problem of RNNs by utilizing a distinct cell
memory state that is manipulated (read/write/reset) by soft-gating operations.
As such, LSTMs are ideal for learning long-term connections within the time-
series input data, such as those exhibited in speech and language. In the
context of HAR, and especially within the utilized data-set, long-term
connections are not probably overly emphasized, as infant movement is not pre-
planned over longer time periods. In addition, LSTM training can exhibit
instability issues when training with smaller data-set sizes, which might make
LSTM modeling non-ideal for many medical applications where access to massive
amounts of existing data may be limited, such as the present dataset.

####  4.4.3 GRU

The gated recurrent unit (GRU [3]; Fig. 4c) is a more recent variant of LSTM-
inspired RNNs. Instead of utilizing distinct cell states, the GRU cell
performs the learned soft-gated operations (read/write/reset) to the previous
output state. This has the advantage of stabilizing the training compared to
LSTMs, but with a trade-off in sensitivity to model long-term connections
within the time-series. As discussed in the LSTM section, the long-term
connections are not expected to be overly important within the present data-
set of infant motility.

####  4.4.4 Bi-directional GRU

The bi-directional GRU (BGRU; Fig. 4d) utilizes two GRU modules to do the
time-series modeling: One in the forward direction, and another in the
backward direction. The outputs of these two sub-modules are concatenated
before the final fully-connected layer. The bi-directional design enables
conditioning of the RNN time-series decisions with the past and future
information of the recording. This comes with the price of increased model
complexity and the system being unable to perform causal inference for real-
time applications.

####  4.4.5 WaveNet

The WaveNet [13] architecture was first proposed as a speech synthesis
specific modification of the ResNet [6] architecture used for image
classification. The WaveNet consists of stacked blocks of gated dilated
convolutions. The outputs of the blocks feed into the next block, and are also
connected to the output of the block stack with residual connections. For
time-series modeling, these properties have multiple desirable features:
First, the residual connections between the blocks enable stacking of multiple
convolutional layers without the vanishing gradient problem, because during
training the error gradient signal is able to flow freely along these
shortcuts to the deeper layers. Second, the stacked dilated convolutions reach
a sufficiently wide _receptive field_ of frames that condition each output
classification without the need for recurrent connections. This allows for
parallel training, which considerably speeds up the development process. In
the present study, the WaveNet timeseries module is the same that was utilized
in out initial study [2]. It uses four residual blocks with dilation sizes of
[1, 2, 4, 8] with filter size 5, reaching a receptive field of 60 frames (69.2
seconds).

###  4.5 Data augmentation methods

Data augmentation methods apply perturbations and/or transformations to the
training data so that the actual raw input data values change according to
potential intra-class variation observed beyond the training data and without
affecting the label-identity of the chosen frame. Augmentation aims at either
increasing 1) absolute model performance or 2) model robustness in expected
real-world settings. We aim to investigate both of these effects with a number
of augmentation methods (with modifications) that have been previously
proposed in [12].

####  4.5.1 Drop-out

Drop-out is generally a popular deep learning method to increase model
generalization: During training, a binary drop-out mask is applied to a given
layer, which sets a given percentage of its activation values to zero.
Effectively, this forces the neural network to learn robust distributed
representations for the same input. For the present study, we experimented
with drop-out at two different positions: 1) in the input tensor, and 2) in
the bottleneck features between the encoder and time-series modeling modules.
The first case can be thought of data augmentation for the case of packet loss
that might occur during a real-world recording. The drop-out probability for
both cases is set to 30% in the present experiments.

####  4.5.2 Sensor drop-out

Sensor dropout aims to increase model robustness by randomly dropping one IMU
sensor of a mini-batch during training (i.e., the sensor’s accelerometer and
gyroscope information is set to zero). The probability of a sensor to drop is
set to 30%, and the sensor to be dropped is selected randomly from a uniform
distribution. This forces the classifier to learn representations that are not
dependent on any one particular limb’s movement information. The results in
[2] suggest that a combination of three or even two of the four sensors are
capable of producing nearly on-par classification performance compared to the
full set when using an SVM classifier.

####  4.5.3 Sensor rotation

Even though the sensor mounts of the Maiju jumpsuit are placed in fixed
positions, the actually realized positioning of each sensor with relation to
the infant’s body is affected by the body type of the infant. This effect can
be simulated with artificial _sensor rotation_ , where a 3x3 rotation matrix
is used to mix the channels of each sensor. To perform random rotations to the
sensors, we utilize the Euler rotation formula where the yaw ($\alpha$), pitch
($\beta$) and roll ($\gamma$) angles are randomly sampled from a uniform
distribution between $\pm 10^{\circ}$:

$\displaystyle R_{z}(\alpha)=\begin{bmatrix}\cos\alpha&-\sin\alpha&0\\\
\sin\alpha&\cos\alpha&0\\\ 0&0&1\end{bmatrix}$ (1) $\displaystyle
R_{y}(\beta)=\begin{bmatrix}\cos\beta&0&\sin\beta\\\ 0&1&0\\\
-\sin\beta&0&\cos\beta\end{bmatrix}$ (2) $\displaystyle
R_{x}(\gamma)=\begin{bmatrix}1&0&0\\\ 0&\cos\gamma&-\sin\gamma\\\
0&\sin\gamma&\cos\gamma\end{bmatrix}$ (3) $\displaystyle
R(\alpha,\beta,\gamma)=R_{z}(\alpha)R_{y}(\beta)R_{x}(\gamma),$ (4)

and the final augmented sensor signal is obtained by multiplying the tri-axial
accelerometer and gyroscope signals with the resultant matrix.

####  4.5.4 Time warping

Time warping augmentation aims to add variability to the training data by
altering the speed at which movements occur. This is simulated frame-by-frame
by synthesizing a new timestamp differential base vector with the formula

$dt_{\text{new}}(dt_{\text{old}})=2+A\sin(2\pi\omega
dt_{\text{old}}+2\pi\phi),$ (5)

where the frequency ($\omega$), phase ($\phi$), and amplitude ($A$) are
randomly sampled from a uniform distribution between [0, 1]. The synthesized
timebase differential vector ($dt_{\text{new}}$) is then integrated, and each
frame’s signals are linearly interpolated to the new timebase. Effectively,
this adds sinusoidal fluctuation to the speed of movement while preserving the
overall length of each frame. It is to be noted that this method of time
warping augmentation does not precisely transform the recorded accelerometer
or gyroscope signals into forms that they would take if the actual limb
movements would be altered with the same time warping operation.

Figure 5: Unweighted average F1-scores as a function of encoder modules
(x-axis placement) and timeseries modules (color coding). Figure 6: Average
performance of encoder modules (’o’s, left side) and timeseries modules (right
side) with their 95% confidence intervals. Statistically significant
differences (Wilcoxon rank sum test;$p<0.05$) between the average effect of
the modules are marked with ’*’.

##  5 Experiments and results

The goals of the experiments were three-fold: 1) investigate the performance
of different sensor encoder and time-series modules, 2) compare their
performance with their computational and memory loads, and 3) study the
effects of data augmentation on system performance. The overall aim was to
identify the best performing solution for infant motility assessment using
Maiju with reasonable computational requirements.

###  5.1 Training and evaluation details

For all of the experiments, each neural network combination was trained end-
to-end with random initialization without any pre-training. To mitigate the
effect of result variability due to the random initialization, each experiment
was repeated three times and their average is reported as the result. The
neural network training was performed with stochastic gradient descent using
minibatches of 100 frames. The Adam algorithm [8] with parameter values of
learning rate=10e-4, $\beta_{1}=0.9$ and $\beta_{2}=0.999$ was used to perform
the weight updates. 7-fold cross-validation was used to obtain classification
results for the entire dataset. To monitor training, 20% of the minibatches
from each recording of the training set were assigned for validation to
monitor convergence and to trigger early stopping. The training was performed
for a maximum of 250 epochs with an early stopping patience of 30 epochs.

The unweighted average F1-score (UWAF) was chosen as the main performance
metric in the present study. F1-score of one category is the harmonic mean of
the precision and recall of that category:

$F1=\frac{2P_{c}R_{c}}{P_{c}+R_{c}}=\frac{tp}{tp+0.5(fp+fn)},$ (6)

where $P_{c}$ is the precision of category $c$, $R_{c}$ is the recall of
category $c$, tp is the number of true positive classifications, fp the number
of false positives, and fn the number of false negatives. After the F1 score
is computed individually for each output category of the data-set, the UWAF
can be computed as their average. UWAF is preferable over the standard F1
score with heavily skewed class distributions, as in the case of the present
study.

###  5.2 Experiment 1: Comparison of architecture models

The first experiment compared the classifier performance of all of the
encoder–time-series module combinations presented in Sections 4.3 and 4.4 as
measured by UWAF. The results, sorted in ascending order of performance and
grouped according to the encoder modules, are presented in Fig. 5. In
addition, the grouped performances of the individual modules are presented in
Fig. 6, where statistically significant differences between the architectures
are reported with the Wilcoxon rank sum test.

The results show that the best overall performance was achieved with the
Conv2D-SI encoder with the WaveNet time-series model. The choice of the
encoder module is seen to have the largest impact in the classifier
performance: The Conv2D -based encoders significantly outperform the
competitors. Within the alternative Conv2D models, the simplest model,
Conv2D-I, has the weakest performance. The Conv2D-SI model slightly
outperforms the previously proposed Conv2D-IS model. The Conv2D-SI model also
has the smallest performance variation across the various time-series models,
which suggests that the bottleneck representations learned by the encoder are
the most robust (i.e., time-series module does not need to learn to compensate
the shortcomings of the encoder representation).

The choice of the time-series model is seen to have a clear hierarchy in
performance, where the WaveNet -based model systematically has the best
performance. An interpretation for this effect is that the WaveNet model has
more powerful modeling capabilities that learn to compensate some of the
inefficiencies of inferior encoder modules. Surprisingly, the GRU module
systematically outperforms the BGRU module, albeit with a minor margin.

An interesting effect is also seen within the LSTM-model: The worse the
encoder performance is (i.e., the weaker the bottleneck representation is),
the worse the LSTM also performs. With the best performing encoder
(Conv2D-SI), LSTM performance becomes on-par with the rest of the RNN-based
models. This finding is in line with the conventional wisdom regarding LSTM
training instability, and suggests that RNN training might benefit from pre-
trained bottleneck features that would make the end-to-end training more
stable.

Figure 7: Average performance of trained systems as a function of model size
(x-axis) and the generation time of the test set (symbol area).

###  5.3 Experiment 2: Effects of model size and computational load to
performance

As further analysis of the first experiment systems, we explored the
relationship of model size and model training time to classifier performance.
Typically, smaller models have better generalization with a loss to detail
modeling capabilities, whereas larger models have better modeling capacity but
are prone to over-fitting during training. For research and development
purposes, model training time is another feature of interest, as it determines
a limit on the rate of experiments that can be performed given a limited
number of computational resources. RNN-based systems are known to have
considerably slower training times because the error backpropagation through
time requires unraveling of the entire computational graph over all of the
timesteps of the mini-batch [17]. In contrast, network structures that do not
include recurrent connections can be trained in parallel, which significantly
speeds up the training.

The performance – model size – training time comparison of the systems is
presented in the bubble chart of Fig. 7. The x-axis represents the number of
trainable parameters in each model, the y-axis represents the classifier
performance, and the bubble areas are proportional to the training times of
the given model. The results show that the conventional wisdom regarding dense
networks and CNNs holds true within the encoder models: The Conv1D encoder
outperforms the Dense encoder with almost a 10-fold reduction in trainable
parameters. However, the Conv2D encoders clearly exhibit an optimal middle-
ground where more nuanced parameter sharing with a moderate increase in
trainable parameters results in superior classifier performance over the
Conv1D and Dense encoders.

Within the time-series models, the RNN-based modules exhibit considerably
longer training times, and the Dense and WaveNet time-series module training
times were roughly equivalent. From the fastest to slowest training times, the
ratios are: 1 (Dense) : 1 (WaveNet) : 4.9 (GRU) : 6.5 (LSTM) : 7.6 (BGRU). The
mean ratio between RNN and non-RNN training times is approximately 6:1. For
the encoder models, the training times have ratios of 1 (Dense) : 1.3 (Conv1D)
: 1.4 (Conv2D-I) : 1.7 (Conv2D-IS) : 2.1 (Conv2D-SI).

As the final model size experiment, we chose the best-performing system,
Conv2D-SI + WaveNet, and tested its performance with a varying number of
bottleneck features. As the Conv2D-SI encoder produces sensor-independent
features, a decrease in bottleneck layer dimensionality directly reduces the
number of features per sensor, thus also reducing the network size. This is an
interesting avenue of investigation when considering embedded solutions that
would perform the bottleneck feature extraction inside the sensor.
Implementing such a scheme would allow data collection without on-line
streaming of the raw data, which would significantly increase recording
reliability and sensor battery life. The effect of varying bottleneck size to
Conv2D-IS + WaveNet system performance is presented in Fig. 8. The results
show that halving the bottleneck size to 20 features per sensor (80 total)
reduces the number of trainable parameters by 62% and the training time by 23%
while having only a 0.5%-point drop in performance. This corresponds to a
substantial compression ratio of 36:1 compared to the raw sensory data.
Further halving of the bottleneck size produces a more considerable drop in
performance.

Figure 8: Effect of the number of bottleneck features for Conv2D-IS to the
unweighted average F1 score.

(a) Sensor drop-out results.

(b) Packet loss results.

Figure 9: Classifier model robustness to noise and the effect of data
augmentation. Augmentation method key: dr1 = input dropout; dr2 = bottleneck
dropout; rot = rotation; tw = time warping; ds = sensor dropout.

###  5.4 Experiment 3: Model robustness to noise

The final set of experiments explored classifier model robustness to noise as
well as the effect of various data augmentation methods. For these
experiments, we chose the best-performing system, Conv2D-IS + WaveNet
alongside two of the systems with the lowest number of trainable parameters:
Conv1D + Dense and Conv1D + GRU. Robustness was tested for sensor-dropout and
packet-loss cases under two conditions: moderate and severe.

For sensor dropout, one (moderate condition) or two sensors (severe condition)
were dropped for each recording. Every possible combination of the dropped
sensors was tested, and the average performance across the tests is reported
in the results. For packet loss, the moderate condition simulated 25% random
drop-out in 4-sample bursts per sensor. The severe condition used a drop-out
value of 50%. For both cases, the drop-out simulation was repeated for five
times and the average result is reported.

The results are presented in Fig. 9. For all of the systems, the application
of data augmentation produces minor differences compared to the baseline
without augmentation when tested with clean data. For the sensor drop-out
conditions, considerable boosts to performance can be observed with the
augmentation methods where sensor drop-out is present: An average effect of
+5.5%-points for 1 sensor dropped and +12.5%-points for the 2 sensors dropped.
For the Conv1D-based models, input-level dropout also produces a considerable
performance boost (average effect of +5%-points and +7%-points).

The results for the packet-loss case are somewhat more diverse: For the
moderate case, the performance of the Conv2D-SI + Wavenet model is on par with
the “no dropout” case even without augmentation. This further suggests that
the weight sharing in the Conv2D-SI encoder produces a robust low-dimensional
representation for the movement sensor data. Rather surprisingly, input-level
dropout does not increase performance of this system even in the severe packet
loss case, but an increase in performance is observed with sensor dropout
during training. For the Conv1D encoder-based systems, input-level dropout
produces the greatest augmentation effect for the moderate and severe packet
loss cases, whereas sensor dropout produces a smaller positive effect. In
summary, the combination of input-level dropout with sensor dropout seems to
be the most beneficial training strategy for the Maiju classifier, as it
maintains the clean baseline performance while making the classifier
considerably more robust in adverse conditions.

##  6 Discussion

This study shows that the choice of end-to-end neural network architectures
has a major impact on the performance of human activity recognition from
multi-sensor recordings. The choice of the encoder module (responsible for
intra- and/or inter-sensor channel fusion) has the greatest significance in
the resulting performance. Out of the tested encoders, the best performance
was obtained with the “Conv2D-SI” model which uses parallel 2-dimensional
convolutions for intra-sensor channel fusion with shared weights for all
sensors. Furthermore, experiments regarding the bottle-neck size of the the
“Conv2D-SI” encoder indicate that a relatively compact feature representation
is obtainable for within-sensor feature extraction without a drastic loss in
classifier performance. Comparison of time-series models reveals several key
effects. First, the best performing system, the “WaveNet” architecture based
on feed-forward dilated convolutions, outperforms the RNN-based models in
performance, training time, and training stability, making it the preferred
time-series modelling method for the task. The same results also show the
importance of input-feature robustness when training RNN-based models,
especially LSTMs. This suggests that some level of pre-training (supervised or
unsupervised) would be preferable before inserting an RNN module into an end-
to-end classifier architecture.

The experiments comparing data augmentation methods did not find additional
benefit to classifier performance when using clean data as input. However,
when studying model robustness in noisy conditions, such as simulated packet
loss or dropped sensor scenarios, drop-out -based (input drop-out or sensor
drop-out) augmentation provided a considerable boost to performance without
affecting the clean data performance. We speculate that the that the lack of
performance boost from data augmentation is due to performance ceiling caused
by the ambiguity in defining the gold-standard annotations of infant movement
categories before the infants reach archetypical adult-like motor skills. Our
previous study showed that the classifier output reaches the inter-rater
agreement $\kappa$ value (of what) between three independent annotators [2].

In conclusion, this study suggests how to optimize end-to-end neural network
training for multi-channel movement sensor data using moderate sized data-
sets. First, the design of the encoder/channel fusion component of the network
greatly benefits from weight sharing and appropriate handling of data
modalities from different sources. Second, feed-forward convolutional
approaches with wide receptive fields are well suited for time-series
modeling, and outperform RNNs in terms of accuracy, training speed and
robustness on our data. Finally, training data inflation by means of data
augmentation is at best a secondary avenue in increasing classifier
performance compared to the overall network structure and data-set design.
Regarding the Maiju jumpsuit, our future research interests include the
expansion of the training data set to contain infants up to 14 months in age,
and in exploring the effect of unsupervised representation learning techniques
such as contrastive predictive coding [14] to the classification results.

## References

  * [1] M. Airaksinen, L. Juvela, P. Alku, and O. Räsänen.  Data augmentation strategies for neural network f0 estimation.  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6485–6489, 2019.
  * [2] M. Airaksinen, O. Räsänen, E. Ilén, T. Häyrinen, A. Kivi, V. Marchi, A. Gallen, S. Blom, A. Varhe, N. Kaartinen, L. Haataja, and S. Vanhatalo.  Automatic Posture and Movement Tracking of Infants with Wearable Movement Sensors.  Scientific Reports, 10(169), 2020.
  * [3] K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio.  On the properties of neural machine translation: Encoder–decoder approaches.  In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111. Association for Computational Linguistics, 2014.
  * [4] S. Ha and S. Choi.  Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors.  In 2016 International Joint Conference on Neural Networks (IJCNN), pages 381–388, July 2016.
  * [5] N. Y. Hammerla, J. M. Fisher, P. Andras, L. Rochester, R. Walker, and T. Plotz.  Pd disease state assessment in naturalistic environments using deep learning.  In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, page 1742–1748. AAAI Press, 2015.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  arXiv:1512.03385, 2015.
  * [7] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [8] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [9] M. A. Lobo, M. L. Hall, B. Greenspan, P. Rohloff, L. A. Prosser, and B. A. Smith.  Wearables for Pediatric Rehabilitation: How to Optimally Design and Use Products to Meet the Needs of Users.  Physical Therapy, 99(6):647–657, 06 2019.
  * [10] I. Novak, C. Morgan, L. Adde, J. Blackman, R. N. Boyd, J. Brunstrom-Hernandez, G. Cioni, D. Damiano, J. Darrah, A.-C. Eliasson, L. S. de Vries, C. Einspieler, M. Fahey, D. Fehlings, D. M. Ferriero, L. Fetters, S. Fiori, H. Forssberg, A. M. Gordon, S. Greaves, A. Guzzetta, M. Hadders-Algra, R. Harbourne, A. Kakooza-Mwesige, P. Karlsson, L. Krumlinde-Sundholm, B. Latal, A. Loughran-Fowlds, N. Maitre, S. McIntyre, G. Noritz, L. Pennington, D. M. Romeo, R. Shepherd, A. J. Spittle, M. Thornton, J. Valentine, K. Walker, R. White, and N. Badawi.  Early, Accurate Diagnosis and Early Intervention in Cerebral Palsy: Advances in Diagnosis and TreatmentEarly, Accurate Diagnosis and Early Intervention in Cerebral PalsyEarly, Accurate Diagnosis and Early Intervention in Cerebral Palsy.  JAMA Pediatrics, 171(9):897–907, 09 2017.
  * [11] C. Shorten and T. M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of Big Data, 6(1):60, 2019.
  * [12] T. T. Um, F. M. J. Pfister, D. Pichler, S. Endo, M. Lang, S. Hirche, U. Fietzek, and D. Kulić.  Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks.  In Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI ’17, page 216–220, New York, NY, USA, 2017. Association for Computing Machinery.
  * [13] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu.  Wavenet: A generative model for raw audio.  arXiv:1609.03499, 2016.
  * [14] A. van den Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding, 2018.
  * [15] P. Vepakomma, D. De, S. K. Das, and S. Bhansali.  A-wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities.  In 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), pages 1–6, 2015.
  * [16] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu.  Deep learning for sensor-based activity recognition: A survey.  Pattern Recognition Letters, 119:3 – 11, 2019.
  * [17] R. J. Williams and D. Zipser.  Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity, page 433–486.  L. Erlbaum Associates Inc., USA, 1995.

## Acknowledgment

The research was funded by Academy of Finland grants no. 314602, 314573,
314450, 335778, and 332017, as well as project grants from Lastentautien
tutkimussäätiö, Aivosäätiö and Juselius foundation.