# Dynamic Chunk Convolution for Unified Streaming and Non-Streaming Conformer
ASR

###### Abstract

Recently, there has been an increasing interest in unifying streaming and non-
streaming speech recognition models to reduce development, training and
deployment cost. The best-known approaches rely on either window-based or
dynamic chunk-based attention strategy and causal convolutions to minimize the
degradation due to streaming. However, the performance gap still remains
relatively large between non-streaming and a full-contextual model trained
independently. To address this, we propose a dynamic chunk-based convolution
replacing the causal convolution in a hybrid Connectionist Temporal
Classification (CTC)-Attention Conformer architecture. Additionally, we
demonstrate further improvements through initialization of weights from a
full-contextual model and parallelization of the convolution and self-
attention modules. We evaluate our models on the open-source Voxpopuli,
LibriSpeech and in-house conversational datasets. Overall, our proposed model
reduces the degradation of the streaming mode over the non-streaming full-
contextual model from 41.7% and 45.7% to 16.7% and 26.2% on the LibriSpeech
test-clean and test-other datasets respectively, while improving by a relative
15.5% WER over the previous state-of-the-art unified model.

Index Terms—  End-to-end speech recognition, Unified ASR, Streaming ASR,
Conformer

##  1 Introduction

End-to-end (E2E) automatic speech recognition (ASR) models such as attention-
based encoder-decoder (AED) [1, 2], CTC [3, 4] and Transducer [5, 6, 7] have
gained a lot of attention over the past decade due to their simplicity in the
integration of the pronunciation, language and acoustic models into a single
neural network. While state-of-the-art E2E models work remarkably well in a
non-streaming fashion, they suffer from degradation when operating in a
streaming manner as the requirement of transcribing text in real time poses an
extra challenge. Numerous works try to bridge the gap with non-streaming ASR
by training a model specific for a streaming ASR task [8, 9], focusing on
mitigating the trade-off between latency and accuracy. While some slight
improvements have been observed, the gap with models that take the full
acoustic sequence as input (a.k.a. full-contextual models) remains non-
negligible [10, 11].

In recent years, efforts have been made to unify streaming and non-streaming
into a single model [12, 13, 14, 15, 16, 17], which helps reduce development,
training and deployment cost. A commonly explored solution is to expose the
unified model to various contexts at training time thereby making the model
less susceptible to accuracy degradation at inference time under different
latency conditions. In [15], a dynamic chunk training technique is adopted
where the input is split into several fixed sized chunks and the audio frames
within each chunk attend on themselves and frames from all the previous
chunks. They vary the chunk size dynamically from 1 to the maximum utterance
length in the batch, so the trained model learns to predict with arbitrary
chunk size. [13] and [18] present a quite similar dynamic chunk-based
attention strategy, but other methods exists too. [19] for instance, first
processes input features with a streaming encoder before passing these to a
non-streaming encoder. A single decoder then learns to decode either using the
output of the streaming or the non-streaming encoder. [14] introduced dual-
mode ASR with shared weights for both streaming and full-context speech
recognition to further optimize the performance of streaming ASR. Similarly,
the Dual Causal/Non-causal (DCN) self-attention network proposed in [16]
processes two sequences of causal and non-causal frames in parallel and
prevents the overall context to grow beyond the look-ahead of a single layer.
The authors in [17] employ self-supervised pre-training with wav2vec 2.0 [20]
and fine-tune the model through dual-mode training.

In general, streaming and non-streaming ASR systems are trained independently
for optimized performance. These recent works on unified ASR are a great step
towards a single, easy-to-use solution regardless of the inference mode.
However, we identify two main shortcomings in the literature: First, the
performance gap between streaming and non-streaming of an unified model still
remains significant, especially when a Conformer encoder is used [13, 16].
Second, the gap between non-streaming and a full-contextual model enlarges
with the increase in the amount of training data [13, 15].

In this work, we propose a dynamic chunk convolution (DCConv), a non-causal
convolution with an improved training strategy for unified non-streaming and
streaming Conformers [21]. This builds further upon the dynamic chunk training
(DCT) from [15], in which the core idea is to divide the input into chunks
with a chunk size that gets dynamically generated at training time. The
difference lies in our novel convolution which better mimics the inference
conditions at training time while keeping a rich acoustic representation and
therefore results in superior performance. Besides the proposed DCConv, we
extend the original DCT with two other key contributions: a) we demonstrate a
better performance in both streaming and non-streaming when the model is fine-
tuned from a baseline full-contextual model; b) we further optimize the
streaming performance by parallelizing the convolution and self-attention
modules within each Conformer block. An extensive ablation study is performed
varying chunk size, overlapping chunk ratio and left context size. Empirical
evaluations measured on Voxpopuli showcase the efficacy of our proposed
approach in terms of the accuracy vs latency trade-off. Overall, the proposed
model achieves an average relative improvement of 15.5% WER over the previous
state-of-the-art [15] and obtained an absolute WER of 2.0% and 2.4% on the
LibriSpeech test-clean dataset in the non-streaming and streaming mode,
respectively.

Fig. 1: (a) An example of a chunk mask of size 4, left context size 8 and
sequence length 20. (b) An example of a window mask of right context size 3,
left context size 8 and sequence length 20.

##  2 Approach

###  2.1 Model architecture

We consider a joint CTC-Attention framework [22, 23] for training our unified
models. It consists of three components: a Shared Encoder, a CTC Decoder and
an Attention Decoder. For our experiments, we only use the CTC Decoder at
inference time and therefore we configure the Attention Decoder with a shallow
transformer [24]. For the Shared Encoder, we consider two variants: the
Conformer architecture [21] and a parallel Conformer [25]. The Conformer
architecture consists of a regular Conformer encoder block [21] in which the
convolution module appears after the multi-head self-attention (MSA). We
demonstrate that using a parallel Conformer (P-Conf) encoder block [25]
instead, which has the convolution and MSA existing next to each other, is
beneficial for the unified streaming scenario. The advantage of the P-Conf is
to capture both global and local context explicitly in the MSA and convolution
branch respectively. As opposed to the recently proposed Branchformer [25],
which also uses the parallel structure, we leverage the block for the
streaming application too. The P-Conf reduces the overall receptive field due
to its parallel nature while maintaining the same model capacity, resulting in
more robust streaming performance.

###  2.2 Dynamic Chunk Training for Self-Attention

For unified models to perform well, they must be exposed to both limited and
full context during their training. To accomplish this, [15] propose dynamic
chunk training (DCT) for self-attention layers. The DCT idea involves varying
the chunk size dynamically from 1 to the max utterance length for different
batches in training. This is achieved by applying a dynamic chunk mask to the
attention score matrix for each self-attention layer, which is illustrated in
Eq. 1:

$\displaystyle\text{Attn}(\bm{Q},\bm{K},\bm{V})$
$\displaystyle=\text{Softmax}(\textbf{Mask}(\bm{Q}\bm{K}^{T})/\sqrt{d})\bm{V}$
(1)

where $Q$, $K$, $V$ and $d$ denote the queries, keys, values and embedding
dimension respectively. Unlike the window mask (Fig. 1b), the chunk mask (Fig.
1a) strictly enforces the look-ahead size by setting the chunk size, while the
receptive field with window masking grows linearly with the stacking of more
layers. In this work, we randomly sample the chunk size between 8 (=320ms) and
32 (=1280ms) frames and the left context size between 0 and all left chunks,
so that the model becomes robust to numerous sizes at inference time.

###  2.3 Dynamic Chunk Convolution

The convolution operator is a key component of Conformer ASR models [21].
However, the conventional convolution results in significant accuracy
degradation due to the mode mismatch between training and inference, as shown
in Fig. 2a. Indeed, the chunk’s rightmost frames can see context from the next
chunk on their right during training. Whereas at inference, this right chunk
context is not available, causing a discrepancy. This inter-chunk correlation
is even more magnified when stacking more Conformer blocks.

One solution that is adopted in [15] consists of using a causal convolution,
as shown in Fig. 2b. The left-shifted convolution kernel restricts its span
from having access to any frames beyond the chunk’s right boundary. This still
leads to performance degradation though, as the lack of within-chunk future
context for the frame being processed results in a poorer acoustic
representation.

In this work, we therefore propose a novel non-causal dynamic chunk
convolution (Fig. 2c). As opposed to the conventional convolution (Fig. 2a),
the chunk convolution operator has no access to any future context beyond its
right boundary. This trick allows training to more closely match the streaming
inference setting where no future context beyond the right chunk boundary is
available either. As opposed to causal convolution (Fig. 2b), the DCConv chunk
has access to a limited within-chunk future context of the current frame. This
extra within-chunk future context results in more accurate acoustic modeling
and therefore better overall accuracy. The authors of [26] introduce a similar
non-causal convolution. Unlike their convolution though, ours caches the
output of the preceding chunk(s) and pads it to the current chunk, resulting
in a superior representation. The non-causal convolution of [26] is also used
in the Emformer [27] architecture for streaming use-cases with a fixed chunk
size, while ours is implemented in a Conformer architecture with DCT and is
applicable to both streaming and non-streaming ASR. These distinctions enable
our model to be utilized in a wider range of settings.

$\displaystyle\bm{X}^{i}_{C}$
$\displaystyle=\bm{X}_{[iC-L:(i+1)C]}=[\bm{X}_{[iC-L:iC]},\bm{X}_{[iC:(i+1)C]}]$
(2) $\displaystyle\bm{X}^{i^{\prime}}_{C}$
$\displaystyle=\text{Conv}(\bm{X}^{i}_{C})\rightarrow\bm{X}^{{}^{\prime}}=\text{Concat}(\bm{X}^{i^{\prime}}_{C[L:]})$
(3)

As shown in Eq. 2-3, we implement the DCConv by splitting the input sequence
$\bm{X}$ into chunks $\bm{X}^{i}_{C}$ where $i$ denotes the index of the chunk
within the sequence and $C$ the chunk size. Each chunk has a left context size
$L=(kernel\\_size-1)/2$. After the convolution is applied on every chunk, we
concatenate $\bm{X}^{i^{\prime}}_{C}$ from which we have removed the first $L$
output frames that correspond to the input left context. This DCConv operator
does not slow down the training since all the chunks are independent from each
other. Furthermore, we ensure to synchronize the size of both the chunk mask
for the self-attention layers and for the DCConv such that the overall look-
ahead size of the encoder is strictly set to the specified common size.

Fig. 2: Different types of convolutions: (a) regular convolution, (b) causal
convolution, and (c) chunk convolution, with example kernel size 5 and chunk
size 16. Table 1: Small- and Large-Scale experiments with different
architectures and training strategies. All streaming evaluations are done with
a 640ms chunk size, 50% overlapping, 1280ms left context and averaged encoder
latency of roughly 480ms.

|  | Relative WER | Absolute WER
---|---|---|---
|  | Conversational | Multi-accent | WSJ | Voxpopuli
Model | Training Strategy | Non-streaming | Streaming | Non-streaming | Streaming | Non-streaming | Streaming | Non-streaming | Streaming
Transformer | (A) Full-context | - | - | - | - | 7.1 | 9.7 | 13.9 | 18.2
(B) DCT | -11.0% | 10.2% | -4.2% | 6.9% | 7.8 | 8.9 | 14.7 | 17.1
Conformer | (C) Full-context | 16.0% | 11.6% | 10.4% | 10.9% | 6.1 | 8.5 | 12.5 | 16.1
(D) DCT | 14.0% | 14.3% | 8.9% | 12.6% | 6.1 | 7.6 | 12.4 | 15.6
(E) DCT w/ Causal Conv | 6.0% | 23.1% | 0.5% | 8.1% | 7 | 9.2 | 15 | 18.9
(F) DCT w/ DCConv | 9.0% | 27.2% | 2.6% | 18.6% | 6.5 | 7.3 | 13.1 | 14.2
(G) + Fine-tune | 16.0% | 29.3% | 9.9% | 21.9% | 6.4 | 7.2 | 12.4 | 14.0
P-Conformer | (H) Full-context | 16.0% | 14.3% | 10.4% | 12.6% | 6.3 | 8.3 | 12.4 | 16
(I) DCT w/ DCConv | 8.0% | 27.9% | 2.1% | 19.4% | 6.5 | 7.3 | 13.0 | 14.0
(J) + Fine-tune | 16.0% | 30.6% | 9.9% | 22.3% | 6.2 | 6.9 | 12.6 | 13.8
Conformer-Large | (K) Full-context | - | - | - | - | 4.5 | 6.2 | 9.2 | 12
(L) DCT w/ DCConv + Fine-tune | 1.9% | 22.9% | 0.0% | 11.0% | 4.6 | 5.6 | 9.1 | 10.5

###  2.4 Fine-tuning Baseline Full-Contextual Model

Lastly, we showcase that fine-tuning from a baseline full-contextual model
results in better overall performances. Instead of training a unified model
from scratch, we initialize the weights from a pre-trained full-contextual
model. This approach allows to leverage the non-streaming performance of the
full-contextual model. Simultaneously, the model will perform better in the
streaming mode too as it can transfer common speech recognition knowledge
gained from the non-streaming pre-training.

##  3 Experimental Settings

###  3.1 Datasets

Training data: We consider 3 different speech corpora varying in size for
training our models: A large-scale 50k+ hour English corpus and a small-scale
5k hour subset, sampled from in-house paired audio and text data. Both corpora
include audio files with a good mix of accents, speakers, sampling rates and
background noise. The third dataset is the open-source LibriSpeech [28]
corpus, for which we combine train-clean-100, train-clean-360 and train-
other-500 to have 960 hours of training data. These 3 data regimes are
representative of a wide range of end-to-end ASR systems for various speech
applications.

Benchmarking: For the LibriSpeech experiments in section 4.3, we evaluate our
models on test-clean and test-other. For the small- and large-scale
experiments in sections 4.1 and 4.2 respectively, we use the following test
sets: (1) Conversational: A 10+ hour in-house dataset with utterances
resembling user inputs to goal oriented conversational dialog systems. The
average utterance length is roughly 10 words; (2) Multi-accent: A 100+ hour
in-house long-form audio dataset, composed of 12 different accents spoken
across the US. The average utterance length is roughly 16 words after
segmentation; (3) Wall Street Journal (WSJ): We use WSJ’s eval_test92 [29],
prepared using Kaldi’s [30] WSJ recipe. The dataset is 0.7h long. The average
utterance length is 16 words; (4) Voxpopuli [31]: We use the English test
partition, which is 4.9h long. The average utterance length is 24 words. We
report absolute word error rate (WER) on WSJ and Voxpopuli and relative WER
(WERR) on in-house datasets.

###  3.2 Experiment Setup

For training, we use the AED architecture with a Conformer as the encoder, and
a shallow single-layer transformer [24] as the attention-based decoder. For
LibriSpeech experiments, we use a Conformer-12x512x8, which consists of 12
encoder layers with 512 feature dimensions and 8 self-attention heads. We
train a 24-layered transformer-based neural LM on the librispeech-train
dataset to use for rescoring. For the small-scale experiments we use a
Conformer-16x512x4, whereas for the large-scale experiments we use a
Conformer-20x512x8. The kernel size of our convolution modules is 31. We
optimise our model via the hybrid CTC and attention losses. All of our models
are trained using ESPNet [32], with the Adam optimizer [33] and a warm-up
learning rate scheduler.

For front-end, we use 80 dimensional log-mel features and SpecAugment [34] to
perform data augmentation. The BPE embedding is 1024 and 2048 for the small-
and large-scale experiments respectively. We train a 4-gram LM on the training
text for shallow fusion. For evaluation, we discard the attention-based
decoder and only use the CTC decoder to generate outputs with a CTC prefix
beam search and beam size of 50. A CTC decoder optimises the real-time factor
(RTF) compared to the attention-based decoder, as the latter is non-
autoregressive and also needs triggered attention [35] for streaming inference
and is therefore slower. We opt for the CTC decoder as ensuring a low RTF is
key for streaming applications.

##  4 Results

###  4.1 Ablation Study on Small-Scale Model

In the next subsections, we start with discussing the different contributions
of our work by analyzing the small-scale results in Table 1.

Fig. 3: Ablation study on small-scale models on how the chunk size,
overlapping ratio and left context size affect the streaming decoding WER on
the Voxpopuli testset. (a) Different chunk sizes with 50% overlapping and
1280ms left context. (b) Different overlapping ratios with 640ms chunk size
and 1280ms left context. (c) Different left context sizes with 640ms chunk
size and 50% overlapping chunk decoding.

####  4.1.1 DCConv vs Causal and Normal Convolutions

We demonstrate the effectiveness of our novel DCConv by performing an ablation
study on three different models: a DCT model with normal convolution (D), a
DCT model with causal convolution (E), and a DCT model with our own DCConv
(F). Our model outperforms the two baselines on every dataset in the streaming
mode. In the non-streaming application on the other hand, our model always
beats the DCT model with causal convolution, but does degrade the model with
regular convolution. Devising our convolution in such a way that within-chunk
future context is used for a more informative acoustic representation (as
opposed to (D)), while refraining of using outside-chunk future context to
match the training and inference modes more closely (as opposed to (E)), is
therefore empirically shown to be advantageous in most cases.

####  4.1.2 Fine-tuning from Full-Contextual Model

Fine-tuning (G) from a pre-trained full-contextual model instead of training a
DCConv model from scratch (F) leads to improvements in both the non-streaming
and streaming mode for every single dataset. We observe an average relative
WER improvement of 5.5% and 2.4% in the non-streaming and streaming modes
respectively. Furthermore, with the exception of the WSJ dataset, we now
always outperform the regular convolution model in the non-streaming mode.
Fine-tuning helps as the model relies on previously gained knowledge from a
pre-trained model instead of learning from scratch. It maintains and even
improves the non-streaming performance of the full-contextual model, while
boosting the streaming performance as it leverages previously learned speech
recognition knowledge common to both streaming modes.

####  4.1.3 P-Conf vs Conformer

Our experiments indicate that using a P-Conf instead of a regular Conformer is
beneficial in the streaming mode, while it shows similar results in non-
streaming. This holds for the full-contextual (H), DCConv (I) and fine-tuned
DCConv (J) models, where using a P-Conf (J) leads to an relative WER
improvement of 1.9%, 0.5%, 4.2% and 1.4% respectively across the 4 datasets
over Conformer (G). We hypothesize that the lower receptive field of the
P-Conf as a result of the parallel MSA and convolution model during training
makes the model more robust in the streaming mode, as it more closely matches
the inference conditions.

####  4.1.4 Ablation Study with Different Streaming Parameters

We take a closer look at the latency vs accuracy trade-off impact with
different model parameters that can be controlled during streaming inference.
In Fig. 3, we run an ablation study on the DCT hyper-parameters. Fig. 3a
demonstrates improvement in WER with the increase in chunk size. Likewise,
Fig. 3b and Fig. 3c reveal better WER performance with more overlap between
successive chunks and greater left context size. All are the result of a wider
and therefore superior acoustic representation. However, these improvements go
at the expense of speed due to the latency-accuracy trade-off. Depending on
the use-case we can then select one or the other setting. More importantly, we
observe for all settings that our fine-tuned DCConv model performs the best,
illustrating the robustness of our approach. In our subsequent experiments, we
opt for a 640ms chunk with a 1280ms left context in order to keep those values
low to mimic a real-life streaming setting. Furthermore, we stick to a 50%
overlapping ratio as this ratio performs only slightly worse than the 75%
ratio but provides better latency.

###  4.2 Result with Large-Scale Model

In Table 1, we also compare a fine-tuned DCConv Conformer model (L) to a
baseline full-contextual model (K) on a large-scale dataset. The results show
no degradation (except for WSJ), and even minor improvements, in the non-
streaming mode. This is despite the fact that our model was trained in a
unified fashion. In the streaming mode, we observe an average WERR improvement
of 14.0% across all datasets. This validates the gains of our suggested
contributions.

###  4.3 Results on LibriSpeech

Table 2: LibriSpeech experiment. Chunk size of 640ms and left context size of
1280ms for 50% overlapping chunk streaming.

Model | test-clean | test-other
---|---|---
| Non-streaming | Streaming | Non-streaming | Streaming
(A) Conformer (Full-context) | 2.1 | 3.6 | 5.1 | 9.4
(B) + DCT w/ causal conv | 2.6 | 2.9 | 5.8 | 6.8
(C) + DCT w/ DCConv | 2.3 | 2.6 | 5.4 | 6.6
(D) + Fine-tune | 2.0 | 2.5 | 4.8 | 6.6
(E) P-Conformer (Full-context) | 2.1 | 3.4 | 4.9 | 9.1
(F) + DCT w/ DCConv + Fine-tune | 2.0 | 2.4 | 4.8 | 6.5

In Table 2, we illustrate the performance of our proposed approach when
trained and tested on the widely used public LibriSpeech dataset. First, we
compare our DCConv model (C) to a full-contextual model trained without DCT
(A) and a DCT model with a regular causal convolution (B) in a non-streaming
and a 50% overlapping streaming mode. We observe a 28.9% WERR improvement
compared to the full-contextual model in the streaming mode, emphasizing the
utility of DCT training. Additionally, we notice an average 7.9% WERR
improvement compared to the DCT model with regular causal convolution for both
streaming modes, proving the effectiveness of our devised convolution.
Furthermore, we demonstrate that fine-tuning (D) instead of training a DCConv
model from scratch is especially helpful if you wish to keep a high non-
streaming performance of your unified model. It even outperforms the full-
contextual model by a WERR improvement of 5.3% in the non-streaming mode,
while keeping the overlapping streaming performance almost intact. Lastly, we
observe a minor 2.8% relative WER improvement when using a P-Conf (F) instead
of the conventional one (D) in the streaming mode.

Overall, compared to the full-contextual Conformer model (A), our final fine-
tuned DCConv P-Conf model (F) improves the WER by 32.1% in the streaming mode
and reduces the degradation of that mode over the non-streaming full-
contextual model from 41.7% and 45.7% to 16.7% and 26.2% on the test-clean and
test-other datasets respectively. Moreover, we further improve on the state-
of-the-art model (B) that reduces this streaming gap too by an average 15.5%
WERR across the 4 settings.

##  5 Conclusion

In this work, we propose a novel dynamic chunk convolution that further
improves the existing dynamic chunk training. We achieve this as our
convolution better mimics the inference conditions at training time, while
keeping a rich acoustic representation. Additionally, we introduce a fine-
tuning mechanism and a parallel Conformer block for the unified ASR setting.
Our results demonstrate that our unified streaming model closes and even
exceeds the gap with a full-contextual model operating in a non-streaming
mode, while also showcasing improvements in the streaming mode under different
latency constraints. Overall, we outperform the previous state-of-the-art by
an average 15.5% WERR across the LibriSpeech datasets.

## References

  * [1] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in Proc. ICASSP, 2016, pp. 4960–4964.
  * [2] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio,  “Attention-based models for speech recognition,”  Advances in neural information processing systems, vol. 28, 2015\.
  * [3] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist Temporal Classification: Labelling unsegmented sequence data with Recurrent Neural Networks,”  in Proc. ICML, 2006, pp. 369–376.
  * [4] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.,  “Deep speech 2: End-to-end speech recognition in English and Mandarin,”  in Proc. ICML, 2016, pp. 173–182.
  * [5] Alex Graves,  “Sequence transduction with recurrent neural networks,”  arXiv preprint arXiv:1211.3711, 2012.
  * [6] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton,  “Speech recognition with deep recurrent neural networks,”  in Proc. ICASSP, 2013, pp. 6645–6649.
  * [7] Siddharth Dalmia, Yuzong Liu, Srikanth Ronanki, and Katrin Kirchhoff,  “Transformer-transducers for code-switched speech recognition,”  in Proc. ICASSP, 2021, pp. 5859–5863.
  * [8] Tara N Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, et al.,  “Two-pass end-to-end speech recognition,”  arXiv preprint arXiv:1908.10992, 2019.
  * [9] Chung-Cheng Chiu and Colin Raffel,  “Monotonic chunkwise attention,”  arXiv preprint arXiv:1712.05382, 2017.
  * [10] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar,  “Transformer Transducer: A streamable speech recognition model with Transformer encoders and RNN-T loss,”  in Proc. ICASSP.
  * [11] Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen,  “Conv-Transformer Transducer: Low latency, low frame rate, streamable end-to-end speech recognition,”  arXiv preprint arXiv:2008.05750, 2020.
  * [12] Anshuman Tripathi, Jaeyoung Kim, Qian Zhang, Han Lu, and Hasim Sak,  “Transformer transducer: One model unifying streaming and non-streaming speech recognition,”  arXiv preprint arXiv:2010.03192, 2020.
  * [13] Binbin Zhang, Di Wu, Zhuoyuan Yao, Xiong Wang, Fan Yu, Chao Yang, Liyong Guo, Yaguang Hu, Lei Xie, and Xin Lei,  “Unified streaming and non-streaming two-pass end-to-end model for speech recognition,”  arXiv preprint arXiv:2012.05481, 2020.
  * [14] Jiahui Yu, Wei Han, Anmol Gulati, Chung-Cheng Chiu, Bo Li, Tara N Sainath, Yonghui Wu, and Ruoming Pang,  “Dual-mode ASR: Unify and improve streaming ASR with full-context modeling,”  arXiv preprint arXiv:2010.06030, 2020.
  * [15] Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei,  “WeNet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit,”  arXiv preprint arXiv:2102.01547, 2021.
  * [16] Niko Moritz, Takaaki Hori, and Jonathan Le Roux,  “Dual causal/non-causal self-attention for streaming end-to-end speech recognition,”  in Proc. Interspeech, 2021, pp. 1822–1826.
  * [17] Chunxi Liu, Yuan Shangguan, Haichuan Yang, Yangyang Shi, Raghuraman Krishnamoorthi, and Ozlem Kalinli,  “Learning a dual-mode speech recognition model via self-pruning,”  arXiv preprint arXiv:2207.11906, 2022.
  * [18] Kwangyoun Kim, Felix Wu, Prashant Sridhar, Kyu J Han, and Shinji Watanabe,  “Multi-mode Transformer Transducer with Stochastic Future Context,”  arXiv preprint arXiv:2106.09760, 2021.
  * [19] Arun Narayanan, Tara N Sainath, Ruoming Pang, Jiahui Yu, Chung-Cheng Chiu, Rohit Prabhavalkar, Ehsan Variani, and Trevor Strohman,  “Cascaded encoders for unifying streaming and non-streaming ASR,”  in Proc. ICASSP.
  * [20] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [21] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented Transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [22] Suyoun Kim, Takaaki Hori, and Shinji Watanabe,  “Joint CTC-attention based end-to-end speech recognition using multi-task learning,”  in Proc. ICASSP, 2017, pp. 4835–4839.
  * [23] Saket Dingliwal, Monica Sunkara, Srikanth Ronanki, Jeff Farris, Katrin Kirchhoff, and Sravan Bodapati,  “Personalization of CTC speech recognition models,”  in Proc. IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 302–309.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [25] Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe,  “Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding,”  in Proc. ICML, 2022, pp. 17627–17643.
  * [26] Yangyang Shi, Chunyang Wu, Dilin Wang, Alex Xiao, Jay Mahadeokar, Xiaohui Zhang, Chunxi Liu, Ke Li, Yuan Shangguan, Varun Nagaraja, et al.,  “Streaming transformer transducer based speech recognition using non-causal convolution,”  in Proc. ICASSP, 2022, pp. 8277–8281.
  * [27] Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer,  “Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition,”  in Proc. ICASSP, 2021, pp. 6783–6787.
  * [28] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in Proc. ICASSP, 2015, pp. 5206–5210.
  * [29] John Garofolo, David Graff, Doug Paul, and David Pallett,  “CSR-I (WSJ0) Complete LDC93S6A,”  Web Download. Philadelphia: Linguistic Data Consortium, vol. 83, 1993.
  * [30] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.,  “The Kaldi speech recognition toolkit,”  in Proc. IEEE workshop on automatic speech recognition and understanding, 2011.
  * [31] Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux,  “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,”  arXiv preprint arXiv:2101.00390, 2021.
  * [32] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al.,  “ESPNet: End-to-end speech processing toolkit,”  arXiv preprint arXiv:1804.00015, 2018.
  * [33] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [34] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [35] Niko Moritz, Takaaki Hori, and Jonathan Le Roux,  “Triggered attention for end-to-end speech recognition,”  in Proc. ICASSP, 2019, pp. 5666–5670.


normalsize[table]font=normalsize

# Human Action Recognition Using Deep Multilevel Multimodal ($M^{2}$) Fusion
of Depth and Inertial Sensors

Zeeshan Ahmad, Student Member, IEEE, Naimul Khan, Member, IEEE

###### Abstract

Multimodal fusion frameworks for Human Action Recognition (HAR) using depth
and inertial sensor data have been proposed over the years. In most of the
existing works, fusion is performed at a single level (feature level or
decision level), missing the opportunity to fuse rich mid-level features
necessary for better classification. To address this shortcoming, in this
paper, we propose three novel deep multilevel multimodal ($M^{2}$) fusion
frameworks to capitalize on different fusion strategies at various stages and
to leverage the superiority of multilevel fusion. At input, we transform the
depth data into depth images called sequential front view images (SFIs) and
inertial sensor data into signal images. Each input modality, depth and
inertial, is further made multimodal by taking convolution with the Prewitt
filter. Creating “modality within modality” enables further complementary and
discriminative feature extraction through Convolutional Neural Networks
(CNNs). CNNs are trained on input images of each modality to learn low-level,
high-level and complex features. Learned features are extracted and fused at
different stages of the proposed frameworks to combine discriminative and
complementary information. These highly informative features are served as
input to a multi-class Support Vector Machine (SVM). We evaluate the proposed
frameworks on three publicly available multimodal HAR datasets, namely, UTD
Multimodal Human Action Dataset (MHAD), Berkeley MHAD, and UTD-MHAD Kinect V2.
Experimental results show the supremacy of the proposed fusion frameworks over
existing methods.

###### Index Terms:

Canonical correlation analysis, fusion of depth and inertial sensors, human
action recognition,, multimodal fusion.

##  I Introduction

HUMAN Action Recognition (HAR) has been a core area of research for the
multimedia community due to its applications in various fields, including
human computer interaction [1], healthcare [2], sports [3],visual surveillance
[4], military, robotics and gaming.††© 2019 IEEE. Personal use of this
material is permitted. Permission from IEEE must be obtained for all other
uses, in any current or future media, including reprinting/republishing this
material for advertising or promotional purposes, creating new collective
works, for resale or redistribution to servers or lists, or reuse of any
copyrighted component of this work in other works.

Before the resurgence of neural networks, especially the deep learning models,
conventional methods for HAR were statistical methods, where the focus was on
designing hand-crafted features. These hand-crafted methods have two
disadvantages: 1) requirement of domain knowledge about the data [5]; 2)
capturing only a subset of the features; resulting in difficulty to generalize
for unseen data. The exemplary performance of deep learning in classification
tasks [6] has diverted the attention of researchers from hand-crafted methods
to deep learning models. CNN in particular has gained significant attention in
computer vision and machine learning due to its strong ability to
automatically learn invariant and hierarchical features directly from the
images.

Low cost sensors including Kinect depth cameras, wearable sensors and
availability of smartphones have been very beneficial for HAR. Kinect depth
cameras provide 3D action data, require less hardware resources and are less
sensitive to lighting changes and clutter as compared to RGB cameras [7].
However there are limitations associated with depth images like view point
variation, noise during image acquisition and constrained space defined by
camera orientation. These shortcomings can be alleviated by utilizing wearable
inertial sensor for HAR such as accelerometer and gyroscope [8]. Wearable
sensors provide multivariate time series data in terms of 3-axis accelerations
from accelerometers and three axis angular velocities from gyroscopes. These
sensors provide data at a high sampling rate and can work in dark and
unbounded environment. Like depth cameras, wearable sensors have limitations
such as sensor drift and intrusiveness, as the humans have to physically wear
them [9].

Multimodal fusion can alleviate the shortcomings of both types of sensors, and
thus improve the performance of HAR. Although multimodal HAR has progressed
significantly over the past few years, the current works fall short of optimal
performance when combining multiple modalities. A major difficulty is in
deciding at which level of information fusion: early fusion, feature level
fusion and decision/late fusion, the modalities should be fused [10].

The purpose of multimodal fusion is to obtain complementary information from
modalities to perform the analysis task accurately. In multimodal fusion the
major consideration is to find the optimal instance or stage to fuse the
modalities. Based on this philosophy, the commonly used strategies are data
level or early fusion, feature level or intermediate fusion and decision level
or late fusion [11]. The emergence of deep learning has resulted in some new
concepts for fusion. Since feature extraction in deep learning models,
especially the CNN, can be performed at any layer, feature level fusion for
deep networks can be further divided into early feature level fusion (fusion
between features of convolutional layers) and late feature level fusion
(fusion between features of fully connected layers) depending upon the layers
of the deep network from which the features are extracted.

Feature level fusion is the most widely used approach for integrating
information in deep learning models. The greatest benefit of feature level
fusion is that it utilizes the correlation among the modalities at an early
stage. Furthermore, only one classifier is required to perform a task, making
the training process less tedious. However, a notable limit of feature level
fusion is time synchronization, since the data in various modalities are
captured at different rates and formats [12]. The other broadly used fusion
tactic is decision level. The significant advantage of this technique is that
it allows us to explicitly examine each modality and thus the chance of
dominance of one modality over the other is greatly reduced. Furthermore,
decision level fusion offers scalability in terms of modality [13], since
adding a new modality is relatively easier. The notable issue with decision
level fusion is the use of more than one classifier. This makes the task time
consuming. Hybrid multimodal fusion has been exercised by investigators to cop
up the shortcomings of feature level and decision level fusion [14]. However,
there is no single fusion method which can be considered as an ultimate
solution yet. Most current works either do feature level fusion or decision
fusion, missing the opportunity of fusing rich mid-level feature
representations that are available in a CNN-based architecture.

To address the aforementioned deficiencies and to exploit different fusion
strategies, in this paper, we present three novel deep multilevel multimodal
($M^{2}$) fusion frameworks for HAR. In each distinct fusion framework, we
applied multilevel fusion to counter the short comings of single stage fusion
so that an optimal framework can be reached methodically. The key
contributions of the presented work are:

  1. 1.

We propose three unique deep $M^{2}$ fusion frameworks : “Deep multistage
feature fusion framework”, “Deep hybrid fusion framework” and “Computationally
efficient fusion framework”. The purpose of presenting three deep fusion
frameworks is to get full advantage of multimodal fusion by applying multiple
fusion strategies to find a robust framework for HAR. We show that through
careful execution of multilevel multimodal ($M^{2}$) fusion, all three fusion
frameworks can be tuned to provide outstanding results for depth-inertial HAR.

  2. 2.

Inspired from the outstanding performance of CNN on image classification task
[6], we transformed both input datasets, depth and inertial, into depth and
signal images respectively. To extract more discriminative and complementary
features, we increased the number of modalities by taking convolution of SFIs
and signal images with the Prewitt filter. Converting input data to images and
generating additional modality by Prewitt filtering enables extraction of
different feature types (e.g. edges, curves, and higher-level abstractions)
with CNN that are not possible with 1D temporal data [15].

The rest of the paper is organized as follows. Section II describes the
related works on HAR using conventional and deep learning-based fusion.
Section III provides technical details of our signal processing and image
conversion ideas, and the proposed deep $M^{2}$ fusion frameworks. In Section
IV, we provide detailed experimental analysis, where the two aforementioned
contributions are analyzed in detail through ablation studies and comparison
with state-of-the-art models. Section V concludes the paper.

##  II Related Work

Since HAR is a very active research field with many research directions, for
brevity, we only discuss existing methods for HAR that attempts at fusing
different modalities.

In [16] the accuracy of HAR is improved by fusing features extracted from
depth and inertial sensor data and using collaborative representation
classifier. Improved accuracy results were achieved due to complementary
aspect of data from both modalities. A decision level fusion is performed
between depth camera data and wearable sensor data to increase the
capabilities of robots to recognize human actions in [17]. An efficient real-
time human action recognition system is developed in [18] using decision level
fusion of depth and inertial sensor data. Depth and inertial data is
effectively merged in [19] to train a hidden Markov model for improving
accuracy and robustness of hand gesture recognition. In [20], a
computationally efficient real-time detection and recognition approach is
presented to identify actions in the smart TV application from continuous
action streams using continuous integration of information obtained from depth
and inertial sensor data. A method for bilateral gait segmentation is proposed
in [21] by multimodal fusion of features obtained from thigh mounted inertial
sensor and depth sensor with the contralateral leg in its field of view. The
proposed method can be used to make lower limb assistive devices for patients
with walking impairments. In [22], novel and robust unsupervised method based
on fusion of depth and inertial sensor data for HAR is proposed. The proposed
Multiview Cauchy Estimator Feature Embedding (MCEFE) method is capable of
finding the optimal unified space and projection matrices by minimizing
empirical risk through the Cauchy estimator. A comprehensive survey on fusion
of depth and inertial sensors is provided in [9] where the recent success of
the fusion and future challenges and trends are discussed in details.

Due to the recent popularity of deep neural networks (DNN) in multimedia
applications, several deep learning based fusion frameworks for HAR has
recently been presented. In [23] a supervised deep multimodal fusion framework
for process monitoring and verification in the medical and healthcare fields
is presented that depends on simultaneous processing of motion data acquired
with wearable sensors and video data acquired with body-mounted camera.
Authors in [24] proposed DNN based fusion of images and inertial data for
improving the performance of human action recognition. Two CNNs were used to
extract features from images and inertial sensors and the fused fearures were
used to train an RNN classifier.

Deep learning based fusion methods for HAR using depth and inertial sensors
have been presented in [25], [26], [27] and [28]. In [25], CNN is used to
extract features from depth images while recurrent neural netwrok (RNN) is
used to capture features from inertial sensor data. Finally, a decision level
fusion is performed on extracted features to improve the accuracy of HAR. In
[26] deep learning based fusion system based on fusing the depth and inertial
data is presented which is capable of detecting and recognizing actions of
interest from continuous action streams. CNN is used to extract features from
depth images, and a combination of CNN and long short-term memory (LSTM)
network is utilized for inertial signals. First, the segmentation is performed
on each sensing modality and then actions of interest were detected. Finally,
the decision level fusion is performed for recognition. In [27], CNNs are used
to extract features from depth images obtained from depth sensor and from
signal images obtained from inertial data. Finally, a concatenation fusion is
performed between the two modalities. A CNN based sensor fusion system is
developed in [28] to detect and monitor transition movements between body
states as well as falls in healthcare applications. Fusion between the
modalities is carried out by accumulating the scores of fully connected layers
of CNNs.

The major shortcoming in existing deep learning based fusion methods for HAR
using depth and inertial sensors is that the fusion is performed at a single
level or stage, either feature level or decision level, thus failing to map
the true semantic information from data to classifier. Since deep learning
models allow us to extract features at all levels of their structure, we get
rich multilevel features comprised of low level, mid-level and high level
features. The existing methods do not take advantage of this rich multi-level
information, leaving room for improvement in the state-of-the-art of deep
learning-based visual-inertial action recognition.

To adress all the shortcomings of the related work, in this paper, we
introduce three novel deep multilevel multimodal ($M^{2}$) fusion frameworks
for improving the performance of HAR using depth and inertial sensor data. In
all three fusion frameworks, fusion is performed at multiple stages to
alleviate the deficiencies of the existing methods. The three ($M^{2}$) fusion
frameworks are explained in detail in section III.

The proposed frameworks are an extension of our recent work [27], where a CNN-
based fusion framework is presented, depth and inertial sensor data is
transformed into images and then CNNs are employed to extract features from
the transformed images. The depth and image features are simply concatenated
(feature level fusion) and served as input to train a multiclass SVM
classifier. We extend the work here by proposing the novel $M^{2}$ fusion
frameworks that perform multilevel multlimodal fusion.

##  III Proposed Method

In this section, we first describe the common components of our frameworks;
converting input signals to images, creating modality within modality, and
fusion technique. Then, the three novel deep multilevel multimodal ($M^{2}$)
fusion frameworks are presented.

Figure 1: Five samples of Sequential Front View Images of Two hand Front Clap
Figure 2: Formation of Signal Image from Inertial sensor data Figure 3: Signal
Images of three different actions

###  III-A Formation of Sequential Front View Images

The depth data provides 3D information, therefore we can generate the front,
top and side view of depth motion maps. We experimentally observe that only
front view information is enough to recognize the actions as the fusion of
front view with other views doesnot significantly increase the recognition
accuracy as shown in Table VI, but only increase the computational cost. The
supplementary information is provided by the inertial dataset. Thus we convert
the depth sequences into images called Sequential Front view Images (SFI) as
shown in Fig. 1. By using only one view in the SFIs, we are reducing the
computational cost. These images are similar to the motion energy images and
motion history images introduced in [29]. These SFIs provide cumulative
information about the action from start to completion.

###  III-B Formation of Signal Images

Inertial sensors generate data in the form of multivariate time series. In our
datasets, we have six sequences of signals : three accelerometer and three
angular velocity sequences obtained from accelerometers and gyroscopes
respectively.

We converted six sequences into 2D virtual images called signal images based
on the algorithm in [30]. The conversion of time series data to signal image
is shown in Fig. 2. Signal image is obtained through row-by-row stacking of
given six signal sequences in such a way that each sequence appears alongside
to every other sequence. The signal images are formed by taking advantage of
the temporal correlation among the signals.

Row wise stacking of six sequences has the following order.

123456135246142536152616

Where the numbers 1 to 6 represent the sequence numbers in a raw signal. Order
of the sequences clearly shows that every sequence neighbors every other
sequence to make a signal image. Thus the final width of signal image becomes
24.

For deciding the length of the signal image we made use of sampling rate of
datasets which is 50Hz for our two datasets. Therefore, to capture granular
motion accurately, the length of the signal image is finalized as 52,
resulting in a final image size of 24 x 52. These signal images are shown in
Fig. 3.

###  III-C Modality within modality

We create a modality within each input modality by convolving images of each
input modality with the Prewitt filter. Prewitt filters are edge detectors and
are simple to implement [31]. The filters deployed by CNN are trainable and
are specifically tuned to a dataset through a long training process. The
purpose of applying Prewitt filter was to create a generic additional modality
for any dataset that can be created efficiently without training, before we
feed multiple modalities to our proposed networks. This can be thought of as
an additional step of pre-processing to create an additional modality. Our
purpose is to show that creating an additional modality can help extracting
complementary and discriminative features. While we used Prewitt filter to
demonstrate our multimodal networks, we believe other type of filters can be
utilized too. Creating edge oriented modalities for both depth and signal
images using Prewitt filter have been experimentally proved significant as
shown in Table IV. These modalities allow CNN to extract further complementary
and discriminative features and thus the availability of rich features due to
these created modalities help classifiers to perform its task efficiently and
accurately.

We apply the following 3-by-3 Prewitt filter that emphasizes horizontal edges.

$h=\begin{bmatrix}1&1&1\\\ 0&0&0\\\ -1&-1&-1\par\end{bmatrix}$

Prewitt filtered sequential front view images and signal images are shown in
Fig. 4 and 5 respectively.

Figure 4: Prewitt Filtered Sequential Front view Images of Two Hand Front Clap
Figure 5: Prewitt Filtered Signal Images of three different actions

###  III-D Canonical Correlation based Fusion (CCF)

Canonical correlation analysis (CCA) is effective and robust multivariate
statistical method for finding the relationship between two sets of variables.

Let $X\in\mathbb{R}^{p\times n}$ and $Y\in\mathbb{R}^{q\times n}$ represents
the feature matrices from two modalities, where $p$ and $q$ are the dimensions
of the first and second feature set respectively and $n$ are the training
samples in each modality. let $\Sigma_{xx}\in\mathbb{R}^{p\times p}$ and
$\Sigma_{yy}\in\mathbb{R}^{q\times q}$ denote the within set covariance
matrices of $X$ and $Y$ respectively and $\Sigma_{xy}\in\mathbb{R}^{p\times
q}$ denotes the between set covariance matrix for $X$ and $Y$ and
$\Sigma_{yx}=\Sigma_{xy}^{T}$. The overall augmented covariance matrix of size
$(p+q)\times(p+q)$ is given by

$cov(X,Y)=\begin{bmatrix}\Sigma_{xx}&\Sigma_{xy}\\\
\Sigma_{yx}&\Sigma_{yy}\par\end{bmatrix}$ (1)

The purpose of CCA is to find the linear combination $X^{\prime}=AX$ and
$Y^{\prime}=BY$ such that the maximum pairwise correlation between the
modalities could be achieved. Matrices $A$ and $B$ are called transformation
matrices for $X$ and $Y$ respectively. The correlation between $X^{\prime}$
and $Y^{\prime}$ is given by

$corr(X^{\prime},Y^{\prime})=\frac{cov(X^{\prime},Y^{\prime})}{var(X^{\prime}).var(Y^{\prime})}$
(2)

where $cov(X^{\prime},Y^{\prime})=A^{T}\Sigma_{xy}B$,
$var(X^{\prime})=A^{T}\Sigma_{xx}A$ and $var(Y^{\prime})=B^{T}\Sigma_{yy}B$.
$X^{\prime}$ and $Y^{\prime}$ are known as canonical variates.

Lagrange’s Optimization method is used to maximize the covariance between
$X^{\prime}$ and $Y^{\prime}$ subject to the constraint that the variance of
$X^{\prime}$ and variance of $Y^{\prime}$ is equal to unity [32].

$var(X^{\prime})=var(Y^{\prime})=1$

On the transformed feature vectors, canonical correlation based fusion (CCF)
is performed by adding the transformed feature vector. Mathematically this
addition is written as

$Z=X^{\prime}+Y^{\prime}=A^{T}X+B^{T}Y$

Figure 6: CNN Architecture for Signal Image. Consists of two convolutional
layers, two pooling layers, and a fully connected layer. The first
convolutional layer has 50 kernels of size 5x5, followed by pooling layer of
size 2x2 and stride 2. The output of the first pooling layer is the input of
the second convolutional layer which has 100 kernels followed by 2x2 pooling
layer with stride 2.

###  III-E $M^{2}$ Fusion frameworks

Finally we present the fusion frameworks utilizing the aforementioned common
components.

Figure 7: Deep Multistage Feature Fusion Framework Figure 8: Deep Hybrid
Fusion Framework Figure 9: Computationally Efficient Fusion Framework.

####  III-E1 Deep Multistage Feature Fusion Framework

Our first network, the deep multistage feature fusion framework is built upon
the philosophy of processing the separate modalities through separate CNNs
first, which act as feature extractors. Then, a two stage feature fusion is
performed, feature concatenation, followed by canonical correlation based
fusion (CCF). The architecture of framework is shown in Fig. 7.

Alexnet (CNN based model) [6] is used to extract features from SFIs and
Prewitt filtered SFIs. Another smaller CNN, whose architecture is shown in
Fig. 6, is used to extract features from signal images and Prewitt filtered
signal images as shown in Fig. 7. We extracted learned features from the
second last fully connected layer of both Alexnets and performed
concatenation. Similarly, learned features extracted from the first fully
connected layer of the CNNs for signal images are also concatenated. The final
fusion with CCF is performed between the concatenated features of both
modalities as shown in Fig.7. For CCF, we downsampled the concatenated
features of depth modality using bicubic interpolation, so that they are the
same size as the concatenated features of inertial modality to avoid rank
deficiency during calculations of transformation matrices. Fusing the two
concatenated layers using CCF results in highly discriminative features
compared to simple concatenation [33]. This fused data is served as input to
multiclass SVM for performing recognition task.

We used SVM as a classifier since we experimentally proved in our previous
work [27] that SVM performs better than softmax, which is typically built into
any CNN framework. Softmax classifier reduces the crossentropy function while
SVM employs a margin based function. Multiclass SVM classifies data by
locating the hyperplane at a position where all data points are classified
correctly. Thus SVM determines the maximum margin among the data points of
various classes. The more rigorous nature of classification is likely the
reason why SVM performs better than softmax. Since we are doing multilevel
fusion, therefore we have to extract and fuse features at more than one stages
and hence we require an external classifier, thus classification within the
CNN framework is meaningless and end-to-end deep learning model cannot be
promoted.

####  III-E2 Deep Hybrid Fusion Framework

The architecture of the deep hybrid fusion framework is shown in Fig. 8.

This framework uses hybrid fusion, combination of feature level and decision
level fusion, as compared to deep multistage fusion framework that used
feature level fusion at two stages. Furthermore two classifiers are used to
generate scores for each modality to perform decision level fusion.

At the input of this framework, input modalities are converted into images and
features are extracted with Alexnet and CNN and fused by concatenation in same
way as that of multistage fusion framework explained in section III-E1. The
two concatenated feature vectors are separately inputted to two SVM classifers
as shown in Fig. 8. The final fusion is a decision level fusion between the
scores generated by two classifiers. We use maximum fusion strategy to
generate final classification results.

####  III-E3 Computationally Efficient Fusion Framework

The architecture of the computationally efficient fusion framework is shown in
Fig. 9.

The first major change in this fusion framework is that only two CNNs are used
as compared with first two frameworks where four CNNs are used, making it
computationally far more efficient. Furthermore, the hybrid fusion in this
framework is performed as combination of image fusion (data level fusion) and
feature level fusion as compared to deep hybrid fusion framework where hybrid
fusion was the combination of feature level and decision level fusions. We
perform image level fusion between signal images and Prewitt filtered signal
images to generate composite images as shown in Fig. 10.

Figure 10: Composite Signal Images of three different actions

Dataset Sampling Rate Modality Training Samples Test Samples UTD MHAD 50 Depth
46639 11660 Inertial 11031 2745 Berkeley MHAD 30 Depth 26400 6500 Inertial
2612 653 UTD Kinect V2 50 Depth 14098 3524 Inertial 3532 884

TABLE I: Dataset Information Figure 11: Confusion Matrix of UTD-MHAD Dataset
for baseline experiment with LSTM. True labels are on vertical axis and
predicted labels are on horizontal axis

The composite image consists of different color channels. Gray region in the
composite image shows the region where the two images have the same
intensities. Green and magenta colors show the region where the intensities
are different. Moreover, in this framework shown in Fig. 9, we did not make
depth modality multimodal to keep this fusion framework computationally
efficient. Alexnet is used to extract features from depth images and the CNN ,
whose architecture is shown in Fig. 6, is used to extract features from the
composite images. The final fusion is CCF to get the most discriminant
features from the input modalities.

We used deep learning models in our proposed fusion framework. Deep learning
models extract features at all levels of their structure and thus low level,
high level and complex features are extracted. On the other hand non-deep
learning models capture only the subset of features and thus have a
generalization problem. Hence the higher accuracy is achieved with the deep
models when compared to non-deep models and this can be seen in the comparison
tables of the datasets. Although the improvement is marginal but at these
higher accuracies, we believe even the marginal improvements are considerable.
It means that our proposed frameworks can successfully classify even the
difficult instances, where other methods are unsuccessful.

Training Parameters | Values
---|---
Momentum | 0.9
Initial Learn Rate | 0.005
Learn Rate Drop Factor | 0.5
Learn Rate Drop Period | 10
$L_{2}$ Regularization | 0.004
Max Epochs | 50
MiniBatchSize | 128
TABLE II: Training Parameters for AlexNet Training Parameters | Values
---|---
Momentum | 0.9
Initial Learn Rate | 0.001
Learn Rate Drop Factor | 0.5
Learn Rate Drop Period | 10
$L_{2}$ Regularization | 0.004
Max Epochs | 100
MiniBatchSize | 64
TABLE III: Training Parameters for the CNN for signal iamges

##  IV Experiments and Results

We experiment on three publicly available Multimodal Human Action Datasets,
namely, UTD-MHAD [34], Berkeley MHAD [35] and UTD Kinect-V2 dataset [36]. We
used subject specific setting for experiments on all datasets. In subject
specific setting, training and testing sets are split randomly across all
subjects.

For our experiments on all deep $M^{2}$ fusion frameworks, we split the
datasets into training and testing samples by randomly splitting 80% data into
training and 20% data into testing samples. The number of training and testing
samples after splitting are shown in Table I. We ran the random split 20 times
and report the average accuracy. We fine tune AlexNet on the SFIs obtained
from the depth sequences and Prewitt filtered SFIs for 50 epochs. The values
of other training parameters are shown in Table II. We reached these values
through using the grid search method for hyperparameter tuning.

Convolutional neural network, whose architecture is shown in Fig. 6, is
trained on signal images and Prewitt filtered signal images with parameters
shown in Table III. The AlexNet and CNN described above are used in
experiments on all three deep $M^{2}$ fusion frameworks explained in section
III. We conduct our experiments on Matlab R2018b on a desktop computer with
NVIDIA GTX-1070 GPU.

Figure 12: Fusion Framework for baseline Experiments

###  IV-A UTD-MHAD Dataset

The UTD-MHAD dataset was collected in an indoor environment and contains both
depth and inertial data components. The names of these 27 different actions
are shown along the vertical and horizontal axis of Fig. 11.

The inertial sensor component of UTD-MHAD dataset is very challenging to train
a CNN. The first deficiency is that inertial sensor was worn either on
volunteer’s right wrist or right thigh depending upon nature of action. Hence
the sensor is worn only on two positions for collecting data of 27 actions
which is not enough to capture all the dependencies and characteristics of
data. The other challenge is that the number of data samples is very small for
training a deep network.

To overcome these problems, we perform data augmentation on signal images of
each dataset to increase the number of training and testing samples by
applying the data augmentation techniques discussed in [27]. The number of
samples shown in Table I are obtained after augmentation of signal images.

We perform experiments with all three deep $M^{2}$ fusion frameworks using
UTD-MHAD datsets with same number of samples shown in Table I and with same
parameters of AlexNet and CNN shown in Tables II and III. The results in terms
of recognition accuracies and their comparison with previous state of art are
shown in Table IV.

####  IV-A1 Baseline Experiments

We performed baseline experiments with UTD-MHAD dataset to validate the effect
of our proposed signal to image transformation scheme. For these experiments,
the same number of training and testing samples shown in Table I were used. We
first trained Long Short-Term Memory (LSTM) Network, a variant of recurrent
neural network (RNN) that can learn long-term dependencies between time steps
of sequence data, on raw inertial data only as LSTM is a well established deep
model for sequential data [37]. The architecture of an LSTM used in our
experiment consists of 200 hidden layers, 27 fully connected layers followed
by a softmax and classification layer. We trained LSTM for 200 epochs and
obtained an accuracy of 86%. The confusion matrix obtained by training LSTM
with raw inertial data is shown in Fig. 11. The blue patches on both sides of
the diagonal show the misclassification results.

In order to improve accuracy we employed ID CNN on raw inertial multivariate
time series. The input is a matrix of 52 time steps times 6 features. 50
kernels of size 5 x 1 are used in first convolutional layer, followed by 2 x 1
subsampling layer. The second convolutional layer contains 100 filters of same
size followed by 2 x 1 subsampling layer, a fully connected layer and a
classification layer. We obtained an accuracy of 77% as shown in Table VI.

This poor performance of raw inertial data with LSTM and 1D CNN compelled us
to transform the raw inertial data into more informative form. Thus to improve
baseline experiments, we transform raw inertial data into signal images and
depth data into depth images and employed CNNs to extract features as CNN is
typically designed to perform well on images.

We first employed AlexNet on signal images. For this we did preprocessing and
convert the original 24 x 52 signal images into 227 x 227. We obtained
accuracy of 91.8%. However by using our own designed CNN, whose architecture
is shown in Fig. 6, we achieved recognition accuracy of 93.7%. Since Alexnet
is a deeper network compared to our CNN architecture, we suspect the reason
for this could be AlexNet overfitting on the training data.

The fusion framework for performing baseline experiments is same as of our
recent work [27] and is shown in Fig 12. The results for baseline experiments
are shown in Table VI. The effect of signal to image transformation is
significant as the accuracy of only inertial data (in the form of signal
images) increases nearly 8%. Furthermore using simple concatenation fusion
between inertial and depth modalities the overall recognition accuracy rises
to 98.4%. The performance of simple fusion framework, where fusion is
performed at single stage, encouraged us to design the proposed $M^{2}$ fusion
frameworks. As we can see in Table IV, the proposed $M^{2}$ fusion frameworks
outperform our baseline that was presented in [27].

The inference speed of all three proposed fusion frameworks is calculated and
is shown in Table V. Inference speed is expressed in microseconds ($\mu$s). It
is defined as a time taken by classifier to classify each test sample.

Further disscussion on results can be found in section IV-D.

Figure 13: Feature visualization from first convolutional layer. (a) Alexnet
on SFIs (b) Alexnet on Prewitt filtered SFIs (c) CNN on signal images (d) CNN
on Prewitt filtered signal images. Feature visualization from last
convolutional layer. (e) Alexnet on SFIs (f) Alexnet on Prewitt filtered SFIs
(g) CNN on signal images (h) CNN on Prewitt filtered signal images.

Previous Methods Accuracy% C.chen et al. [34] 97.1 Bulbul et al. [38] 88.4
N.Dawar et al. [25] 89.2 N.Dawar et al. [26] 92.8 Chen et al. [18] 97.2
Mahjoub et al. [39] 98.5 Our baseline [27] 98.4 Performance of our proposed
fusion frameworks 1\. Deep Multistage Feature Fusion Framework SFI + Signal
image (SI) 98.5 SFI + SI + Prewitt filtered SI 98.6 SFI + Prewitt filtered SFI
+ Signal image 98.9 SFI + Prewitt filtered SFI + SI + Prewitt filtered SI 99
2\. Deep Hybrid Fusion Framework SFI + Signal image (SI) 99 SFI + SI + Prewitt
filtered SI 99.08 SFI + Prewitt filtered SFI + Signal image 99.2 SFI + Prewitt
filtered SFI + SI + Prewitt filtered SI 99.3 3\. Computationally Efficient
Fusion Framework 99.2

TABLE IV:  Comparison of Accuracies of Deep $M^{2}$ Fusion Frameworks with
previous methods on UTD-MHAD dataset

Fusion Frameworks Inference Speed ($\mu$s) Deep Multistage Feature Fusion
Framework 158 Deep Hybrid Fusion Framework 235 Computationally Efficient
Fusion Framework 147

TABLE V: Inference Speed of Proposed Fusion Framework for UTD-MHAD Dataset Methods | Accuracy%
---|---
Raw Inertial data with RNN | 86
Raw Inertial data with 1D CNN | 77
Signal Images (SI) with AlexNet | 91.8
Signal Images with CNN of Fig. 6 | 93.7
Front view Images with AlexNet | 96.1
Fusion of Front, Side and Top view Images | 96.8
Baseline fusion [27] | 98.4
Decision level fusion (Depth + Inertial) | 98.6
TABLE VI: Baseline Experiments on UTD-MHAD Dataset

###  IV-B Berkeley Multimodal Human Action Dataset

The Berkeley MHAD contains 11 actions performed five times by seven male and
five female subjects.

There are five modalities in the dataset from which we used depth and inertial
sensor modalities for experiments as the combination of these to modalities is
cost effective and easy to handle.

Inertial part of the dataset contains six accelerometers and each generates
three sequences. For generating signal images explained in section III-B, we
need six sequences in a row. Thus we used two accelerometer($A_{1}$ and
$A_{4}$) and stacked them row wise to make six sequences. The reason for
selecting $A_{1}$ and $A_{4}$ is that they are worn on the left wrist and
right hip, respectively and are able to generate more useful information than
those worn on both ankles [16]. We performed experiments on all three deep
$M^{2}$ fusion frameworks using Berkeley MHAD dataset with same settings
described in Table I, II and III. The results and their comparison with
previous state of art are shown in Table VII. Results are analyzed in section
IV-D.

Previous Methods Accuracy% F. Ofli et al. [35] 97.81 Alireza Shafaei et al.
[40] 98.1 Earnest Paul Ijjina et al. [41] 98.38 Chen Chen et al. [16] 99.54
Deep Multilevel Multimodal Fusion Frameworks Deep Multistage Feature Fusion
Framework 99.4 Deep Hybrid Fusion Framework 99.8 Computationally Efficient
Fusion Framework 99.6

TABLE VII: Comparison of Accuracies of Deep $M^{2}$ Fusion Frameworks with
previous methods on Berkeley MHAD dataset

Previous Methods Accuracy% Chen et al. [42] 99.5 Z.Ahmad et al. [27] 99.8 Deep
Multilevel Multimodal Fusion Frameworks Deep Mulitstage Feature Fusion
Framework 99.6 Deep Hybrid Fusion Framework 99.8 Computationally Efficient
Fusion Framework 99.8

TABLE VIII: Comparison of Accuracies of Deep $M^{2}$ Fusion Frameworks with
previous methods on Kinect V2 dataset

###  IV-C UTD kinect V2 Dataset

KinectV2 action dataset is another publicly available dataset that contains
both depth and inertial data. It is a new dataset using the second generation
of kinect. It contains 10 actions performed by six subjects with each subject
repeating the action 5 times.

We performed experiments on all three deep $M^{2}$ fusion frameworks using
Kinect V2 dataset with same settings described in Table I, II and III. The
results in terms of recognition accuracies and their comparison with previous
state of art are shown in Table VIII.

††The preprocessed dataset and related code can be found at
https://github.com/zaamad/Deep-Multilevel-Multimodal-Fusion

###  IV-D Discussion on Results

Experimental results on proposed deep $M^{2}$ fusion frameworks are shown in
Tables IV, VII and VIII. These results show the consistency and state of art
performance of the proposed fusion frameworks described in section III. The
uniform performance of all deep $M^{2}$ fusion frameworks is obvious from the
fact that all yield high accuracy for Kinect V2 dataset and lower accuracy for
UTD-MHAD dataset. This is due to the fact that in UTD-MHAD there are actions
which are less discriminative such as ”sit to stand” and ”stand to sit” and
”right arm swipe to left” and ”right arm swipe to right”. Furthermore,
interclass discrimination in Kinect V2 dataset is higher than UTD-MHAD and
Berkeley MHAD datasets.

The objective of making input modalities further multimodal by convolving with
Prewitt filter, played significant role in the state of art performance of the
proposed fusion frameworks. Prewitt filtering intensify the edges or sharp
changes in the images and allow CNN to extract more discriminative features
alongside unfiltered images and thus enhance the performance of proposed
frameworks. These discriminative and complementary features extracted from the
first and last convolutional layers of AlexNets and CNNs can be visualized in
Fig. 13. For visualization and to observe the discriminative and complementary
nature of features at different levels of AlexNet and CNN, we extracted 16
features from each of the first and last convolutional layers of AlexNet and
CNN shown in the constituent figures of Fig. 13. The distinguishable
orientation, brightness, intensity and other related features such as color,
shown in these 16 extracted features reflect the discriminative and
complemenatary nature of the features at different levels of AlexNet and CNN
and thus consolidate the idea of creating modality within the modality to
improve the performance of the deep $M^{2}$ fusion frameworks.

##  V Conclusion

In this paper, we present three novel deep multilevel multimodal ($M^{2}$)
fusion frame works for improving the accuracy of Human Action Recognition
(HAR) using depth and inertial modalities. At the input of each fusion
framework, we transform the depth data and inertial sensor data into
Sequential Front View Images (SFI) and Signal Images (SI) respectively. We
made input modalities, depth and inertial, further multimodal by taking
convolution with an edge detector called Prewitt filter. We extract distinct
and complementary features from the input modalities by employing CNNs and
fused these features at more than one stages in our three novel fusion
frameworks. Rich features obtained after multilevel fusion are inputted to the
SVM classifier. The state of the art recognition accuracy achieved by our
three fusion frameworks on three publicly available multimodal human action
datasets show the dominance of the proposed deep $M^{2}$ fusion frameworks.

## References

  * [1] C. Chen, N. Kehtarnavaz, and R. Jafari, “A medication adherence monitoring system for pill bottles based on a wearable inertial sensor,” in _2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 2014, pp. 4983–4986.
  * [2] P. Corbishley and E. Rodriguez-Villegas, “Breathing detection: towards a miniaturized, wearable, battery-operated monitoring system,” _IEEE Transactions on Biomedical Engineering_ , vol. 55, no. 1, pp. 196–204, 2008.
  * [3] B. Zhou, M. Sundholm, J. Cheng, H. Cruz, and P. Lukowicz, “Never skip leg day: A novel wearable approach to monitoring gym leg exercises,” in _2016 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2016, pp. 1–9.
  * [4] J. Qin, L. Liu, Z. Zhang, Y. Wang, and L. Shao, “Compressive sequential learning for action similarity labeling,” _IEEE Transactions on Image Processing_ , vol. 25, no. 2, pp. 756–769, 2016.
  * [5] T. Plötz, N. Y. Hammerla, and P. L. Olivier, “Feature learning for activity recognition in ubiquitous computing,” in _Twenty-Second International Joint Conference on Artificial Intelligence_ , 2011.
  * [6] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [7] J. K. Aggarwal and L. Xia, “Human activity recognition from 3d data: A review,” _Pattern Recognition Letters_ , vol. 48, pp. 70–80, 2014.
  * [8] A. Y. Yang, R. Jafari, S. S. Sastry, and R. Bajcsy, “Distributed recognition of human actions using wearable motion sensor networks,” _Journal of Ambient Intelligence and Smart Environments_ , vol. 1, no. 2, pp. 103–115, 2009\.
  * [9] C. Chen, R. Jafari, and N. Kehtarnavaz, “A survey of depth and inertial sensor fusion for human action recognition,” _Multimedia Tools and Applications_ , vol. 76, no. 3, pp. 4405–4425, 2017.
  * [10] D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 96–108, 2017.
  * [11] D. L. Hall and J. Llinas, “An introduction to multisensor data fusion,” _Proceedings of the IEEE_ , vol. 85, no. 1, pp. 6–23, 1997.
  * [12] Z. Wu, L. Cai, and H. Meng, “Multi-level fusion of audio and visual features for speaker identification,” in _International Conference on Biometrics_. Springer, 2006, pp. 493–499.
  * [13] P. K. Atrey, M. S. Kankanhalli, and J. B. Oommen, “Goal-oriented optimal subset selection of correlated multimedia streams,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , vol. 3, no. 1, p. 2, 2007.
  * [14] J. Ni, X. Ma, L. Xu, and J. Wang, “An image recognition method based on multiple bp neural networks fusion,” in _International Conference on Information Acquisition, 2004. Proceedings._ IEEE, 2004, pp. 323–326.
  * [15] N. Hatami, Y. Gavet, and J. Debayle, “Classification of time-series images using deep convolutional neural networks,” in _Tenth International Conference on Machine Vision (ICMV 2017)_ , vol. 10696. International Society for Optics and Photonics, 2018, p. 106960Y.
  * [16] C. Chen, R. Jafari, and N. Kehtarnavaz, “Improving human action recognition using fusion of depth camera and inertial sensors,” _IEEE Transactions on Human-Machine Systems_ , vol. 45, no. 1, pp. 51–61, 2015.
  * [17] A. Manzi, A. Moschetti, R. Limosani, L. Fiorini, and F. Cavallo, “Enhancing activity recognition of self-localized robot through depth camera and wearable sensors,” _IEEE Sensors Journal_ , vol. 18, no. 22, pp. 9324–9331, 2018.
  * [18] C. Chen, R. Jafari, and N. Kehtarnavaz, “A real-time human action recognition system using depth and inertial sensor fusion,” _IEEE Sensors Journal_ , vol. 16, no. 3, pp. 773–781, 2016.
  * [19] K. Liu, C. Chen, R. Jafari, and N. Kehtarnavaz, “Fusion of inertial and depth sensor data for robust hand gesture recognition,” _IEEE Sensors Journal_ , vol. 14, no. 6, pp. 1898–1903, 2014.
  * [20] N. Dawar and N. Kehtarnavaz, “Real-time continuous detection and recognition of subject-specific smart tv gestures via fusion of depth and inertial sensing,” _IEEE Access_ , vol. 6, pp. 7019–7028, 2018.
  * [21] B. H. Hu, N. E. Krausz, and L. J. Hargrove, “A novel method for bilateral gait segmentation using a single thigh-mounted depth sensor and imu,” in _2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob)_. IEEE, 2018, pp. 807–812.
  * [22] Y. Guo, D. Tao, W. Liu, and J. Cheng, “Multiview cauchy estimator feature embedding for depth and inertial sensor-based human action recognition,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 47, no. 4, pp. 617–627, 2017.
  * [23] E. A. Bernal, X. Yang, Q. Li, J. Kumar, S. Madhvanath, P. Ramesh, and R. Bala, “Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors,” _IEEE Transactions on Multimedia_ , vol. 20, no. 1, pp. 107–118, 2018.
  * [24] I. Hwang, G. Cha, and S. Oh, “Multi-modal human action recognition using deep neural networks fusing image and inertial sensor data,” in _2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)_. IEEE, 2017, pp. 278–283.
  * [25] N. Dawar, S. Ostadabbas, and N. Kehtarnavaz, “Data augmentation in deep learning-based fusion of depth and inertial sensing for action recognition,” _IEEE Sensors Letters_ , vol. 3, no. 1, pp. 1–4, 2019.
  * [26] N. Dawar and N. Kehtarnavaz, “Action detection and recognition in continuous action streams by deep learning-based sensing fusion,” _IEEE Sensors Journal_ , vol. 18, no. 23, pp. 9660–9668, 2018.
  * [27] Z. Ahmad and N. Khan, “Towards improved human action recognition using convolutional neural networks and multimodal fusion of depth and inertial sensor data,” in _2018 IEEE International Symposium on Multimedia (ISM)_. IEEE, 2018, pp. 223–230.
  * [28] N. Dawar and N. Kehtarnavaz, “A convolutional neural network-based sensor fusion system for monitoring transition movements in healthcare applications,” in _2018 IEEE 14th International Conference on Control and Automation (ICCA)_. IEEE, 2018, pp. 482–485.
  * [29] A. F. Bobick and J. W. Davis, “The recognition of human movement using temporal templates,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 23, no. 3, pp. 257–267, 2001.
  * [30] W. Jiang and Z. Yin, “Human activity recognition using wearable sensors by deep convolutional neural networks,” in _Proceedings of the 23rd ACM international conference on Multimedia_. Acm, 2015, pp. 1307–1310.
  * [31] R. C. Gonzalez, R. E. Woods _et al._ , “Digital image processing,” 2002.
  * [32] V. Uurtio, J. M. Monteiro, J. Kandola, J. Shawe-Taylor, D. Fernandez-Reyes, and J. Rousu, “A tutorial on canonical correlation methods,” _ACM Computing Surveys (CSUR)_ , vol. 50, no. 6, p. 95, 2018.
  * [33] Q.-S. Sun, S.-G. Zeng, Y. Liu, P.-A. Heng, and D.-S. Xia, “A new method of feature fusion and its application in image recognition,” _Pattern Recognition_ , vol. 38, no. 12, pp. 2437–2448, 2005.
  * [34] C. Chen, R. Jafari, and N. Kehtarnavaz, “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in _2015 IEEE International conference on image processing (ICIP)_. IEEE, 2015, pp. 168–172.
  * [35] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, “Berkeley mhad: A comprehensive multimodal human action database,” in _2013 IEEE Workshop on Applications of Computer Vision (WACV)_. IEEE, 2013, pp. 53–60.
  * [36] Kinect2d dataset. [Online]. Available: http://www.utdallas.edu/ kehtar/Kinect2DatasetReadme.pdf
  * [37] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, “Lstm: A search space odyssey,” _IEEE transactions on neural networks and learning systems_ , vol. 28, no. 10, pp. 2222–2232, 2016.
  * [38] M. F. Bulbul, Y. Jiang, and J. Ma, “Dmms-based multiple features fusion for human action recognition,” _International Journal of Multimedia Data Engineering and Management (IJMDEM)_ , vol. 6, no. 4, pp. 23–39, 2015.
  * [39] A. B. Mahjoub and M. Atri, “An efficient end-to-end deep learning architecture for activity classification,” _Analog Integrated Circuits and Signal Processing_ , pp. 1–10, 2018.
  * [40] A. Shafaei and J. J. Little, “Real-time human motion capture with multiple depth cameras,” in _2016 13th Conference on Computer and Robot Vision (CRV)_. IEEE, 2016, pp. 24–31.
  * [41] E. P. Ijjina and C. K. Mohan, “Human action recognition based on mocap information using convolution neural networks,” in _2014 13th International Conference on Machine Learning and Applications_. IEEE, 2014, pp. 159–164.
  * [42] C. Chen, R. Jafari, and N. Kehtarnavaz, “Fusion of depth, skeleton, and inertial data for human action recognition,” in _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2016, pp. 2712–2716.