# Boosting Monocular Depth Estimation with Lightweight 3D Point Fusion

Lam Huynh1 Phong Nguyen1 Jiri Matas2 Esa Rahtu3 Janne Heikkilä1
1University of Oulu 2Czech Technical University in Prague 3Tampere University

###### Abstract

In this paper, we propose enhancing monocular depth estimation by adding 3D
points as depth guidance. Unlike existing depth completion methods, our
approach performs well on extremely sparse and unevenly distributed point
clouds, which makes it agnostic to the source of the 3D points. We achieve
this by introducing a novel multi-scale 3D point fusion network that is both
lightweight and efficient. We demonstrate its versatility on two different
depth estimation problems where the 3D points have been acquired with
conventional structure-from-motion and LiDAR. In both cases, our network
performs on par with state-of-the-art depth completion methods and achieves
significantly higher accuracy when only a small number of points is used while
being more compact in terms of the number of parameters. We show that our
method outperforms some contemporary deep learning based multi-view stereo and
structure-from-motion methods both in accuracy and in compactness.

##  1 Introduction

Depth estimation from 2D images is a classical computer vision problem that
has been mostly tackled with methods from multiple view geometry [15, 45].
Conventional stereo, structure-from-motion and SLAM approaches are already
well-established and integrated to many practical applications. However, they
rely on feature detection and matching that can be challenging especially when
the scene lacks distinct details, and as a result the 3D reconstruction often
becomes sparse and incomplete.

More recently, learning-based approaches have been introduced that enable
dense depth estimation by exploiting priors learned from training images. In
particular, monocular depth estimation that leverages only a single image
together with learned priors has become a popular area of research, where deep
neural networks are used to implement models that directly predict a depth map
for given input image [39, 4, 17, 33, 55, 19]. While the basic idea is simple
and attractive, the accuracy of the monocular depth estimation methods is
limited by the lack of strong geometric constraints such as parallax. Thus,
considerably more accurate depth maps can be achieved with deep learning based
multi-view stereo methods [53, 54, 34, 2]. However, the accuracy comes at the
cost of increased computational complexity as multiple images need to be
aggregated by the network to produce a single depth map.

Another approach for dense depth estimation is to start from depth sensors
like LiDARs, and use depth completion to interpolate the missing depth values
based on RGB data. Despite of impressive results achieved by recent methods
such as [56, 36, 18] they are mainly suitable for cases with relatively high
3D point density, but do not perform well with sparse point clouds.

Figure 1: Dense depth prediction on the NYU-Depth-v2 [44] test set. A point
cloud is produced by a conventional point-based sparse reconstruction method.
The sparse 3D points and a single RGB image are fed to the network to estimate
a high-quality depth map. The dense reconstructed point cloud (top right)
preserves the geometry well. The chart (bottom right) shows values along the
A-B line of the estimated depth map, confidence map, and the sampling points.
The estimated depth value tend towards sampling point values. Confidence value
around sampling point areas are higher. (All values in the chart are
normalized, and sparse point are enhanced for visualization)

In this paper, we start from monocular depth estimation and use a set of 3D
points as constraints to obtain high-quality and dense depth maps as
illustrated in Figure 1. The main difference of our approach to previous depth
completion methods is that the point cloud can be extremely sparse and
unevenly sampled, which enables using various methods for acquiring the 3D
data, including conventional multi-view stereo, structure-from-motion and SLAM
pipelines but also range sensors such as LiDARs. We argue that sparsity is
important as it provides flexibility and cost savings to depth sensing. For
example, in mobile imaging, existing AR frameworks, i.e. ARCore [23], ARKit
[22], and AREngine [24], provide sparse 3D point clouds, while in robotics and
autonomous driving applications, low-resolution range sensors become
sufficient. To this end, we propose a novel learning-based scheme for fusing
RGB and 3D point data. More specifically, our contributions are the following:

  * •

We introduce a novel multi-scale 3D point fusion neural network architecture,
which is more lightweight than the existing state-of-the-art depth completion
methods while being able to efficiently exploit the geometric constraints
provided by a sparse set of 3D points.

  * •

We demonstrate state-of-the-art results on NYU-Depth-v2 and KITTI datasets
with a network that uses only a fraction of the number of parameters compared
to the other recent architectures.

  * •

We also show that our method combined with 3D point clouds obtained by using
COLMAP [43, 42] outperform recent deep learning based multi-view stereo and
structure-from-motion methods both in accuracy and in compactness.

##  2 Related work

#### Single image depth estimation (SIDE):

SIDE was first introduced by Saxena et al. [41] and it gained momentum from
the work by Eigen et al. [10, 9]. Since then, the number of related studies
has grown rapidly [28, 14, 37, 40, 30, 26, 17, 4, 12, 39, 33, 32, 29, 19]. At
first, the proposed SIDE methods improved the accuracy by employing large
architectures [28, 17] and more complex encoding-decoding schemes [4]. Then,
they started to diverge into using semantic labels [26], exploiting the
relationship between depth and surface normal [37], reformulating as a
classification problem [14] or mixing both [40]. Other studies suggested to
estimate relative depth [30] or to learn calibration patterns to improve the
generalization ability. Recent SIDE approaches exploit monocular priors such
as occlusion [39], and planar structures either explicitly [33, 32, 55] or
implicitly [19]. Despite these efforts, SIDE still generalizes quite poorly to
unseen data. In this work, we leverage SIDE’s ability to produce dense depth
estimations and inject it with a small set of depth measurements to boost the
accuracy while further shrinking the network size.

#### Dense depth estimation from sparse depth:

Depth completion is a related problem where the aim is to densify or inpaint
an incomplete depth map. Diebel and Thrun [8] is one of the first studies to
tackle this problem using Markov random fields. Hawe et al. [16] estimate
disparity using wavelet analysis. The problem gained popularity as commodity
depth sensors and laser scanners (or LiDARs) become more available. Uhrig et
al. [47] proposed sparse convolution to train a sparse invariant network.
Jaritz et al. [25] leveraged semantics to train the network at varying
sparsity levels. Ma et al. [35] concatenated the sparse depth map to an RGB
image, and used this RGBD volume for training. Xu et al. [51] filled in the
missing depth values using the depth normal constraint. Imran et al. [21]
addressed the depth completion problem using depth coefficients as a
representation. Qiu et al. [38] suggested depth and normal fusion using
learned attention maps. Methods based on a spatial propagation network (SPN)
iterative optimize the dense depth map either in local [6, 7] or non-local
[36] affinity. Chen et al. [5] suggested fusing features from an image and 3D
points to produce the dense depth. However, these depth completion methods
usually aim for outdoor environments and street views where the points come
from a LiDAR.

Figure 2: Overview architecture of the 3D point fusion network. Our model
consists of five Fusion-Nets that iteratively extract and fuse 2D and 3D
features at multiple-scale before predicting the final depth map at the
highest spatial resolution.

The difficulty of the depth completion problem much depends on the density of
the 3D points used as an input to the algorithm. For example, LiDARs can
produce relatively dense and regularly sampled point clouds without large
holes, while passive image-based 3D reconstruction techniques, such as stereo
or SLAM, result in substantially sparser set of points where the sampling is
highly irregular and depends on the surface details. Thus, we argue that depth
completion becomes a much harder problem when using a sparse point cloud from
image-based reconstruction rather than from a LiDAR, and consequently, it also
requires better regularization for the depth. To this end, we introduce a
novel 3D fusion point network that efficiently learns to fuse image and
geometric features to boost the performance of a monocular depth estimation
network. It is a generic approach that can exploit RGB and 3D point data from
various sources and environments. It can deal with indoor scenes that are
often more diverse and challenging than outdoor environments, but it can be
also used for depth estimation from street view scenes. Our work is inspired
by [5], but instead of sequentially fusing features at the same resolution, we
build a deeper model to extract and fuse features at multiple-scales. This is
crucial since [5] has been developed for depth completion of LiDAR data and as
shown in our experiments it fails with a sparse set of points whereas thanks
to the multi-scale approach our method can achieve reasonable accuracy from a
few or even zero depth measurements.

##  3 Method

An overview of our 3D point fusion network is shown in Figure 2. It is a fully
convolutional framework that takes an RGB image and sparse 3D points as inputs
to estimate a dense depth map. The 3D points serve as constraints to fix the
overall geometry of the depth map produced by the network. To deal with the
unstructured 3D point cloud, the points are first projected to the image plane
and their $z$ coordinates are used to create a sparse depth map. Next, the RGB
image is stacked with the sparse depth to form an RGBD image. We also apply
two convolutional layers to the sparse depth and the RGBD image separately.
The two outputs are concatenated to build the low-level input features that
are fed to the first fusion-net module. The core network consists of five
Fusion-Nets that operate at different feature resolutions. Each Fusion-Net
contains a feature fusion encoder (E), a confidence predictor (C), a decoder
(D), and a refinement (R) module as illustrates in Figure 3. We describe these
modules in the following subsections and finish this section by giving details
about our loss function.

Figure 3: Details of Fusion-Net $n\in[1,5]$, where $n$ is the scale
resolution. Main components include the feature fusion encoder (E), confidence
predictor (C), decoder (D) and refinement (R) are color coded as gray, cyan,
orange and yellow, respectively.

###  3.1 Feature Fusion Encoder

Convolutional neural networks are good in processing regularly sampled data in
a tensor form. Because our input point clouds are sparse and they represent
geometric constraints unlike the image data, we cannot just rely on simple
concatenation to fuse the information, but we need better representations.
Inspired by a recent depth completion method [5], we design a feature fusion
encoder to extract low-level features from RGB images and 3D points.

Our feature fusion encoder takes a 3D tensor ($C\times H\times W$) and a set
of sparse points ($3\times N$) as inputs, where $C$ is the number of feature
channels, $H$ and $W$ are the height and width of the input tensor, and $N$ is
the number of 3D points. The output is a 3D tensor with a similar shape to the
input tensor. Details of the feature fusion encoder are shown in the gray box
of Figure 3. It consists of two 2D branches, one 3D branch, and one
convolutional layer for feature fusion.

The 2D convolutional branches: The 2D branches are convolved at two different
resolutions to learn multi-scale representations from the input 3D tensor. The
first 2D branch has one convolutional layer with stride one to extract
features at the same size as the input volume. The second 2D branch is a
cascade of a stride two convolutional, a stride one convolutional, and an
upsampling layer to obtain coarser features of the input tensor. The two
outputs are summed to aggregate appearance features at different resolutions.

The 3D point convolution branch: The 3D branch aims to extract structural
features from the sparse points. This is difficult for 2D convolutions that
operate on local neighbors as 3D points are located on an irregular grid.
Therefore, we utilize the feature-kernel alignment convolution (FKAConv) [3]
that operates directly on 3D points to avoid this problem. The key idea of the
FKAConv is to learn a linear transformation to align the neighboring points
with the grid-like kernel. After that, it performs a weighted sum between this
kernel and the features of the 3D points. One can see that 2D convolution is a
special case where the learned linear transformation is always an identity
matrix.

As shown in Figure 3, our 3D branch consists of two FKAConv layers. We first
extract the features of the 3D points from the input tensor using their
projected 2D indices on the image plane. This volume has the size of $C\times
N$. Next, we feed the point features and their 3D coordinates to the FKAConv
layers. FKAConv selects a set of k-neighboring points for every input point
and learns a transformation matrix to align the 3D points with its kernel. The
point features are then convolved with the aligned 3D points to produce a 2D
tensor of shape $C\times N$. The output features are projected back to an
empty 3D tensor of size $C\times H\times W$ using the projected 2D indices.
Features of other positions are set to zero.

2D-3D Feature Fusion: Output volumes from the 2D and 3D branches have the same
shape as the input tensor ($C\times H\times W$). Therefore, to fuse these
features, we sum them together before applying a 2D convolutional layer to
output a 3D tensor of the size $C\times H\times W$. Finally, we add a residual
connection to avoid vanishing gradient during training.

###  3.2 Encoder, Decoder, and Confidence Predictor modules

Encoder and Decoder Module: Designing efficient decoder and refinement modules
is essential for the depth estimation problem [13, 50]. A common practice is
to create large and complex decoders to produce accurate depth maps with sharp
edges and fine details. However, we argue that by iteratively fusing relevant
depth measurements from the 3D points with appearance features from image
pixels, we can significantly reduce the size of our decoder and refinement
designs. That is, our decoder and refinement modules have only two
convolutional layers for each component. To simplify further, we use the same
decoder and refinement designs for all Fusion-Nets.

As shown in the orange box of Figure 3, the decoder transforms the fused
features from the encoder before feeding them to the refinement module (the
yellow box in Figure 3). We then initially obtain an output tensor of the
decoder and a depth map. The estimated confidence map later modifies these two
outputs.

Figure 4: Depth map rectification by predicted confidence (green arrow). Confidence map values range from 0.0 (low) to 1.0 (high).  Table 1: Evaluation on the NYU dataset. Metrics mark $\downarrow$ mean lower is better while $\uparrow$ is otherwise. Methods with ‡ are trained using extra data. Figures with ⋆ indicates 3D COLMAP points while figures without ⋆ are obtained using randomly sampled points from GT depths. Architecture | #3D pts | #params | REL$\downarrow$ | RMSE$\downarrow$ |  $\boldsymbol{\delta_{1}}$$\uparrow$ |  $\boldsymbol{\delta_{2}}$$\uparrow$ |  $\boldsymbol{\delta_{3}}$$\uparrow$
---|---|---|---|---|---|---|---
SharpNet |  Ramam.’19‡ [39] | 0 | 80.4M | 0.139 | 0.502 | 0.836 | 0.966 | 0.993
Revisited mono-depth |  Hu’19 [17] | 0 | 157.0M | 0.115 | 0.530 | 0.866 | 0.975 | 0.993
SARPN |  Chen’19 [4] | 0 | 210.3M | 0.111 | 0.514 | 0.878 | 0.977 | 0.994
VNL |  Yin’19 [55] | 0 | 114.2M | 0.108 | 0.416 | 0.875 | 0.976 | 0.994
DAV |  Huynh’20 [19] | 0 | 25.1M | 0.108 | 0.412 | 0.882 | 0.980 | 0.996
Point-Fusion | Ours | 0 | 8.7M | 0.128 | 0.505 | 0.847 | 0.971 | 0.994
NLSPN |  Park’20 [36] | 2 | 25.8M | 0.300 | 1.152 | 0.393 | 0.697 | 0.879
Point-Fusion | Ours | 2 | 8.7M | 0.109 | 0.470 | 0.875 | 0.975 | 0.995
NLSPN |  Park’20 [36] | 32 | 25.8M | 0.114 | 0.554 | 0.825 | 0.947 | 0.985
Point-Fusion | Ours | 32 | 8.7M | 0.057 | 0.319 | 0.963 | 0.992 | 0.998
Sparse & Dense |  Jaritz’18 [25] | 200 | 58.3M | 0.050 | 0.194 | 0.930 | 0.960 | 0.991
S2D |  Ma’18 [35] | 200 | 42.8M | 0.044 | 0.230 | 0.971 | 0.994 | 0.998
GuideNet |  Tang’20 [46] | 200 | 63.3M | 0.024 | 0.142 | 0.988 | 0.998 | 1.000
NLSPN |  Park’20 [36] | 200 | 25.8M | 0.019 | 0.136 | 0.989 | 0.998 | 0.999
Point-Fusion | Ours | 200 | 8.7M | 0.015 | 0.112 | 0.995 | 0.999 | 1.000
FuseNet |  Chen’19 [5] | 500 | 1.9M | 0.318 | 0.859 | 0.688 | 0.789 | 0.887
CSPN |  Cheng’18 [6] | 500 | 18.5M | 0.016 | 0.117 | 0.992 | 0.999 | 1.000
DeepLiDAR |  Qiu’19 [38] | 500 | 53.4M | 0.022 | 0.115 | 0.993 | 0.999 | 1.000
Depth Coefficients |  Imran’19 [21] | 500 | 45.7M | 0.013 | 0.118 | 0.994 | 0.999 | -
DepthNormal |  Xu’19 [51] | 500 | 29.1M | 0.018 | 0.112 | 0.995 | 0.999 | 1.000
CSPN++ |  Cheng’20 [7] | 500 | 28.8M | - | 0.116 | - | - | -
GuideNet |  Tang’20 [46] | 500 | 63.3M | 0.015 | 0.101 | 0.995 | 0.999 | 1.000
NLSPN |  Park’20 [36] | 500 | 25.8M | 0.012 | 0.092 | 0.996 | 0.999 | 1.000
Point-Fusion | Ours | 500 | 8.7M | 0.014 | 0.090 | 0.996 | 0.999 | 1.000
MVSNet |  Yao’18 [53] | - | 124.5M | 0.043 | 0.162 | 0.940 | 0.972 | 0.996
CodeSLAM |  Bloesch’18 [2] | - | 66.3M | 0.096 | 0.251 | 0.910 | 0.962 | 0.989
Consistent depth |  Luo’20 [34] | - | 178.2M | 0.086 | 0.345 | 0.916 | 0.959 | 0.984
NLSPN |  Park’20 [36] |  500⋆ | 25.8M | 0.042 | 0.144 | 0.949 | 0.981 | 0.999
Point-Fusion | Ours |  500⋆ | 8.7M | 0.022 | 0.126 | 0.994 | 0.999 | 1.000

Confidence Predictor: Although the input 3D sparse points provide useful depth
measurements, they can also contain noise. Hence, we proposed a simple yet
efficient confidence predictor to attenuate the effect of noise. As
illustrated in the cyan box of Figure 3, the output volumes from the feature
fusion encoder are fed to three convolutional layers followed by a sigmoid to
output the probability for every pixel. This information is then used to alter
the initial depth map and the output features of the decoder. Moreover, we add
residual connections at the end of the decoder and refinement blocks to
prevent the vanishing gradient problem and regularize the confidence map’s
errors. The initial depth map is corrected based on the confidence map, as
illustrated in Figure 4.

###  3.3 Multi-scale Loss function

We calculate the loss at multiple feature resolutions to train our network.
The full loss is defined as:

$\mathcal{L}=\sum_{i=1}^{n=5}\gamma^{i}(\mathcal{L}^{i}_{log}+\mu\mathcal{L}^{i}_{grad}+\theta\mathcal{L}^{i}_{norm})$
(1)

where $n$ is the number of resolution scales and $\gamma^{i}\in\mathbb{R}^{+}$
is the loss weight at scale $i$, $\mathcal{L}_{log}$ is a variation of the
$L_{1}$ norm that minimizes error on the sparse depth pixels,
$\mathcal{L}_{grad}$ optimizes the error on edge structures, and
$\mathcal{L}_{norm}$ penalizes angular error between the ground truth and
predicted normal surfaces. These loss terms were introduced by Hu et al. [17]
and widely adopted by state-of-art monocular depth estimation methods [4, 19].
Subsection 4.2 describes in detail how the network is trained using these loss
functions.

##  4 Experiments

In this section, we evaluate the performance of the proposed method and
compare it with several baselines on the NYU-Depth-v2 and KITTI datasets.

###  4.1 Dataset and Evaluation metrics

#### Datasets.

The NYU-Depth-v2 dataset contains approximately $120K$ RGB-D images recorded
from 464 indoor scenes. We extract the raw RGB frames from the original videos
and reconstruct sparse 3D point clouds using the COLMAP [42, 43] structure-
from-motion software. COLMAP is also used to extract the camera poses for
multi-view stereo methods. The 3D points are back-projected to each input view
to obtain a sparse set of depth values. We use 60K images for training and 654
images from the official test set for evaluating the methods. For KITTI, we
utilize 85K images for training, 1000 images for validation and 1000 images
for testing on the KITTI depth completion benchmark [47].

Figure 5:  Top: RMSE and iRMSE metrics v. the number of parameters plots for
recent depth completion methods on NYU-Depth-v2 (a) and KITTI (b). Bottom:
RMSE metrics for different sparsity and patterns for NYU-Depth-v2 (c) and
KITTI (d).

#### Evaluation metrics.

We report the results in terms of standard metrics for each dataset. For NYU-
Depth-v2 we compute the mean absolute relative error (REL), root mean square
error (RMSE), and thresholded accuracy ($\delta_{i}$). For KITTI, we also
calculate RMSE plus mean absolute error (MAE), root mean square error (iRMSE)
and mean absolute error (iMAE) of the inverse depth values. The detailed
definitions of the measures are provided in the supplementary material.

Figure 6: Qualitative results on NYU-v2 test set. Note that all methods use
200 randomly sampled 3D points as input.

###  4.2 Implementation details

The proposed model is trained for 150 epochs on a single TITAN RTX using batch
size of 32, the Adam optimizer [27] with
$(\beta_{1},\beta_{2},\epsilon)=(0.9,0.999,10^{-8})$, and the loss function
presented in (1). The initial learning rate is $1.2*10^{-4}$, but from epoch
10 the learning is reduced by $6\%$ per $5$ epochs. We set the number of
scales $n$ in (1) to 5, weight loss coefficients $\mu,\theta$ to $1.0$, and
the scale weight losses
$\gamma^{1},\gamma^{2},\gamma^{3},\gamma^{4},\gamma^{5}$ to
$1.0,0.75,0.5,0.25$ and $0.125$ respectively. To remove the effect of the
arbitrary scale of the COLMAP points, we center and normalize the 3D inputs to
a unit sphere before the training. During training, we augment the input RGB
and ground truth depth images using random rotation ([-5.0, +5.0] degrees),
horizontal flip, rectangular window droppings, and colorization (RGB only).

###  4.3 Comparison with State-of-the-art

The proposed method is related to multiple partially overlapping problems and,
therefore, we compare it with several baseline methods in monocular depth
estimation [39, 17, 4, 55, 19], depth completion [5, 6, 7, 11, 18, 20, 21, 25,
31, 35, 36, 38, 46, 48, 51, 52], deep multi-view stereo [53], deep structure-
from-motion/SLAM [34, 2]. The baseline results are obtained using the pre-
trained models [4, 17, 39, 55, 34, 18, 11], re-training using the official
NYU-v2 [2, 35, 36, 53] code, using our own re-implementations [19, 25], and
from the original papers [7, 38, 51, 21, 46, 52].

#### NYU-Depth-v2.

The performance metrics, computed between the estimated depth maps and the
ground truth, are provided in Table 1. In addition, we report the number of
method parameters, and the number of 3D points used in the estimation.
Compared to monocular depth estimation studies, the proposed method provides a
substantial improvement according to all metrics. For instance, REL, RMSE and
thresholded accuracy ($\delta_{i}$) are improved by $47\%,22.5\%$ and $10\%$,
respectively, by using only $35\%$ of the model parameters and 32 additional
3D points. Table 1 also shows that our method produces results close to state-
of-the-art even without using any 3D inputs, while 2 points are already enough
to be on par with the baseline approaches.

Table 2: Evaluation results on the test set of the KITTI depth completion
benchmark. Performance figures are color-coded as red, green, and blue,
corresponding to first, second and third best results, respectively.

Architecture #param RMSE MAE iRMSE iMAE Sparse&Dense [25] 58.3M 917.6 234.8
2.17 0.95 NConv [11] 0.36M 829.9 233.2 2.60 1.03 DepthNormal [51] 29.1M 777.1
235.2 2.42 1.13 FusionNet [48] 2.5M 772.9 215.1 2.19 0.95 FuseNet [5] 1.9M
752.9 221.2 2.34 1.14 DeepLiDAR [38] 53.4M 748.4 226.5 2.56 1.15 DDP [52]
29.1M 832.9 203.9 2.10 0.85 MSG-CHN [31] 1.25M 762.2 220.4 2.30 0.98 CSPN++
[7] 28.8M 743.7 209.3 2.07 0.90 NLSPN [36] 25.8M 741.7 199.6 1.99 0.84
GuideNet [46] 63.3M 736.2 218.8 2.25 0.99 ENet [18] 131.6M 741.3 216.3 2.14
0.95 PENet [18] 133.7M 730.1 210.6 2.17 0.94 TWISE_2 [20] - 840.2 195.6 2.08
0.82 Point-Fusion(Ours) 8.7M 741.9 201.1 1.97 0.85

Figure 7: Examples from the KITTI validation set. Input images (a), ground
truth LiDAR (b). Results from (c) Depth2Depth [56], (d) DeepLiDAR [38], (e)
NLSPN [36], (f) ENet [18], (g) PENet [18], and (h) the proposed method.

Compared to the depth completion methods, we obtain state-of-the-art
performance while using clearly less model parameters as shown in Figure 5 a).
The best performing baselines, NLSPN [36], DepthNormal [51], GuideNet [46] use
$2.9$, $3.4$, and $7.3$ times more parameters compared to our method,
respectively. Instead of using the explicit 3D points, the multi-view stereo
[53], structure-from-motion [34], and SLAM [2] methods utilise multiple RGB
images with camera poses. The results in Table 1 indicate that the proposed
model outperforms also these methods using only a fraction of the model
parameters.

Figure 6 shows qualitative results of the predicted depth maps and
reconstructed points cloud for our method and for [36]. The baseline [36]
results are obtained using the pre-trained model provided by the authors.
Although both methods produce high quality depth maps, the proposed model is
better in recovering fine details in challenging regions and introduces less
distortions on flat surfaces.

We also provide examples where we reconstructed a very sparse set of 3D points
(32 points) from two images and utilized those as the 3D inputs. The dense
depth maps obtained with this setting using our method, NLSPN [36] and MVSNet
[53] are illustrated in Figure 8. We argue that state-of-the-art depth
completion methods are usually vulnerable in high sparsity cases, while deep
multi-view stereo performance degrades with less input views. One the other
hand, our method produces high quality depth maps with significantly less
distortions. Additional results are provided in the supplementary material.

#### KITTI.

To demonstrate the versatility of the proposed method, we also experiment with
outdoor data. For this purpose, we train and test our model with the KITTI
depth completion dataset [47], in which we perform on par with state-of-the-
art methods using a significantly smaller number of parameters as shown in
Table 2. We notice that there is a clear trade-off between model parameters
and performance as shown in Figure 5 b). Figure 7 presents qualitative
comparison with baseline methods. The proposed method produces finer depth
details as emphasized in the highlight areas. However, the difference is
largest with a small number of input 3D points as depicted in Figure 5 d). The
results suggest that high-quality depth maps can be obtained by using only a
few LiDAR points enabling more cost efficient solutions. Additional results
are also added to the supplementary material.

Figure 8:  NYU test set examples (a). Dense depth maps and reconstructed point
cloud from two images: (b) ground truth, (c) NLSPN [36], (d) MVSNet [53], and
(e) the proposed method. Figure 9: Qualitative comparison of the pattern
(left) and quantity (right) of input points. Random points are sampled from
the dense ground truth depth map. For COLMAP points, we extract the image
frames from raw NYU data and run COLMAP to obtain the points. The pattern and
number of points are kept similar in all cases. Left: the number of points in
use is 64. Results in the first row show the random input point have better
spatial distribution than COLMAP points since they cover flat surfaces like
walls, floors or doors. Right: example results show the predicted depth maps
using random sampled set of 32, 128 and 200 points respectively. Our method
perform consistently better than NLSPN [36] in all cases. (Points are enhanced
for visualization)

###  4.4 Ablation studies

#### Number and sampling of input 3D points.

To analyze how the quantity and spatial distribution of the input 3D point
affect the results, we performed experiments with varying 3D point patterns.
For this purpose we generate sparse point sets by randomly sampling from the
dense ground truth or from COLMAP output. We expect that by sampling from
dense depth map provides better results compared to the COLMAP points. This is
because, dense depth map covers also flat textureless surfaces such as walls,
floor, and doors. However, such points might not be easy to obtain in
practice, whereas COLMAP points represent location which are often
reconstructed by SfM or SLAM methods.

Figure 5 c) presents the RMSE errors for different number of input points for
both types. The results confirm the initial assumption that sampling from a
dense depth map results in better performance. Moreover, we notice that the
proposed method obtains higher accuracy compared to NLSPN [36] with all point
sets. In fact, we obtain similar performance using COLMAP points as NLSPN [36]
using points from the dense depth map. Figure 9 shows qualitative comparison
with NLSPN [36].

#### Confidence predictor.

We study the impact of the confidence predictor module by training our method
with and without this component. We report the results in Table 3. When
compared to a model without the confidence map, REL improves $\sim 3.5\%$, and
RMSE $\sim 7.2\%$.

Table 3: Ablation studies of models without and with the confidence predictor (CP) on NYU-Depth-v2. Training | REL$\downarrow$ | RMSE$\downarrow$ |  $\boldsymbol{\delta_{1}}$$\uparrow$ |  $\boldsymbol{\delta_{2}}$$\uparrow$ |  $\boldsymbol{\delta_{3}}$$\uparrow$
---|---|---|---|---|---
w/o CP | 0.015 | 0.097 | 0.994 | 0.997 | 0.999
w/ CP | 0.014 | 0.090 | 0.996 | 0.999 | 1.000

#### Multi-scale Fusion-Net.

We assess how the number of Fusion-Nets affects the performance. For this
purpose, we train our model using $2-6$ Fusion-Nets. The corresponding RMSE
for the NYU-Depth-v2 test set are provided in Table 4. The results improve by
increasing the number of Fusion-Net to five and degrade after that. As each
Fusion-Net perform at a different feature resolution, we argue that five is
the optimal cascade size for the network to learn the geometric features from
the 3D inputs.

Table 4: Performance of our model using different numbers of Fusion-Net on NYU-Depth-v2. # of FusionNet | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---
RMSE | 0.105 | 0.097 | 0.094 | 0.090 | 0.093

#### 3D point convolutions.

We study the effect of different types of 3D point convolutions by training
our model using the deep parametric continuous convolution (PCC) [49] and the
FKAConv [3]. The results are provided in Table 5. The comparison with the PCC
shows that the FKAConv module reduces the network size by $\sim 5\%$ while
slightly improves the performance by $\sim 3\%$. We also trained our model
without the 3D branch and the performance dropped considerably as shown in
Table 5.

Table 5: Performance of our model applying different types of 3D point convolutions on NYU-Depth-v2. Training | #params | REL$\downarrow$ | RMSE$\downarrow$ |  $\boldsymbol{\delta_{1}}$$\uparrow$ |  $\boldsymbol{\delta_{2}}$$\uparrow$
---|---|---|---|---|---
w/o 3D branch | 7.6M | 0.044 | 0.196 | 0.980 | 0.993
w/ PCC | 9.1M | 0.015 | 0.096 | 0.994 | 0.996
w/ FKAConv | 8.7M | 0.014 | 0.090 | 0.996 | 0.999

##  5 Conclusion

We propose a novel and pragmatic approach that fuses RGB monocular depth
estimation with information from a sparse set of 3D points for dense depth
estimation. Experiments on common indoor and outdoor datasets show that we
achieve state-of-the-art results while being compact in terms of the number of
parameters. Moreover, our method can also produce, unlike the competitors,
high-quality depth maps using an extremely sparse set of 3D points, which
enables a cost-efficient solution for obtaining accurate depth maps for
various applications where dense depth is needed.

## References

  * [1] Jonathan T Barron and Ben Poole.  The fast bilateral solver.  In European Conference on Computer Vision, pages 617–632. Springer, 2016.
  * [2] Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davison.  Codeslam—learning a compact, optimisable representation for dense visual slam.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2560–2568, 2018.
  * [3] Alexandre Boulch, Gilles Puy, and Renaud Marlet.  FKAConv: Feature-Kernel Alignment for Point Cloud Convolution.  In 15th Asian Conference on Computer Vision (ACCV 2020), 2020.
  * [4] Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha.  Structure-aware residual pyramid network for monocular depth estimation.  In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 694–700. AAAI Press, 2019.
  * [5] Yun Chen, Bin Yang, Ming Liang, and Raquel Urtasun.  Learning joint 2d-3d representations for depth completion.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10023–10032, 2019.
  * [6] Xinjing Cheng, Peng Wang, and Ruigang Yang.  Depth estimation via affinity learned with convolutional spatial propagation network.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 103–119, 2018.
  * [7] Xinjing Cheng, Peng Wang, and Ruigang Yang.  Learning depth with convolutional spatial propagation network.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [8] James Diebel and Sebastian Thrun.  An application of markov random fields to range sensing.  In Advances in neural information processing systems, pages 291–298, 2006.
  * [9] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2650–2658, 2015.
  * [10] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in neural information processing systems, pages 2366–2374, 2014.
  * [11] Abdelrahman Eldesokey, Michael Felsberg, and Fahad Shahbaz Khan.  Confidence propagation through cnns for guided sparse depth regression.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [12] Jose M Facil, Benjamin Ummenhofer, Huizhong Zhou, Luis Montesano, Thomas Brox, and Javier Civera.  Cam-convs: camera-aware multi-scale convolutions for single-view depth.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11826–11835, 2019.
  * [13] Zhicheng Fang, Xiaoran Chen, Yuhua Chen, and Luc Van Gool.  Towards good practice for cnn-based monocular depth estimation.  In The IEEE Winter Conference on Applications of Computer Vision, pages 1091–1100, 2020.
  * [14] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2002–2011, 2018.
  * [15] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [16] Simon Hawe, Martin Kleinsteuber, and Klaus Diepold.  Dense disparity maps from sparse disparity measurements.  In 2011 International Conference on Computer Vision, pages 2126–2133. IEEE, 2011.
  * [17] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani.  Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries.  In IEEE Winter Conf. on Applications of Computer Vision (WACV), 2019\.
  * [18] Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and Xiaojin Gong.  Towards precise and efficient image guided depth completion.  ICRA, 2021.
  * [19] Lam Huynh, Phong Nguyen-Ha, Jiri Matas, Esa Rahtu, and Janne Heikkilä.  Guiding monocular depth estimation using depth-attention volume.  In European Conference on Computer Vision, pages 581–597. Springer, 2020.
  * [20] Saif Imran, Xiaoming Liu, and Daniel Morris.  Depth completion with twin-surface extrapolation at occlusion boundaries.  In In Proceeding of IEEE Computer Vision and Pattern Recognition, Nashville, TN, June 2021.
  * [21] Saif Imran, Yunfei Long, Xiaoming Liu, and Daniel Morris.  Depth coefficients for depth completion.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12438–12447. IEEE, 2019.
  * [22] Apple Inc.  ARKit 2, Accessed 28 Feb 2019.
  * [23] Google Inc.  ARCore Resources, Released 01 Mar 2018.
  * [24] Huawei Inc.  AREngine, Released 30 June 2019.
  * [25] Maximilian Jaritz, Raoul De Charette, Emilie Wirbel, Xavier Perrotton, and Fawzi Nashashibi.  Sparse and dense data with cnns: Depth completion and semantic segmentation.  In 2018 International Conference on 3D Vision (3DV), pages 52–60. IEEE, 2018.
  * [26] Jianbo Jiao, Ying Cao, Yibing Song, and Rynson Lau.  Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 53–69, 2018.
  * [27] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [28] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 239–248. IEEE, 2016.
  * [29] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh.  From big to small: Multi-scale local planar guidance for monocular depth estimation.  arXiv preprint arXiv:1907.10326, 2019.
  * [30] Jae-Han Lee and Chang-Su Kim.  Monocular depth estimation using relative depth maps.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2019.
  * [31] Ang Li, Zejian Yuan, Yonggen Ling, Wanchao Chi, Chong Zhang, et al.  A multi-scale guided cascade hourglass network for depth completion.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 32–40, 2020.
  * [32] Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz.  Planercnn: 3d plane detection and reconstruction from a single image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4450–4459, 2019.
  * [33] Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa.  Planenet: Piece-wise planar reconstruction from a single rgb image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2579–2588, 2018.
  * [34] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf.  Consistent video depth estimation.  ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH), 39(4), 2020.
  * [35] Fangchang Mal and Sertac Karaman.  Sparse-to-dense: Depth prediction from sparse depth samples and a single image.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2018.
  * [36] Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon.  Non-local spatial propagation network for depth completion.  In Proc. of European Conference on Computer Vision (ECCV), 2020\.
  * [37] Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia.  Geonet: Geometric neural network for joint depth and surface normal estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 283–291, 2018.
  * [38] Jiaxiong Qiu, Zhaopeng Cui, Yinda Zhang, Xingdi Zhang, Shuaicheng Liu, Bing Zeng, and Marc Pollefeys.  Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [39] Michael Ramamonjisoa and Vincent Lepetit.  Sharpnet: Fast and accurate recovery of occluding contours in monocular depth estimation.  The IEEE International Conference on Computer Vision (ICCV) Workshops, 2019.
  * [40] Haoyu Ren, Mostafa El-khamy, and Jungwon Lee.  Deep robust single image depth estimation neural network using scene understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 37–45, 2019.
  * [41] Ashutosh Saxena, Sung H Chung, and Andrew Y Ng.  Learning depth from single monocular images.  In Advances in neural information processing systems, pages 1161–1168, 2006.
  * [42] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [43] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision (ECCV), 2016.
  * [44] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In European Conference on Computer Vision, pages 746–760. Springer, 2012.
  * [45] Richard Szeliski.  Structure from motion.  In Computer Vision, pages 303–334. Springer, 2011.
  * [46] Jie Tang, Fei-Peng Tian, Wei Feng, Jian Li, and Ping Tan.  Learning guided convolutional network for depth completion.  IEEE Transactions on Image Processing, 30:1116–1129, 2020.
  * [47] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant cnns.  In 2017 international conference on 3D Vision (3DV), pages 11–20. IEEE, 2017.
  * [48] Wouter Van Gansbeke, Davy Neven, Bert De Brabandere, and Luc Van Gool.  Sparse and noisy lidar completion with rgb guidance and uncertainty.  In 2019 16th International Conference on Machine Vision Applications (MVA), pages 1–6. IEEE, 2019.
  * [49] Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun.  Deep parametric continuous convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2589–2597, 2018.
  * [50] Zbigniew Wojna, Vittorio Ferrari, Sergio Guadarrama, Nathan Silberman, Liang-Chieh Chen, Alireza Fathi, and Jasper Uijlings.  The devil is in the decoder: Classification, regression and gans.  International Journal of Computer Vision, 127(11-12):1694–1706, 2019.
  * [51] Yan Xu, Xinge Zhu, Jianping Shi, Guofeng Zhang, Hujun Bao, and Hongsheng Li.  Depth completion from sparse lidar data with depth-normal constraints.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2811–2820, 2019.
  * [52] Yanchao Yang, Alex Wong, and Stefano Soatto.  Dense depth posterior (ddp) from single image and sparse range.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3353–3362, 2019.
  * [53] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  European Conference on Computer Vision (ECCV), 2018.
  * [54] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  Computer Vision and Pattern Recognition (CVPR), 2019.
  * [55] Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan.  Enforcing geometric constraints of virtual normal for depth prediction.  In The IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [56] Yinda Zhang and Thomas Funkhouser.  Deep depth completion of a single rgb-d image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 175–185, 2018.


# Speech-Based Human-Exoskeleton Interaction for
Lower Limb Motion Planning

Eddie Guo1,2, Christopher Perlette3, Mojtaba Sharifi1,4,5, Lukas Grasse3,
Matthew Tata3,
Vivian K. Mushahwar5, and Mahdi Tavakoli1,4,5
1Department of Electrical and Computer Engineering, University of Alberta,
2Cumming School of Medicine, University of Calgary
3Department of Neuroscience, University of Lethbridge
4Department of Mechanical Engineering, San Jose State University
5Sensory Motor Adaptive Rehabilitation Technology (SMART) Network, University
of Alberta

###### Abstract

This study presents a speech-based motion planning strategy (SBMP) developed
for lower limb exoskeletons to facilitate safe and compliant human-robot
interaction. A speech processing system, finite state machine, and central
pattern generator are the building blocks of the proposed strategy for online
planning of the exoskeleton’s trajectory. According to experimental
evaluations, this speech-processing system achieved low levels of word and
intent errors. Regarding locomotion, the completion time for users with voice
commands was 54% faster than that using a mobile app interface. With the
proposed SBMP, users are able to maintain their postural stability with both
hands-free. This supports its use as an effective motion planning method for
the assistance and rehabilitation of individuals with lower-limb impairments.

Keywords: Human-robot interaction, speech processing, finite state machine,
central pattern generator, lower limb exoskeleton

##  1 Introduction

Assistive robotic systems can enhance the quality of life of people affected
by neurological impairments [1]. These systems include lower limb
exoskeletons, such as ReWalk [2], Indego [3], HAL [4], and Exo-H3 (used in
these experiments) [5], which are designed to rehabilitate individuals with
neurological impairments. In comparison with traditional physical therapies,
wearable exoskeletons allow users to interact more easily with their
environment, improving mobility and independence in non-ambulatory individuals
[1].

To facilitate safe and compliant human-robot interactions (HRIs), the
exoskeleton motion planning strategy should be intuitive and efficient to use
[6]. Social HRI, where humans use body language, gestures, and speech to
interact with robots, shows promise in addressing issues with physical
interactions through quick and efficient identification of user intentions
[7].

Although speech recognition (SR) is increasingly adopted for HRI, this
adoption is generally limited to humanoid robots [7, 8]. Nonetheless, the
integration of SR into other forms of robotics may improve their ergonomics
and practicality for human use. For example, SR reduces the need for
conventional interaction methods such as button-based interfaces or mobile
apps, allowing for hands-free use of an exoskeleton. This aspect of SR
benefits users who require their hands to grasp stability aids, such as
crutches or a walker. Furthermore, it increases safety in emergencies, where
users may find voice commands more intuitive than a tactile controller.
However, these interactions are only effective if the robot can perceive what
a user is saying.

Computational SR is a complex task that requires turning auditory information
into text. Over the last 40 years, there have been several breakthroughs that
have advanced the technology to the point where it can effectively analyze
speech and produce an accurate response even under adverse conditions [9]. In
particular, a neural network architecture known as the time delay neural
network factorization model has been shown to be effective at acoustic
modeling and speech perception [10, 11].

Natural language understanding (NLU) addresses the problem of interpreting
user intentions through two primary methods. The first method (true NLU) uses
a combination of keyword analysis, semantic processing, discourse processing,
and context analysis to determine the meaning and intent of speech [12].
However, this strategy is not always practical because it is computationally
intensive. The other primary method of intent determination involves creating
a series of phrases mapped to predetermined intents. This second method
(phrase mapping) is less complex and requires fewer computational resources to
execute than true NLU, making it suitable for real-time intent determination.
Once an NLU system has determined the user’s intent, it becomes possible to
translate high-level commands into a sequence of actions that can be performed
by a platform.

With high-level speech commands as an input, control systems should translate
wearer intents into low-level controllers, such as position, force, or
impedance controllers, to synchronize gait planning in a smooth and time-
continuous manner. Furthermore, high-level commands should be subject to
safety constraints to avoid sudden movements, which may lead to injuries.
Finite state machines (FSMs) address this motion planning problem by acting as
a central planner for transitions between exoskeleton states, such as standing
and walking [13]. Thus, FSMs may serve as an interface to translate high-level
user intents into motion plans for low-level controllers.

One bioinspired strategy for exoskeleton control is the central pattern
generator (CPG). The CPG consists of connected nodes that can generate
rhythmic patterns without receiving rhythmic inputs, facilitating joint motion
synchronization necessary to recapitulate rhythmic motor behaviours such as
bipedal locomotion [14, 15]. CPGs typically include parameters that allow for
modulation of locomotion, which provides additional control and flexibility
for the wearer [16, 17]. The ability of CPGs to generate time-continuous
rhythmic motions makes them an appropriate candidate for shaping the
trajectories of lower limb exoskeletons, and they have been investigated in
previous studies where joint synchronization was achieved through systems of
coupled differential equations [18, 19, 20, 21, 22].

In this study, a speech-based locomotion planning strategy combined social
HRI, FSM, and CPG for the intelligent motion planning of a lower limb
exoskeleton for bipedal locomotion. As opposed to a button-based interface,
voice input was used as the primary mode of determining user intent, allowing
hands-free use of an exoskeleton. These high-level commands were processed by
an FSM to ensure safe and natural state transitions before being executed in
low-level position controllers. The major contributions of the proposed
control scheme can be summarized as follows:

  * •

Integration of SR and a lower limb exoskeleton to create a system that allows
a user’s hands to be free to use mobility aids while still controlling the
exoskeleton.

  * •

The SR system used a combination of denoising, speech perception, and NLU
modules to perform SR. By using gated recurrent units for denoising alongside
probabilistic phrase mapping for NLU, we achieved an accurate SR system
capable of running on a low-power device in real-time without requiring a
remote machine to perform computation. This allows for a self-contained SR
platform that can move with a user, allowing for a broader application of
speech-based controls.

  * •

A novel set of CPG dynamics was proposed to synchronize time-continuous
transitions between exoskeleton locomotion states (e.g., sit, stand, walk) in
response to discrete user inputs. Speech inputs were processed through an FSM
alongside joint angles and velocities to streamline state transitions (e.g.,
speed up, slow down). Although voice-activated robotic systems have previously
been investigated [23, 24], previous CPG dynamics have not incorporated
speech-based inputs for lower limb exoskeletons [14].

The proposed three-stage motion planning scheme is outlined as follows: The
speech processing system is presented in Sec. 2, the CPG and motion planning
strategy is presented in Sec. 3, the experimental evaluations of the proposed
motion planning strategy are presented and discussed in Sec. 4, and the
concluding remarks are provided in Sec. LABEL:sec:conclusion.

##  2 Speech Processing System

The proposed speech processing system was a customized version of the pipeline
(Fig. 1) developed by Reverb Robotics [25], consisting of a denoising function
to increase the signal-to-noise ratio (SNR), a speech perception module to
process raw speech data, and a natural language understanding model to
determine speech intent. Importantly, the system was able to run on a
Raspberry Pi, which allowed for increased portability and for it to be applied
to mobile platforms. Once an intent was determined, the Pi sent the command
over a network to the central exoskeleton motion planner as shown in Fig. 1.
The overall system takes approximately 0.5 seconds process and send the
command to the exoskeleton control system. When factoring in network latency
this results in a 500-1000 ms delay between a command being spoken and
actuation of the exoskeleton.

Figure 1: Proposed strategy for speech-based planning of lower limb
exoskeletons.

###  2.1 Denoising Pipeline

The denoising function processed raw user inputs before passing the denoised
signal to the speech perception function, increasing the SNR with the
intention of improving the accuracy of the speech perception module in noisy
environments. Denoising was achieved by detecting voice activity, analyzing
its spectral signature to estimate noise, and subtracting its spectral
signature of the noise from the input audio to produce the denoised audio
(Fig. 2). We used RNNoise, a deep recurrent neural network, to denoise the
input audio for the SR system along with embeddings provided by the developers
trained on 140 hours of data. RNNoise used gated recurrent units to track
long-term patterns that were difficult to process with conventional recurrent
units [26]. For example, the denoising function detected fan noise and
subtracted that sound from speech. Moreover, when a user moved to a different
area without that noise, the function stopped removing that spectral signature
and adapted to remove any new noise.

Gated recurrent units function similarly to normal long-short term memory
recurrent units with the addition of two sigmoid-activated gates. These
activation gates control when the recurrent unit learns and forgets
information, addressing the vanishing gradient problem with classical
recurrent neural networks [27]. This feature of gated recurrent units allowed
the denoising function to remove persistent background noise from an input
signal while preserving speech information.

Raw Audio Noise Estimation Voice Activity Detection Spectral Subtraction
Denoised Speech Figure 2: Major functions for denoising voice activity.

###  2.2 Speech Perception Network

The speech processing system uses an implementation of the Vosk speech
perception system [28] to perceive speech and form words. The Vosk
implementation combines the time delay neural network factorization model and
a multi-stream convolutional neural network. When the time delay neural
network factorization architecture is combined with a multi-stream
convolutional neural network architecture that allowed for simultaneous
processing of several temporal windows, the result was a robust speech
perception model. We used a pre-trained set of open-source weights called
‘vosk-model-small-en’ provided by Alpha Cephei [29]. Additionally, we used a
constrained vocabulary to reduce the search space for SR and improve the
recognition of words in the vocabulary. This is implemented by simply creating
a list of words the network can perceive.

###  2.3 Natural Language Understanding

To address the problem of intent determination, the SR system used Snips NLU
[30] to create a list of intents along with phrases that trigger those
intents. Although a variety of phrases were associated with each intent,
rather than using exact phrase checking, the intent parser used a
probabilistic engine that focuses on keyword analysis, enabling the NLU engine
to associate perceived speech with the proper intent without that speech being
in the phrase list. Once an intent was extracted, the program determined
whether the speech was directed at the exoskeleton. We checked if the command
contained the word ‘robot’ to prevent the exoskeleton from performing an
unintended action. Thus, the specific key words or phrases to trigger a state
transition was ‘robot {keep moving, don’t change, maintain speed, stand up,
stand, sit down, sit, stop moving, stop, walk forward, walk, move forward,
move, forward, slow down, slow, speed up, go faster, faster},’ where one of
the key words or phrases in the set are used (Fig. 3).

##  3 Motion Planning and Control

###  3.1 Overview of the Exoskeleton Motion Planning System

The proposed exoskeleton locomotion motion planning strategy combined a speech
processing system, FSM, and CPG to ensure safe time-continuous transitions
between exoskeleton states (Fig. 1). Firstly, high-level voice commands were
processed through the Reverb Robotics speech processing system [25]. Then,
user intents were sent to the FSM of the exoskeleton. When a new intent was
received, the FSM sent an input to the CPG, which fed into a Fourier series to
generate joint trajectories subject to safety constraints. The period and
amplitude of the trajectory could be modulated via inputs to the FSM. Then,
these signals fed into position controllers, which controlled the exoskeleton.
Finally, exoskeleton joint angles and velocities fed back into the FSM in a
closed loop to ensure safe transitions between exoskeleton states.

Table 1: CPG parameters, initial conditions, and constants for the proposed motion planning scheme. For parameters and initial conditions, $l$ and $r$ refer to the left and right side of the exoskeleton, respectively. Parameters | Initial conditions | Constants
---|---|---
$v_{ij}=0.1$ | $A_{0}(k)=1$ | $c_{r}=2.5$
$\phi_{ll}=$0\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ | $\Omega_{0}(k)=$\pi/2\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ | $c_{\theta}=$2\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$
$\phi_{rr}=$0\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ | $\theta_{l}(0)=$(2+\pi)\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ | $\beta_{\omega}=10\pi$
$\phi_{lr}=$\pi\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ | $\theta_{r}(0)=$2\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ | $\beta_{r}=10\pi$
|  | $T=$2\text{\,}\mathrm{s}$$

###  3.2 Synchronization of Joint Trajectories

The proposed CPG dynamics for the phase $\theta_{i}(t)$ and amplitude $r(t)$
of the $i$th exoskeleton joint are governed by the following equations based
on the Kuramoto model for the synchronization of coupled oscillators [31].

$\begin{split}\dot{\theta}_{i}(t)&=\omega(t)+\sum_{j=1}^{N}v_{ij}\sin(\theta_{i}(t)-\theta_{j}(t)-\phi_{ij})\\\
\ddot{\omega}(t)&=\lambda(t)\beta_{\omega}\left(\frac{\beta_{\omega}}{4}(\Omega_{n}(k)-\omega(t))-\dot{\omega}(t)\right)\\\
\ddot{r}(t)&=\lambda(t)\beta_{r}\left(\frac{\beta_{r}}{4}(A_{n}(k)-r(t))-\dot{r}(t)\right)\end{split}$
(1)

where $N$ is the number of joints, $v_{ij}$ is the coupling strength,
$\phi_{ij}$ is the phase offset, $\beta_{\omega}$ and $\beta_{r}$ are fixed
constants, $\Omega_{n}(k)$ and $A_{n}(k)$ are user-adjustable constants which
modulate the frequency and amplitude of the system, respectively, and
$\lambda(t)$ is a user-triggered ramping system which multiplies the CPG
signal by a linear time-dependent gain. In particular, $\Omega_{n}(k)$ and
$A_{n}(k)$ update in response to user inputs, $k$ (Table 1).

$\begin{split}\Omega_{n}(k)&=\begin{cases}\Omega_{n-1}+c_{\theta},&k=\text{speed
up}\\\ \Omega_{n-1}-c_{\theta},&k=\text{slow down}\\\
\Omega_{n-1},&k=\text{otherwise}\end{cases}\\\
A_{n}(k)&=\begin{cases}A_{n-1}+c_{r},&k=\text{speed up}\\\
A_{n-1}-c_{r},&k=\text{slow down}\\\ A_{n-1},&k=\text{otherwise}\end{cases}\\\
\end{split}$ (2)

Here, $\Omega_{n}$ and $A_{n}$ are the updated speed constants, $\Omega_{n-1}$
and $A_{n-1}$ are the current speed constants, and $c_{\theta}$ and $c_{r}$
are constants that adjust the frequency and amplitude of the dynamics in (1).

The ramping system is defined as follows:

$\lambda(t)=\begin{cases}t/T,&\text{stand-to-walk}\\\ 1-t/T,&\text{walk-to-
stop}\\\ 1,&\text{walking}\\\ 0,&\text{otherwise}\end{cases}$ (3)

where the stand-to-walk and walk-to-stop conditions are triggered by user
commands and held for a period $T$ (Table 1).

Coupling between all joints is maintained by the same principal frequency of
$\omega(t)$ and amplitude $r(t)$ to synchronize locomotion trajectories.
Furthermore, the coupling expression in (1),
$v_{ij}\sin(\theta_{i}(t)-\theta_{j}(t)-\phi_{ij})$, returns the exoskeleton
joints to their original trajectories if the joint phases are perturbed.

The desired walking trajectory $q_{w_{i}}(t)$ for the joint $i$ of the
exoskeleton is defined as

$q_{w_{i}}(t)=r(t)\left(a_{0_{i}}+\sum_{k=1}^{N_{i}}(a_{k_{i}}\cos
k\theta_{i}(t)+b_{k_{i}}\sin k\theta_{i}(t))\right)$ (4)

where $a_{k_{i}}$ and $b_{k_{i}}$ are the coefficients of Fourier series with
$N_{i}$ terms (Table 2). The amplitude and phase of these oscillatory motions
are updated in real-time by $\theta_{i}(t)$ and $r(t)$. Trajectories for the
ankle, knee, and hip during walking were obtained from Subject 6 in the
experiments of Lencioni et al. [32].

Table 2: Coefficients of the Fourier series for the hip and knee from sitting, standing, and walking.∗ State | Joint | $a_{0}$ | $a_{1}$ | $a_{2}$ | $a_{3}$ | $a_{4}$ | $a_{5}$ | $a_{6}$ | $b_{1}$ | $b_{2}$ | $b_{3}$ | $b_{4}$ | $b_{5}$ | $b_{6}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Sitting/ Standing | Hip | $105.40$ | $-1.52$ | $1.29$ | $0.42$ | $0.36$ | $-0.04$ | $-0.12$ | $-3.86$ | $2.92$ | $0.24$ | $-0.49$ | $-0.28$ | $0.01$
Knee | $140.20$ | $-38.30$ | $-1.75$ | $0.55$ | $1.32$ | $0.14$ | n/a | $-29.22$ | $8.39$ | $4.14$ | $0.15$ | $-0.39$ | n/a
Ankle | $49.59$ | $22.01$ | $-1.31$ | $0.28$ | $-0.07$ | $0.15$ | n/a | $35.11$ | $-11.48$ | $-4.05$ | $-0.16$ | $0.40$ | n/a
Walk | Hip | $40.69$ | $23.22$ | $-4.49$ | $0.40$ | $0.70$ | $1.08$ | $-0.27$ | $-8.65$ | $3.34$ | $1.39$ | $0.80$ | $0.34$ | $0.07$
Knee | $25.70$ | $-3.83$ | $-8.54$ | $1.91$ | $1.09$ | $2.05$ | $-0.31$ | $-19.28$ | $17.93$ | $3.77$ | $1.50$ | $0.58$ | $-0.90$
Ankle | $-0.99$ | $5.29$ | $-8.58$ | $-0.48$ | $1.69$ | $-0.04$ | $1.30$ | $3.61$ | $-5.95$ | $5.92$ | $-2.12$ | $1.02$ | $-0.72$
${}^{*}R^{2}_{\text{adj}}\geq 0.99$ for each trajectory. |  |  |  |  |

The sit-to-stand and stand-to-sit trajectories for the ankle, knee, and hip
are obtained from the experiments of Nuzik et al. [33], which included motion
data for 55 healthy adults (38 women, 17 men). The mean trajectory values of
all subjects for each joint were used to compute the desired sit-to-stand and
stand-to-sit trajectory $q_{s_{i}}(t)$ for the joint $i$ of the exoskeleton.

$q_{s_{i}}(t)=a_{0_{i}}+\sum_{k=1}^{N_{i}}(a_{k_{i}}\cos
k\omega_{s_{i}}t+b_{k_{i}}\sin k\omega_{s_{i}}t)$ (5)

where $\omega_{s_{i}}$ is the angular frequency of the trajectory, and
$a_{k_{i}}$ and $b_{k_{i}}$ are the coefficients of the Fourier series with
$N_{i}$ terms (Table 2). Equation (5) was set to satisfy the time-dependent
boundary conditions $\dot{q}_{s_{i}}(t_{0})=0$ and $\dot{q}_{s_{i}}(T)=0$,
where $t_{0}$ is the initial time and $T$ is the time after one sit-to-stand
or stand-to-sit period (Table 1). The output of (5) was transformed into the
coordinate system of the exoskeleton via the linear function

$q_{t,s_{i}}(t)=-q_{s_{i}}(t)+\max(q_{s_{i}}(t))$ (6)

where $q_{t,s_{i}}(t)$ is the transformed angle. This function ensures that
the endpoint for the sit-to-stand trajectory for the ankle, knee, and hip
terminates at $0^{\circ}$. The stand-to-sit trajectory is implemented as the
reverse of the sit-to-stand trajectory (i.e., $q_{t,s_{i}}(t)\mapsto
q_{t,s_{i}}(-t)$ in equation 6).

###  3.3 Safety Considerations

The finite state machine was designed to ensure safe transitions between
exoskeleton states. The states and intents for triggering transition can be
seen in figure 3, and the initial state of the exoskeleton is either sitting
or standing, depending on the initial position of the user. Position
controller constraints include limitations on the maximum torque and velocity
and maximum and minimum joint angles.

Sit Stand Locomotion Initiation Locomotion Completion Walk
standsitwalkstopspeed upslow down Figure 3: Proposed FSM to plan the
transitions between exoskeleton states.

##  4 Experimental Evaluations

###  4.1 Speech Processing Experiments

Experimental assessments were performed to evaluate the efficacy of the SR
system with ethics permission from the University of Lethbridge Human Subjects
Review Board, number 2013-037. In the experiment, participants removed their
protective mask being worn due to public health restrictions and spoke a
series of pre-defined commands consistent with what would be used in normal
use of the exoskeleton. After each command, the output of the SR system and
NLU engine was recorded. Then, the participant put on their mask and repeated
the experiment. SR performance was measured with two metrics, word error rate
(WER) and intent error rate (IER). WER is a standard metric for speech
perception performance that is determined by the ratio of insertions ($I$),
substitutions ($S$), and deletions ($D$) required to transform a response into
the target to the number of words in the target $N$:

$\text{WER}=\frac{I+S+D}{N}\times 100\%$ (7)

IER is a measure of whether the correct intent was derived from the user’s
speech:

$\text{IER}=\frac{1}{N}\left(\sum_{\text{all
trials}}\begin{cases}0,&\text{correct output}\\\ 1,&\text{incorrect
output}\end{cases}\right)\times 100\%$ (8)

This metric has a similar function to the Stanford Sentiment Treebank (SST)
metric[34] that is a part of a standard NLU testing corpus General Language
Understanding Evaluation[35]. SST describes the accuracy of sentiment analysis
in a single sentence, though given the highly constrained nature of the model
used in our project, directly using SST would not provide an accurate
assessment of this model. Therefore IER was used for intent accuracy analysis.
After WER and IER were calculated for individual trials, the results were
aggregated to get a cumulative average for both metrics.

(a)

(b)

Figure 4: Results for the human SR trials for (a) WER and (b) IER. F$=$Female,
M$=$Male, C$=$Combined. There were no significant differences between or
within groups.

Experiments were performed by 4 male and 3 female participants. Participants
were taken from a population of undergraduate and graduate students at the
University of Lethbridge aged between 19 and 31. WER and IER scores from each
participant were aggregated and computed (Fig. 4).

One drawback of the SR system has to do with the intent recognition system. By
implementing a constrained vocabulary, words spoken by the user may be similar
enough to words in the vocabulary where an intent might be triggered. For
example, ‘hello’ may register as ‘slow’. We reduced this problem by checking
for the keyword ‘robot’ before intent determination. In locomotion
experiments, this feature resulted in the robot not executing commands when
the word ‘robot’ was missed, particularly when a user moved into a new
acoustic environment (participants moved between a quiet lab area and a room
with fume hood noise) where the denoising function was not updating its model
fast enough.

###  4.2 Locomotion Tasks With and Without Speech Commands

(a)

(b)

Figure 5: Exo-H3 lower limb exoskeleton worn by two able-bodied users for
walking. The participants use a wireless headset to command the exoskeleton.
(a) User 1 (21-year-old) and (b) user 2 (33-year-old).

The proposed motion planning strategy was assessed experimentally to provide
proof-of-concept evidence for the effectiveness of speech-based locomotion
planning with the Exo-H3 lower limb exoskeleton from Technaid. The proposed
motion planning scheme was implemented in real-time using MATLAB Simulink
which received sensory data and controlled motors at a sampling rate of
$100\text{\,}\mathrm{Hz}$ via a CAN interface (Vector VN1610) with 2 channels.
The SR system was run on a Raspberry Pi 3, which received speech input from
the microphone on a Logitech Wireless Headset H600 at a sampling rate of
$16\,000\text{\,}\mathrm{Hz}$, and the processed signals were transmitted to
the laptop via the user datagram protocol. The major computations in this
strategy involve time integration of the CPG dynamics in equation (1) and
calculating each joint angle for the exoskeleton with Fourier series in
equations (4), (5), and (6). Preliminary tests were performed in a trial-and-
error manner to obtain reasonable parameters for speed modulation
($c_{\theta}$ and $c_{r}$ in equation 2) and initial values and parameters of
the proposed CPG system (Table 1). In the experiments, two able-bodied users
wore the exoskeleton using a walker for postural stability (Fig. 5).

(a)

(b)

(c)

Figure 6: Actual and desired trajectories for the left (a) hip, (b) knee, and
(c) ankle for the A to B locomotion task for user 1. Vertical lines correspond
to the labelled speech inputs.

The first experiment involved walking from one location, A, to another
location, B, in a straight line for approximately $12\text{\,}\mathrm{m}$. The
participants began sitting at point A, then stood and walked to point B, where
they sat. Between points A and B, the user could choose to speed up or slow
down ad-lib. A 21-year-old participant, user 1, and a 33-year-old participant,
user 2, completed 9 and 12 trials, respectively. The time to walk in these
trials for user 1 was $56\pm 2.\text{\,}\mathrm{s}$ and that for user 2 was
$66\pm 2.\text{\,}\mathrm{s}$ ($P=0.0023$ for a two-tailed t-test assuming
equal variances, Fig. 8). The significant difference in time between users 1
and 2 to complete this experiment can be attributed to user preferences in
walking speeds. The trajectory for walking from point A to point B for one
representative trial of user 1 is plotted in Fig. 6 and that for user 2 is
plotted in Fig. 7. These figures show the response of the exoskeleton to voice
commands.

Figure 7: Desired trajectories for the left hip, knee, and ankle for the A to
B locomotion task for user 2. Vertical lines correspond to the labeled speech
inputs.

The same experiment from A to B was performed using a button-based smartphone
app from Technaid with user 1 as the participant for 10 trials, and the time
taken to walk from A to B was $113\pm 5.\text{\,}\mathrm{s}$. In comparison
with the voice command time of $62\pm 2.\text{\,}\mathrm{s}$, a two-tailed
t-test assuming equal variances yielded $P=4.0\times 10^{-13}$ (Fig. 8). This
significant difference can be attributed to the additional time the user
needed to stop the exoskeleton, remove their hands from their walker, search
for the appropriate command on the exoskeleton remote control, and press that
button.

Figure 8: Time taken by users 1 and 2 to complete a locomotion task using
voice control ($n_{1}=9,\ n_{2}=12$). Voice is the weighted mean time of users
1 and 2 to complete the task with voice control. Button is the mean time of
user 1 to complete the task with a button-based interface ($n=10$).

##  5 Conclusions

In this study, a speech-based locomotion planning strategy was developed and
tested to provide safe and intuitive motion planning of a lower limb
exoskeleton. The proposed strategy combined a speech processing system, FSM,
and CPG to plan the exoskeleton movements based on user intents. Furthermore,
a novel set of speech-based CPG dynamics was developed to synchronize
exoskeleton joints in a time-continuous manner in response to discrete user
inputs.

To test the performance of the proposed speech processing system, WER and IER
were computed as 10.29% and 12.24%, respectively, using speech data from 4
male and 3 female participants aged between 19 and 31 under both masked and
unmasked conditions. The IER demonstrated is relatively low when considering
limitations imposed by the hardware, as desktop computer-executed models have
shown 9.6-4.4% error rates in similar tests [36], though there is certainly
room for improvement. Overall the proposed SR system could be a practical
option for speech-based HRI applications when considering its performance in
constrained tasks and hardware it is capable of running on.

In the user studies with the exoskeleton, voice control was compared with
button control in a point-to-point locomotion task. The completion time using
voice commands was 54% faster than employing a button-based mobile app
interface, suggesting that voice commands may be more efficient than button-
based interfaces. In addition to being efficient, speech-based planning
allowed users to have both hands free to enhance postural stability and
support. Thus, the combination of safety and efficiency of speech-based
planning suggests it may be a promising candidate to plan the motion of lower
limb exoskeletons.

###  5.1 Future Directions

The results of this research provide direction for future studies. First, we
intend to test this system with individuals with disabilities as all
participants were able-bodied. Second, we would like to improve the denoising
function so it can better compensate for dynamic background noise. Finally, we
want to expand the FSM and implement personalized locomotion trajectories to
improve the real-world viability of the system.

## CRediT Authorship Contribution Statement

Eddie Guo: Conceptualization, methodology, software, experiments, and writing.
Chris Perlette: Methodology, software, experiments, and writing. Mojtaba
Sharifi: Conceptualization, methodology, experiments, writing, supervision,
review, and editing. Lukas Grasse: Software. Matthew Tata: Supervision,
review, and editing. Vivan K. Mushahwar: Supervision, review, and editing.
Mahdi Tavakoli: Supervision, review, and editing.

## Declaration of Competing Interests

Authors Matthew Tata and Lukas Grasse own Reverb Robotics and developed the
ReRo-Core software used as the basis for the speech recognition system.

## Funding

This work was supported by the Natural Sciences and Engineering Research
Council (NSERC), Canada Foundation for Innovation (CFI), Alberta Jobs, Economy
and Innovation Ministry’s Major Initiatives Fund to the Center for Autonomous
Systems in Strengthening Future Communities, and Autonomous Systems Initiative
(ASI).

## References

  * [1] A. Rodríguez-Fernández, J. Lobo-Prat, and J. M. Font-Llagunes, “Systematic review on wearable lower-limb exoskeletons for gait training in neuromuscular impairments,” _Journal of NeuroEngineering and Rehabilitation_ , vol. 18, 2021.
  * [2] G. Zeilig, H. Weingarden, M. Zwecker, I. Dudkiewicz, A. Bloch, and A. Esquenazi, “Safety and tolerance of the rewalk™ exoskeleton suit for ambulation by people with complete spinal cord injury: A pilot study,” _The Journal of Spinal Cord Medicine_ , vol. 35, no. 2, pp. 96–101, 2012\.
  * [3] S. A. Murray, R. J. Farris, M. Golfarb, C. Hartigan, C. Kandilakis, and D. Truex, “Fes coupled with a powered exoskeleton for cooperative muscle contribution in persons with paraplegia,” in _2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_ , 2018, pp. 2788–2792.
  * [4] O. Jansen, D. Grasmuecke, R. C. Meindl, M. Tegenthoff, P. Schwenkreis, M. Sczesny-Kaiser, M. Wessling, T. A. Schildhauer, C. Fisahn, and M. Aach, “Hybrid assistive limb exoskeleton hal in the rehabilitation of chronic spinal cord injury: Proof of concept; the results in 21 patients,” _World Neurosurgery_ , vol. 110, pp. e73–e78, 2018.
  * [5] K. A. Inkol and J. McPhee, “Assessing control of fixed-support balance recovery in wearable lower-limb exoskeletons using multibody dynamic modelling,” in _2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob)_ , 2020, pp. 54–60.
  * [6] V. Villani, F. Pini, F. Leali, and C. Secchi, “Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications,” _Mechatronics_ , vol. 55, pp. 248–266, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957415818300321
  * [7] T. B. Sheridan, “Human–robot interaction: Status and challenges,” _Human Factors_ , vol. 58, no. 4, pp. 525–532, 2016.
  * [8] J. Kennedy, S. Lemaignan, C. Montassier, P. Lavalade, B. Irfan, F. Papadopoulos, E. Senft, and T. Belpaeme, “Child speech recognition in human-robot interaction: Evaluations and recommendations,” in _Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_ , ser. HRI ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 82–90.
  * [9] X. Huang, J. Baker, and R. Reddy, “A historical perspective of speech recognition,” _Commun. ACM_ , vol. 57, no. 1, p. 94–103, jan 2014. [Online]. Available: https://doi.org/10.1145/2500887
  * [10] T. F. Abidin, A. Misbullah, R. Ferdhiana, M. Z. Aksana, and L. Farsiah, “Deep neural network for automatic speech recognition from indonesian audio using several lexicon types,” in _2020 International Conference on Electrical Engineering and Informatics (ICELTICs)_ , 2020, pp. 1–5.
  * [11] N. Moritz, J. Schröder, S. Goetze, J. Anemüller, and B. Kollmeier, “Acoustic scene classification using time-delay neural networks and amplitude modulation filter bank features,” _complexity_ , vol. 12, p. 13, 2016.
  * [12] M. Bates, “Models of natural language understanding,” _Proceedings of the National Academy of Sciences_ , vol. 92, no. 22, pp. 9977–9982, 1995.
  * [13] K. A. Strausser and H. Kazerooni, “The development and testing of a human machine interface for a mobile medical exoskeleton,” in _2011 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2011, pp. 4911–4916.
  * [14] A. J. Ijspeert, “Central pattern generators for locomotion control in animals and robots: A review,” _Neural Networks_ , vol. 21, no. 4, pp. 642–653, 2008, robotics and Neuroscience.
  * [15] J. Arcos-Legarda, J. Cortes-Romero, and A. Tovar, “Robust compound control of dynamic bipedal robots,” _Mechatronics_ , vol. 59, pp. 154–167, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S095741581930039X
  * [16] M. Sharifi, J. K. Mehr, V. K. Mushahwar, and M. Tavakoli, “Autonomous locomotion trajectory shaping and nonlinear control for lower-limb exoskeletons,” _IEEE/ASME Transactions on Mechatronics_ , vol. 27, no. 2, pp. 645–655, 2022.
  * [17] S. O. Schrade, Y. Nager, A. R. Wu, R. Gassert, and A. Ijspeert, “Bio-inspired control of joint torque and knee stiffness in a robotic lower limb exoskeleton using a central pattern generator,” in _2017 International Conference on Rehabilitation Robotics (ICORR)_ , 2017, pp. 1387–1394.
  * [18] M. Sharifi, J. K. Mehr, V. K. Mushahwar, and M. Tavakoli, “Adaptive cpg-based gait planning with learning-based torque estimation and control for exoskeletons,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 8261–8268, 2021.
  * [19] K. Gui, H. Liu, and D. Zhang, “A generalized framework to achieve coordinated admittance control for multi-joint lower limb robotic exoskeleton,” in _2017 International Conference on Rehabilitation Robotics (ICORR)_ , 2017, pp. 228–233.
  * [20] D. Zhang, Y. Ren, K. Gui, J. Jia, and W. Xu, “Cooperative control for a hybrid rehabilitation system combining functional electrical stimulation and robotic exoskeleton,” _Frontiers in Neuroscience_ , vol. 11, p. 725, 2017.
  * [21] J. K. Mehr, M. Sharifi, V. K. Mushahwar, and M. Tavakoli, “Intelligent locomotion planning with enhanced postural stability for lower-limb exoskeletons,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 7588–7595, 2021.
  * [22] X. Zhang and M. Hashimoto, “Synchronization-based trajectory generation method for a robotic suit using neural oscillators for hip joint support in walking,” _Mechatronics_ , vol. 22, no. 1, pp. 33–44, 2012. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957415811001723
  * [23] Y. Guo, W. Xu, S. Pradhan, C. Bravo, and P. Ben-Tzvi, “Personalized voice activated grasping system for a robotic exoskeleton glove,” _Mechatronics_ , vol. 83, p. 102745, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957415822000058
  * [24] X. Wang, P. Tran, S. Callahan, S. Wolf, and J. Desai, “Towards the development of a voice-controlled exoskeleton system for restoring hand function,” 04 2019, pp. 1–7.
  * [25] L. S. Grasse and M. S. Tata, “An end-to-end platform for human-robot speech interaction,” _Sound in HRI Workshop. 16th Annual Conference for Basic and Applied Human-Robot Interaction._ , Mar 2021.
  * [26] J.-M. Valin, “A hybrid dsp/deep learning approach to real-time full-band speech enhancement,” 2018.
  * [27] S. Hochreiter, “The vanishing gradient problem during learning recurrent neural nets and problem solutions,” _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ , vol. 06, no. 02, pp. 107–116, 1998. [Online]. Available: https://doi.org/10.1142/S0218488598000094
  * [28] K. J. Han, J. Pan, V. K. N. Tadala, T. Ma, and D. Povey, “Multistream cnn for robust acoustic modeling,” 2021.
  * [29] “Vosk models.” [Online]. Available: https://alphacephei.com/vosk/models
  * [30] A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, T. Lavril, M. Primet, and J. Dureau, “Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces,” _CoRR_ , vol. abs/1805.10190, 2018.
  * [31] F. A. Rodrigues, T. K. D. Peron, P. Ji, and J. Kurths, “The kuramoto model in complex networks,” _Physics Reports_ , vol. 610, pp. 1–98, 2016, the Kuramoto model in complex networks.
  * [32] T. Lencioni, I. Carpinella, M. Rabuffetti, A. Marzegan, and M. Ferrarin, “Human kinematic, kinetic and emg data during different walking and stair ascending and descending tasks,” _Scientific data_ , vol. 6, no. 1, Dec. 2019\.
  * [33] S. Nuzik, R. Lamb, A. VanSant, and S. Hirt, “Sit-to-Stand Movement Pattern: A Kinematic Study,” _Physical Therapy_ , vol. 66, no. 11, pp. 1708–1713, 11 1986.
  * [34] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013, pp. 1631–1642.
  * [35] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” 2018. [Online]. Available: https://arxiv.org/abs/1804.07461
  * [36] X. Liu, P. He, W. Chen, and J. Gao, “Multi-task deep neural networks for natural language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1901.11504