# The Change You Want to See(Now in 3D)

Ragav Sachdeva  Visual Geometry Group, Dept. of Engineering Science,
University of Oxford  Andrew Zisserman  Visual Geometry Group, Dept. of
Engineering Science, University of Oxford

###### Abstract

The goal of this paper is to detect what has changed, if anything, between two
“in the wild” images of the same 3D scene acquired from different camera
positions and at different temporal instances. The open-set nature of this
problem, occlusions/dis-occlusions due to the shift in viewpoint, and the lack
of suitable training datasets, presents substantial challenges in devising a
solution.

To address this problem, we contribute a change detection model that is
trained entirely on synthetic data and is class-agnostic, yet it is performant
out-of-the-box on real world images without requiring fine-tuning. Our
solution entails a “register and difference” approach that leverages self-
supervised frozen embeddings and feature differences, which allows the model
to generalise to a wide variety of scenes and domains. The model is able to
operate directly on two RGB images, without requiring access to ground truth
camera intrinsics, extrinsics, depth maps, point clouds, or additional before-
after images. Finally, we collect and release a new evaluation dataset
consisting of real-world image pairs with human-annotated differences and
demonstrate the efficacy of our method. The code, datasets and pre-trained
model can be found at: https://github.com/ragavsachdeva/CYWS-3D

Figure 1: On the left: An illustration of the problem we aim to address in
this work. Given an image pair of a 3D scene, with a significant shift in
camera pose, can we detect what has changed? On the right: Top prediction by
our model on images from the popular TV show The Big Bang Theory depicting the
missing DNA-model.

##  1 Introduction

From the way leaves rustle in the wind to the shifting patterns of clouds in
the sky, our world is in a constant state of flux. Yet detecting and
localising changes in complex 3D scenes remains a challenging task for
computer vision. Imagine being able to identify the changes between two images
of the same scene captured at separate moments in time and from different
viewpoints, as shown in Figure The Change You Want to See(Now in 3D). This is
the challenge we aim to address in this work. With applications in fields such
as robotics, facility monitoring, forensics and augmented reality, our work
has the potential to unlock new possibilities for understanding and
interacting with our dynamic world.

We formulate the problem we study as follows: Given a pair of 2D images of a
3D scene, captured with a significant shift in camera position and at
different temporal instances, we wish to localise the changes between them, if
any. In particular, we wish to capture everything that is physically different
in the regions that are visible in both the images while disregarding areas
that appear or disappear from view due to the shift in camera pose or
occlusion. This includes objects that may have been added or removed from the
scene, and text or decorations that may have been added to an object, but we
wish to ignore photometric differences such as a lighting change. Under this
setting, differentiating true changes from occlusions or dis-occlusions is
intrinsically ill-posed using 2D images alone. In other words, answering the
question “Is this object missing in the other image, hidden behind another
object, or simply out of frame?” fundamentally requires the 3D shape of the
scene, which is not directly available to us apriori. Furthermore, it is not
possible to compute the shape of the given 3D scene using two-view stereo
methods as they rely on corresponding points in the two images which are
inherently non-existent in case of changes such as missing objects.
Consequently, in the absence of the scene’s ground truth geometry, it is
theoretically infeasible to reason about the relative position, scale and
shape of the objects in the scene, and how or where they might appear when
observed from a different viewpoint (see Figure The Change You Want to See(Now
in 3D)). This, coupled with the lack of large-scale real-world datasets that
include image pairs of changing scenes captured from significantly different
viewpoints, makes devising a solution to this general two-view change
detection problem very challenging.

Nevertheless, our objective is to detect changes in “in the wild” real world
images with minimal constraints and operate on RGB images only, without
assuming access to ground truth geometry, depth, camera parameters, camera
poses etc. Our solution is to build on the insight that once the two views are
registered, it is relatively easy to determine what has changed. We thus
proceed in two stages: (1) register by warping the spatial features from one
image to the other, and this fundamentally needs to be in 3D; (2) determine
the differences by training a detector on top of these registered spatial
feature maps in order to identify the significant changes, and ignore
“nuisance” variables such as changes in lighting. Briefly, we first use an
off-the-shelf pre-trained visual backbone to extract spatial feature maps for
an RGB image pair. We then lift these 2D feature maps to 3D by making use of
state-of-the-art monocular depth estimation models, followed by a
differentiable feature registration module (DFRM) to align and render the
features maps back to 2D in the other view. Finally, we utilise a simple
detection head to process these features and output the changes. To overcome
the issue of lack of real-world training data, we train our model exclusively
on synthetic data with controlled 3D changes. In order to permit sim2real, we
keep the visual backbone frozen throughout training, and decode difference of
registered features.

Since the notion of change necessitates a pair of corresponding regions with
some differences, our formulation requires model predictions in both the
images. For instance, if a car is present in one image but missing in the
other, we expect the model to put bounding boxes around both – where the car
is, and where the car “should have been”. Furthermore, since the changes in
“in the wild” images are customarily open-set, our model is designed to be
class-agnostic. We demonstrate that a model trained in this fashion zero-shot
generalises to a wide variety of datasets including 2D scenes, 3D scenes,
synthetic and real-world images.

In the following we provide an overview of the existing literature (Sec. 2),
details of the proposed method (Sec. 3), experiments and results (Sec. 4), and
finally some concluding remarks (Sec. 5). We will release the code, datasets
and trained model.

##  2 Related Work

The problem of exploring visual changes has been studied in several different
flavours previously. In this section we loosely group these works into two
categories, 2D and 3D, and describe how our problem setting relates/differs
from them.

2D (-ish): A typical scenario for the change detection problem is one where we
have a pair of before-after RGB images, where the camera is either fixed i.e.
the images are related by an identity transformation (e.g. images from
surveillance cameras), there is a planar-scene assumption (e.g. bird’s eye
view or satellite images), or in the general case there is limited shift in
viewpoint (e.g. street scenes looking at distant objects/buildings), and the
model is expected to identify the changes between them. Most of the existing
works in the change detection literature belong to this category.

[18, 17, 21, 12] tackle the change captioning problem where the goal is to
describe the changes in an image pair in natural language. These methods
mainly evaluate their approach on the STD [10] (images from fixed video
surveillance camera), or CLEVR-based change datasets [18, 21, 12] (synthetic
images of 3D objects of primitive shapes). Since the problem these works
address is change captioning, their approaches do not deal with precise
localisation of changed regions.

Figure 2: DFRM: Given two images, we first obtain corresponding points and
depth maps which are used to estimate the 3D transformation $T_{r\rightarrow
q}$. Then given a feature map for each image along with the depth map, we
create a point cloud of $c$-dimensional feature vectors, warp it using the
estimated transformation and render it to 2D grid. Notice the black regions in
the rendered grid. These regions are dis-occluded and contain $c$-dimensional
$\boldsymbol{0}$ vectors. We obtain a soft visibility mask to counteract the
effect of dis-occluded regions when computing the difference.

[27, 1, 28, 14] tackle the change localisation problem, specifically for
street-view images where the goal is to produce segmentation masks for changed
regions. These methods mainly evaluate their approach on TSUNAMI [27]
(panaromic, street-view images), VL-CMU [1] (images of urban scenes with
macroscopic changes) and PCSD [28] (panoramic, street-view images). While
these datasets do not technically have the planar-scene assumption or a fixed
camera, the scenes are of distant buildings and there is no “peeking behind”
objects due to a shift in camera pose. Since pixel-wise change annotation is
expensive, these datasets often suffer from non-comprehensively labelled test
sets, limited to a set classes.

Recently, [26] proposed a class-agnostic method that tackles the change
detection problem for arbitrary images that are related by a homography
transformation, as a bounding-box based detection problem. They evaluate their
approach on STD [16], Kubric-Change [26] (3D objects resting on a 2D plane,
camera is not fixed but the images are captured from bird’s eye view) and
others.

Similar to the methods above, we also tackle the change detection problem in a
pair of RGB images. However, unlike previous methods our setting involves
general two-view images of 3D scenes (we do not assume fixed camera or planar
scenes and in our case there is a significant shift in camera pose) and we
particularly focus on making our model work on open-set, real-world images.
The problem formulation closest to ours is that of [26] in that [26] also
train a class-agnostic model that produces bounding boxes around changed
regions in both the images except they train and evaluate on 2D scenes.

3D: The change detection problem has also been studied explicitly for 3D data
(e.g. multiview before-after images, 3D scans etc.). [19] tackle the change
captioning problem in a 3D setting by assuming multi-view images are available
both before and after the change. [20] further propose an end-to-end framework
for describing scene changes from various input modalities, namely, RGB
images, depth images, and point cloud data. Recently, [22] proposed a task to
explicitly localise changes in the form of 3D bounding boxes from two point
clouds and describe detailed scene changes for a fixed classes of objects.
Unlike these works, we only assume access to a single-view image for both
before and after scenes and do not operate on explicit 3D data like point
clouds. Since RGB images are more readily available, it allows our method to
be easily applied to real-world scenes. The closest setting to our is of [5]
who tackle the change detection problem for general two-view images, except
their model is trained and tested on the same synthetic setting with 4 fixed
classes111Not available publicly at the time of writing..

##  3 Method

###  3.1 Overview

Given an image pair of a 3D scene, captured with a significant shift in camera
viewpoints, our goal is to localise the changes between them in the form of
bounding box predictions for each image. In particular, we only wish to
capture changes in the regions that are visible in both the images while
disregarding areas that appear or disappear from view due to the shift in
camera pose.

Our approach, which is overviewed in Figure 2, begins by extracting dense
spatial image descriptors from each image using a pre-trained transformer-
based visual backbone, which are then processed to obtain feature maps at
multiple spatial resolutions using a U-Net [24] style encoder. Next, in order
to reason between changes and occlusions/dis-occlusions, we need 3D
information. Inspired by [31], our DFRM “lifts” the features to 3D, and
differentiably registers and renders them. After obtaining registered
features, we compute their difference in order to identify what has changed.
Finally, we decode the difference of registered features using a U-Net style
decoder, followed by a bounding box prediction head. We next describe the
architecture, which is illustrated in Figure 3.

Figure 3: Architecture: Given two images, we first extract dense spatial
feature maps using a pre-trained visual backbone. Following this, a CNN-based
encoder is used to extract visual descriptors at multiple spatial resolutions.
We then use a differentiable feature registration module (DFRM) to warp
features from one image to another (such that they are registered), and take
their difference (only one resolution is shown for the purpose of
visualisation, however this operation is performed at multiple resolutions).
Finally, these difference of feature maps are processed by a CNN-based decoder
followed by a bounding box detection head. For brevity, we only show the
prediction for one of the images, however, the pipeline is symmetric for the
other. Please see Sec. 3 for more details.

###  3.2 Architecture

Backbone: Given two images $I^{1}\in\mathbb{R}^{3\times H\times W}$ and
$I^{2}\in\mathbb{R}^{3\times H\times W}$, we first encode $I^{1},I^{2}$ using
a pre-trained frozen visual backbone, represented by $\Phi_{B}(\cdot)$, to
obtain two sets of feature descriptors per image represented by
$f^{1}_{s},f^{1}_{d}=\Phi_{B}(I_{1})$ and
$f^{2}_{s},f^{2}_{d}=\Phi_{B}(I_{2})$. In practise, $\Phi_{B}(\cdot)$ is the
ViT-B/8 [6] architecture, where $f_{s}$ and $f_{d}$ represent shallow and deep
features.

U-Net Encoder: We process $f^{1}_{d},f^{2}_{d}$ using a U-Net style encoder,
represented by $\Phi_{E}(\cdot)$, to obtain
$g^{1}_{n}=\Phi_{E}(f^{1}_{d})_{n}$ and $g^{2}_{n}=\Phi_{E}(f^{2}_{d})_{n}$
after each downsampling block $n$, resulting in a set of multi-resolution
feature maps $G^{1}=\\{f^{1}_{d},g^{1}_{1},g^{1}_{2},g^{1}_{3},g^{1}_{4}\\}$
and $G^{2}=\\{f^{2}_{d},g^{2}_{1},g^{2}_{2},g^{2}_{3},g^{2}_{4}\\}$, for image
$I^{1}$ and $I^{2}$ respectively.

Feature Registration and Difference: Given $G^{1},G^{2}$, we use a
differentiable feature registration module (described in Sec. 3.3) to obtain a
warp of feature maps at each spatial resolution such that the original
features of one image are registered with the warped features of the other
image. We then compute their element-wise difference and mask out
occluded/dis-occluded regions. Specifically, for feature maps at spatial
resolution $i$, we obtain the feature map $H^{1}_{i}=v_{2\rightarrow
1}(G^{1}_{i}-\tau_{2\rightarrow 1}(G^{2}_{i}))$ and $H^{2}_{i}=v_{1\rightarrow
2}(G^{2}_{i}-\tau_{1\rightarrow 2}(G^{1}_{i}))$, for $I^{1},I^{2}$
respectively, where $\tau_{r\rightarrow q}$ represents the 3D feature warp
operator from image $I^{r}$ to $I^{q}$, and $v_{r\rightarrow q}$ represents
the soft visibility mask.

U-Net Decoder: Following this, we decode the set of feature maps $H^{1}$ and
$H^{2}$ using a U-Net decoder modulated with scSE blocks [25], represented by
$\Phi_{D}(\cdot)$ to produce feature maps $k^{1}$ and $k^{2}$ respectively.

Bbox Head: Finally, feature maps $[f^{1}_{s}\mathbin{\|}k^{1}]$ and
$[f^{2}_{s}\mathbin{\|}k^{2}]$, where $[\ \mathbin{\|}\ ]$ is the
concatenation operation (along channel dimension), are fed into a CenterNet
head [32], which minimises the detection loss function as described in [32],
to produce bounding boxes around changed regions in both the images. The
motivation for concatenating shallow features $f_{s}$ with $k$ is that it
serves as the final skip connection before the prediction head, which is
consistent with the typical U-Net style model, and additionally shallow
features are known to capture more positional information, as opposed to deep
features which capture more semantic information [2], and therefore can help
with localisation.

###  3.3 Differentiable Feature Registration Module (DFRM)

Given images $I^{r},I^{q}\in\mathbb{R}^{3\times H\times W}$, along with their
feature maps $f^{r},f^{q}\in\mathbb{R}^{c\times h\times w}$ respectively, we
use the following three-step process, represented by $\tau_{r\rightarrow
q}(\cdot)$, to warp $f^{r}$ such that $f^{q}$ and $\tau_{r\rightarrow
q}(f^{r})$ are registered. See Figure 2 to visually conceptualise all the
moving parts.

Step 1: Estimate a 3D linear transformation. In order to register the two
feature maps, we must estimate a 3D transformation between the two images. To
do so, first we obtain a set of $n$ corresponding points
$P^{r},P^{q}\in\mathbb{R}^{n\times 2}$ (in normalised coordinates) in image
$I^{r},I^{q}$ respectively, using a correspondence extractor represented by
$\mathcal{C}(\cdot)$. Following this, we estimate the depth maps
$D^{r},D^{q}\in\mathbb{R}^{3\times H\times W}$ using a monocular depth
estimator represented by $\mathcal{D}(\cdot)$. Finally, to estimate the
transformation, we first back-project each point $(x_{j},y_{j})\in
P^{r},P^{q}$ as,

$\hat{P_{j}}=\begin{bmatrix}d_{j}x_{j}&d_{j}y_{j}&d_{j}&1\end{bmatrix}^{T}$
(1)

where $d_{j}$ is the depth value at point $P_{j}$, resulting in two sparse 3D
point clouds $\hat{P^{r}},\hat{P^{q}}\in\mathbb{R}^{n\times 4}$ in homogenous
coordinates. Following this, we estimate a transformation matrix
$T_{r\rightarrow q}\in\mathbb{R}^{4\times 4}$, such that $T_{r\rightarrow
q}\hat{P^{r}_{j}}\approx\hat{P^{q}_{j}}$, using the following closed-form
solution:

$T_{r\rightarrow q}=\left(\hat{P^{r}}^{+}\hat{P^{q}}\right)^{T}$ (2)

where $\hat{P^{r}}^{+}\in\mathbb{R}^{4\times n}$ is the Moore-Penrose inverse
of $\hat{P^{r}}$.

Step 2: Lift the features to 3D and warp. For each $c$-dimensional feature
vector in $f^{r}\in\mathbb{R}^{c\times h\times w}$, we project its normalised
2D grid coordinates $(x_{j},y_{j})$ to a point
$(\begin{smallmatrix}x_{j}^{{}^{\prime}}/k,&y_{j}^{{}^{\prime}}/k,&z_{j}^{{}^{\prime}}/k\end{smallmatrix})$
in 3D, where

$\begin{bmatrix}x_{j}^{{}^{\prime}}&y_{j}^{{}^{\prime}}&z_{j}^{{}^{\prime}}&k\end{bmatrix}^{T}=T_{r\rightarrow
q}\begin{bmatrix}d_{j}x_{j}&d_{j}y_{j}&d_{j}&1\end{bmatrix}^{T}$ (3)

where $d_{j}$ is the estimated depth value of this point, resulting in a 3D
point cloud of $f^{r}$ feature vectors that are aligned with $f^{q}$.

Step 3: Differentiable feature rendering. Given the 3D point cloud, we render
it to the 2D grid using a differentiable renderer $\mathcal{R}(\cdot)$.
However, instead of rendering RGB colours, we render $c$-dimensional feature
vectors for each point. For this, we employ a differentiable point cloud
renderer, as in [31]. The advantages of this differentiable point cloud
renderer are two fold: (1) it solves the “small neighbourhood” problem,
wherein each feature-point projects to only one or a few pixels in the
rendered view, by splatting the points to a disk of controllable size, and (2)
the “hard z-buffer” problem, wherein each rendered pixel is only affected by
the nearest point in the z-buffer, by accumulating the effects of $K$ nearest
points. Both of these allow for better gradient propagation during training.
In addition to obtaining the rendered features, we also obtain a visibility
mask $v_{r\rightarrow q}$ to deal with occluded/dis-occluded regions (see
Figure 2). This is obtained by setting the 2D coordinates of the rendered
points to $1$ on a grid initialised with $0$s. However, due to the splatting
behaviour of the renderer, the visibility mask obtained is soft and not
binary.

Alternate registration strategies: Since DFRM is training-parameter free,
during inference $\tau_{r\rightarrow q}(\cdot)$ may utilise alternate
registration strategies. For instance, if ground truth depth is known for each
image, it can directly replace $D^{r},D^{q}$ obtained using the monocular
depth estimator $\mathcal{D}(\cdot)$. In addition to ground truth depth, if
the camera intrinsics and extrinsics are known, points can be directly back-
projected and warped using relative camera poses without needing to estimate
the transformation $T_{r\rightarrow q}$. Furthermore, if only $D^{r}$ is
available, Perspective-n-Point methods [9] can be used to warp features
$f^{r}$ onto $f^{q}$.

On the other hand, if it is known apriori that the images are of planar
scenes, depth is not needed and each 2D grid coordinate can be warped using
$T_{r\rightarrow q}\in\mathbb{R}^{3\times 3}$ which can either be supplied or
estimated using standard homography estimation methods. If the scene consists
of multiple planes (e.g. floor and wall), it may also be possible to obtain
desired results using a multi-grid [15] or multi-plane homography estimation
[9] to register the images.

###  3.4 Details and discussion

The ViT-B/8 [6] backbone $\phi_{B}(\cdot)$ is initialised with DINO model
weights [4]. DINO features are known to encode powerful high level semantic
information at fine spatial granularity and recent works have shown that
scalar product of DINO features can be used to compute high quality semantic
correspondences [2]. Orthogonally to computing correspondences, we utilise
these powerful DINO features to compute changes. To prevent the model from
overfitting to synthetic data and corrupting the quality of DINO features, we
keep the backbone frozen during training. In order to allow sim2real, our
decoder only ever operates on the difference of features, which is negatively
proportional to the scalar product i.e. the higher the scalar product the
smaller the difference, and therefore we only consider the similarity of
features (or the lack of) rather than the features themselves (whether
synthetic or real). Following [2], we increase the resolution of model
features by changing the stride of the patch extraction (from 8 to 4) and
adjusting the positional encoding appropriately via interpolation, allowing us
to operate on more granular spatial features. The shallow features and deep
features $f_{s},f_{d}$ are extracted from the keys of the Multiheaded Self-
Attention layers of the third and the last block respectively, an insight also
from [2].

For correspondence extractor $\mathcal{C}(\cdot)$, we experimented with [11,
30] but found SuperGlue [29] to work the best at generating high quality dense
correspondences and at high inference speed. After extracting 2D
correspondences from SuperGlue, we filter out outliers using RANSAC. Finally,
for the monocular depth estimator $\mathcal{D}(\cdot)$, we tried [23, 7] but
found the recently released ZoeDepth [3] to consistently produce better
results.

Figure 4: Qualitative results: We show the bounding box predictions in yellow
(solid) of our model on all the test sets, along with the ground truth in blue
(dashed). Figure 5: Failure cases: Here we show some RC-3D images where our
model failed to localise the correct changed region. We classify an output as
a failure if the top-5 most confidence bounding boxes do not contain the
ground truth changed region. Predictions are in yellow (solid), ground truth
is in blue (dashed).

##  4 Experiments

This section describes the data we used to train our model, evaluation
benchmarks and baselines, and various implementation details. Please see Table
1 for an overview of the datasets used and Figure 4 for some example images.

###  4.1 Training datasets

Due to the lack of existing large-scale real-world datasets for the change
detection problem, as formulated in this work, we fall-back to training our
model entirely on synthetic data. Specifically, we train our model jointly on
the following two datasets:

KC-3D: Similar to [26], we make use of the Kubric dataset generator [8] to
curate 86407 image pairs (of which 4548 images pairs are for validation) of 3D
scenes with controlled changes. The scenes consist of randomly selected set of
3D objects spawned at random locations on a randomly textured plane, where we
iteratively remove the objects and capture “before” and “after” image pairs.
However, unlike [26] where they capture birds-eye view images only, we capture
images from a wide variety of camera poses inside a cylindrical space around
the objects. In addition, we also capture the depth maps, camera intrinsics
and extrinsics for each image to supply to DFRM during training.

COCO-Inpainted: While KC-3D captures the underlying challenges of our task in
terms of viewpoint shift quite well, it lacks diversity in terms of kinds and
sizes of objects. Therefore, we additionally utilise the recently introduced
COCO-Inpainted dataset [26] for training. While this is a 2D dataset, where
the image pairs are perturbed by an affine transformation, it helps the model
learn to predict changes of various kinds and sizes.

###  4.2 Testing datasets

To test the performance of our model, we evaluate it on the following test
sets.

KC-3D: Following the same pipeline as before, we curate an additional 4548
image pairs of 3D scenes from Kubric for testing purposes (making the total
number of image pairs in KC-3D to be 90955).

RC-3D: To quantify our model’s capacity to generalise to real-world images, we
manually collect and label a small-scale test set consisting of 100 images
pairs, capturing a diverse set of common objects found in everyday places like
office, kitchen, lounge etc. The images were captured using a handheld Apple
iPad Pro (4th Gen) and Apple iPhone 14 Pro, which come with a built-in LiDAR
giving us aligned RGB-D images.

Cyws Test Sets: Since the problem we are tackling subsumes the change
detection problem in planar scenes, we also evaluate our model on 2D test
datasets proposed in [26], namely COCO-Inpainted, VIRAT-STD, Kubric-Change,
and Synthtext-Change.

Test set | COCO-Inpainted | Synthtext-Change | VIRAT-STD | Kubric-Change | KC-3D | RC-3D
---|---|---|---|---|---|---
type | inpainting | text | surveillance | sim | sim | general 2-view
change | synthetic | synthetic | real | synthetic | synthetic | real
geometry | affine | identity | identity | 2D-ish (bird’s eye) | 3D | 3D
# train images | 60000+ | - | - | - | 81859 | -
# test images | 4408 | 5000 | 1000 | 1605 | 4548 | 100

Table 1: Datasets. KC-3D and RC-3D are ours. Others are from [26].

| test dataset | COCO-Inpainted | VIRAT-STD | Synthtext-Change | Kubric-Change | KC-3D | RC-3D
---|---|---|---|---|---|---|---
small | medium | large | all
|  | depth | Const. | Const. | Const. | Const. | GT | GT | Est.
| registration | GT | GT | Est. | Id. | Id. | Est. | GT | Est. | Est.
method | training data | (Const. = Constant, Id. = identity, Est. = Estimated, GT = Ground Truth)
cyws | coco-inpainted | 0.46 | 0.79 | 0.85 | 0.63 | 0.65 | 0.89 | 0.76 | 0.13 | 0.12
cyws | coco-inpainted + KC-3D | 0.41 | 0.73 | 0.78 | 0.57 | 0.54 | 0.87 | 0.76 | 0.87 | 0.14
ours | coco-inpainted | 0.34 | 0.69 | 0.76 | 0.52 | 0.51 | 0.46 | 0.85 | 0.84 | 0.14 | 0.10 | 0.35 | 0.27
ours | KC-3D | 0 | 0.03 | 0.06 | 0.02 | 0.02 | 0.01 | 0.01 | 0.23 | 0.83 | 0.69 | 0.19 | 0.19
ours | coco-inpainted + KC-3D | 0.36 | 0.72 | 0.77 | 0.53 | 0.52 | 0.49 | 0.84 | 0.84 | 0.82 | 0.68 | 0.50 | 0.41

Table 2: Results: We report the AP of cyws [26] and our model on test sets
described in Sec 4.2.

###  4.3 Baseline and metrics

To the best of our knowledge, no prior works have tackled the change detection
problem in a general two-view and class-agnostic setting like us which makes
it difficult to directly compare our work with prior art. Nevertheless, we use
cyws [26] as a baseline as their formulation is the same as ours, except
restricted to 2D transformations. To allow for a fair comparison, in addition
to reporting the results using their open-sourced model, we also finetune
their pretrained model on the same training dataset as ours for $100$ epochs
(best model is picked using lowest loss on val set). We use the average
precision metric to report our results, similarly to [26].

###  4.4 Training details

We trained the model on 4$\times$ A40 GPUs for 50 epochs using the DDP
strategy with a batch size of 16, where the best model is chosen using a
validation set (COCO-Inpainted val set as in [26] \+ a KC-3D val set). Images
were augmented with CropAndResize, HorizontalFlips and ColourJittering, and
resized to $224\times 224$ due to the DINO-ViT [4] requirements. Since our
training data is entirely synthetic, during training we use ground truth data
for $\tau_{r\rightarrow q}(\cdot)$ rather than estimating it. The overall
objective was optimised using Adam [13] with learning rate of 0.0001 and
weight decay of 0.0005.

###  4.5 Results

We evaluate our model, which is trained on synthetic data described in Sec.
4.1, on a diverse set of testing datasets as described in Sec. 4.2 with no
further training/finetuning. Table 2 contains the quantitative results in
terms of average precision, while we show some qualitative predictions of our
model in Figure 4 and some failure cases in Figure 5.

3D scenes (KC-3D, RC-3D): From Table 2 it is evident that our model produces
impressive results on both synthetic and “in the wild” real-world images.
Particularly in the case of RC-3D, the model produces almost 4$\times$ better
results than the baseline and performs well even in the most challenging
setting when only RGB image pairs are available as input. Despite only having
been trained on synthetic data, the remarkable performance of the model on
real-world images validates the design choice of only using feature
differences and not features directly (unlike cyws, where their co-attention
module concatenates the cross-attended features). On the other hand, we
observe a surprising result from the cyws model, which is able to produce
impressive results on KC-3D dataset when finetuned on it. It is likely that
the cyws model is “over-fitting” to the Kubric setting given that the set of
objects and scenes in Kubric [8] are limited (different scenes just have
different random combinations from the same set of objects and backgrounds).
Despite this fact, it is still interesting that the model is able to reason
about changes despite not being 3D aware. Nevertheless, this performance does
not generalise to real-world images as observed by its poor results on RC-3D.

2D/fixed-camera scenes (Cyws test sets): In the 2D setting, we found that our
model is often comparable but not strictly better than cyws. In particular, we
observed that our model particularly struggles with detecting really small
changes in comparison to cyws which is likely due to the fact that cyws
operates at higher input resolution than us ($256\times 256$ vs $224\times
224$) and that the number of trainable parameters in our model is 31.5M which
is much less than 49.5M in cyws. Furthermore, we found that a lot of the
annotations (for COCO-Inpainted small and VIRAT-STD), are extremely small
(handful of pixels, almost indiscernible to human-eye). This becomes
problematic for our model when the input resolution is reduced to $224\times
224$. In addition, the ground-truth annotations for VIRAT-STD are noisy (both
false positives and missing annotations, see Figure 4 where our model predicts
valid changes that are missing from ground truth).

###  4.6 Limitations

Despite the remarkable ability of our model to localise changes in real-world
general two-view images, it suffers from a few limitations. A potential
concern is the large size of the model. While the number of trainable
parameters is 31.5M (which is less than 49.5M in cyws), accounting for the
frozen DINO-ViT backbone [4] with 85.8M parameters, our total model size is
roughly 117M parameters. This makes it much slower to train and infer from
than cyws. However, it must be acknowledged that this large backbone comes
with the added benefit of making our model generalisable to real-world images
even though it has only been trained on synthetic data. Another potential
cause of concern is that it relies on a good estimated registration, which in
turn relies on reliable correspondences and depth, which may not be always
available. On the other hand, it also means that as better correspondence
extractors and monocular depth estimators become available, our model’s
results can improve at no additional cost.

##  5 Conclusion

In the ever-changing landscape of our world, the task of detecting changes in
a 3D scene is daunting for both humans and machines alike. In this work we
take a step closer towards solving this problem by automatically detecting
changes in real-world images captured from significantly different viewpoints.
Due to the lack of large-scale real-world training datasets for this problem,
we propose a model that is trained entirely on synthetic data but can
generalise to real-world scenes by leveraging the recent advances in Following
previous works, we largely focused on detecting missing objects as they are
easy to acquire and precise. However, we note that our model should not be
confused with a “(missing) object detector”. While it is trained on object-
centric datasets, our model can zero-shot detect all sorts of open-set
changes.
Acknowledgements: We would like to thank Jaesung Huh and Robert McCraith for
their assistance with data collection, and Luke Melas-Kyriazi and Jaesung Huh
for proof-reading the paper. This research is supported by EPSRC Programme
Grant VisualAI EP/T028572/1 and a Royal Society Research Professorship
RP\R1\191132.

## References

  * [1] Pablo F. Alcantarilla, Simon Stent, German Ros, Roberto Arroyo, and Riccardo Gherardi.  Street-view change detection with deconvolutional networks.  In Proceedings of Robotics: Science and Systems, AnnArbor, Michigan, June 2016.
  * [2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep vit features as dense visual descriptors.  ECCVW What is Motion For?, 2022.
  * [3] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth, 2023\.
  * [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [5] Kento Doi, Ryuhei Hamaguchi, Yusuke Iwasawa, Masaki Onishi, Yutaka Matsuo, and Ken Sakurada.  Detecting object-level scene changes in images with viewpoint differences using graph matching.  Remote Sensing, 14(17):4225, 2022.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [7] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir.  Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10786–10796, 2021.
  * [8] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al.  Kubric: A scalable dataset generator.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749–3761, 2022.
  * [9] Richard I. Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * [10] Harsh Jhamtani and Taylor Berg-Kirkpatrick.  Learning to describe differences between pairs of similar images.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
  * [11] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi.  Cotr: Correspondence transformer for matching across images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6207–6217, 2021.
  * [12] Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyunsung Park, and Gunhee Kim.  Agnostic change captioning with cycle consistency.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2095–2104, 2021.
  * [13] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [14] Yinjie Lei, Duo Peng, Pingping Zhang, Qiuhong Ke, and Haifeng Li.  Hierarchical paired channel fusion network for street scene change detection.  IEEE Transactions on Image Processing, 30:55–67, 2021.
  * [15] L Nie, C Lin, K Liao, S Liu, and Y Zhao.  Depth-aware multi-grid deep homography estimation with contextual correlation. arxiv 2021.  arXiv preprint arXiv:2107.02524.
  * [16] Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al.  A large-scale benchmark dataset for event recognition in surveillance video.  In CVPR 2011, pages 3153–3160. IEEE, 2011.
  * [17] Ariyo Oluwasanmi, Enoch Frimpong, Muhammad Umar Aftab, Edward Y. Baagyere, Zhiguang Qin, and Kifayat Ullah.  Fully convolutional captionnet: Siamese difference captioning attention model.  IEEE Access, 7:175929–175939, 2019.
  * [18] Dong Huk Park, Trevor Darrell, and Anna Rohrbach.  Robust change captioning.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4623–4632, 2019.
  * [19] Yue Qiu, Yutaka Satoh, Ryota Suzuki, Kenji Iwata, and Hirokatsu Kataoka.  3d-aware scene change captioning from multiview images.  IEEE Robotics and Automation Letters, 5(3):4743–4750, 2020.
  * [20] Yue Qiu, Yutaka Satoh, Ryota Suzuki, Kenji Iwata, and Hirokatsu Kataoka.  Indoor scene change captioning based on multimodality data.  Sensors, 20(17):4761, 2020.
  * [21] Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh.  Describing and localizing multiple changes with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1971–1980, 2021.
  * [22] Yue Qiu, Shintaro Yamamoto, Ryosuke Yamada, Ryota Suzuki, Hirokatsu Kataoka, Kenji Iwata, and Yutaka Satoh.  3d change localization and captioning from dynamic scans of indoor scenes.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1176–1185, 2023.
  * [23] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179–12188, 2021.
  * [24] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [25] Abhijit Guha Roy, Nassir Navab, and Christian Wachinger.  Concurrent spatial and channel ‘squeeze & excitation’in fully convolutional networks.  In International conference on medical image computing and computer-assisted intervention, pages 421–429. Springer, 2018.
  * [26] Ragav Sachdeva and Andrew Zisserman.  The change you want to see.  In Winter Conference on Applications of Computer Vision (WACV), 2023\.
  * [27] Ken Sakurada and Takayuki Okatani.  Change detection from a street image pair using cnn features and superpixel segmentation.  pages 61.1–61.12, 01 2015.
  * [28] Ken Sakurada, Mikiya Shibuya, and Weimin Wang.  Weakly supervised silhouette-based semantic scene change detection.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6861–6867, 2020.
  * [29] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [30] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922–8931, 2021.
  * [31] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7467–7477, 2020.
  * [32] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.


ADLshort=ADL,long=Activities of Daily Living BCIshort=BCI,long=Brain Computer
Interface BMIshort=BMI,long=Brain Machine Interface CoMshort=CoM,long=Center
of Mass CSshort=CS,long=Constant Strain COMshort=COM,long=Center of Mass
DOFshort=DOF,long=Degrees of Freedom EEGshort=EEG,long=Electroencephalography
EOMshort=EOM,long=Equations of Motion ERDshort=ERD,long=Event-Related
Desynchronization ERSshort=ERS,long=Event-Related Synchronization
GBNshort=GBN,long=Generalized Binary Noise HMIshort=HMI,long=Human Machine
Interface HSAshort=HSA,long=Handed Shearing Auxetic LDAshort=LDA,long=Linear
Discriminant Analysis PCSshort=PCS,long=Piecewise Constant Strain
RMSEshort=RMSE,long=Root Mean-Squared Error

# Guiding Soft Robots with Motor-Imagery Brain Signals and Impedance Control

Maximilian Stölzle∗,1, Sonal Santosh Baberwal∗,2, Daniela Rus3, Shirley
Coyle†,2, and Cosimo Della Santina†,1 ∗Authors contributed equally, †Authors
supervised equally.The work by Maximilian Stölzle was supported under the
European Union’s Horizon Europe Program from Project EMERGE - Grant Agreement
No. 101070918. The work by SS. Baberwal was supported by a grant from Science
Foundation Ireland under Grant numbers 18/CRT/6183, SFI/12/RC/2289 P21M.
Stölzle, and C. Della Santina are with the Cognitive Robotics department,
Delft University of Technology, Mekelweg 2, 2628 CD Delft, Netherlands
{M.W.Stolzle<EMAIL_ADDRESS>Baberwal and S. Coyle are with
the School of Electronics Engineering, Dublin City University, Dublin,
Ireland<EMAIL_ADDRESS><EMAIL_ADDRESS>Rus is with the
MIT Computer Science and Artificial Intelligence Laboratory (CSAIL),
Massachusetts Institute of Technology, Cambridge, MA 02139 USA,
<EMAIL_ADDRESS>
###### Abstract

Integrating Brain-Machine Interfaces into non-clinical applications like robot
motion control remains difficult - despite remarkable advancements in clinical
settings. Specifically, EEG-based motor imagery systems are still error-prone,
posing safety risks when rigid robots operate near humans. This work presents
an alternative pathway towards safe and effective operation by combining
wearable EEG with physically embodied safety in soft robots. We introduce and
test a pipeline that allows a user to move a soft robot’s end effector in real
time via brain waves that are measured by as few as three EEG channels. A
robust motor imagery algorithm interprets the user’s intentions to move the
position of a virtual attractor to which the end effector is attracted, thanks
to a new Cartesian impedance controller. We specifically focus here on planar
soft robot-based architected metamaterials, which require the development of a
novel control architecture to deal with the peculiar nonlinearities - e.g.,
non-affinity in control. We preliminarily but quantitatively evaluate the
approach on the task of setpoint regulation. We observe that the user reaches
the proximity of the setpoint in 66% of steps and that for successful steps,
the average response time is 21.5s. We also demonstrate the execution of
simple real-world tasks involving interaction with the environment, which
would be extremely hard to perform if it were not for the robot’s softness.

###### Index Terms:

Soft Robots, Brain Machine Interface, Model-based Control, HSA robots

##  I Introduction

BMI [1] facilitate the translation of neural activity into actionable
commands, enabling individuals to control external devices and systems through
their thoughts and attention [2, 3]. Compared to traditional bulky EEG setups
[4], one of the emerging avenues towards practical and wearable EEG devices
are systems based on motor imagery signals due to their intuitiveness and no
external dependency on (e.g., visual) stimuli. These have been use in stroke
rehabilitation [5]. Several works in literature have considered using this
technology to control robot manipulators [6, 7, 8].

However, the state-of-the-art classifiers on few-channel, online EEG signals
are still limited in achieving an accuracy of 65-75 $\mathrm{\char 37\relax}$
[9, 8] and are prone to producing outliers, which make it very challenging to
operate robots safely and robustly using these techniques [1]. In (rigid)
robotics literature, this has been addressed by relying on force-based (i.e.,
impedance) control [6] and by making the robot’s behavior more predictable
[7]. In this work, we follow a different path and investigate embodying safety
by pairing soft robots [10, 11] with BMI. This way, risks can be mitigated,
and more natural interactions with an unstructured environment can be achieved
by relying on structural compliance.

Figure 1: Scheme of the proposed approach to control HSA robots with motor
imagery. Brain signals steer an attractor in operational space: first, we
switch the active coordinate axis when we detect jaw clenching. If no jaw
clenching is detected, we classify the EEG signals based on left/right motor
imaginations into positive and negative movements along the active axis. Next,
we regulate the robot towards the chosen attractor position $x^{\mathrm{at}}$
with a Cartesian impedance controller. This controller first cancels all
static forces and the residual coupling of the null space on the operational
space dynamics. This allows us to now shape our own potential with a PD term
in operational space. As the robot is underactuated, we optimize least-squares
to identify the actuation $\phi^{\mathrm{d}}$ so that the residual between the
desired and actual torques in the configuration space is minimized. Icons
created by Flaticon©.

While BMI-based assistance has been investigated with a focus on soft exosuits
assisting hand-rehabilitation [12] or with strenuous arm acitivites [13], it
is still an open challenge how BMI can be used for controlling soft
manipulators. In this work, we make a first step towards solving this
challenge by proposing a pipeline (see Fig. 1) that lets the user steer the
soft robot’s end-effector in Cartesian space. The two key ingredients are a
novel mapping strategy transforming the brain signals into meaningful
references and Cartesian impedance control. The latter is essential because it
allows for preserving the robot’s compliance in closed-loop [14]. We build the
proposed BMI pipeline around a HSA soft robot [15, 16], which relies on
architected metamaterials and electrical actuation to elongate, bend, and
twist. This makes the control problem especially challenging because of the
peculiarity of these systems’ dynamics, namely underactuation and non-affinity
in control. We provide more details on innovation from the model-based control
standpoint in Section II-D.

We quantitatively verify the entire approach on mind-controlled setpoint
regulation involving tracking a reference consisting of nine-step functions
and demonstrate the qualitative behavior when assisting with a simple daily
living activity. Furthermore, we compare the performance of our motor imagery-
based control to approaches giving the computational controller access to the
privileged information of setpoints, which can considered to be an upper bound
on performance.

Our contributions are: (i) Establishing a BMI strategy for continuum soft
robots. This strategy is supported by experiments in which we perform setpoint
regulation with a planar HSA robot and motor imagery, (ii) A Cartesian
impedance controller for HSA robots, which we experimentally validated on a
simple ADL task involving environment interaction: the user needs to steer the
end-effector towards the tip of a hairspray container, apply force for
releasing the fluid, and finally let the robot retreat from the contact.

A video attachment to this paper including recordings of experimental results
can be found on YouTube111https://youtu.be/wZTOxBPZmPc. Furthermore, we have
open-sourced our code including the OpenVibe pipeline on
GitHub222https://github.com/tud-phi/sr-brain-control.

##  II Task-space Impedance Control

In this work, we let the user steer with motor imaginary brain signals the
Cartesian position $x\in\mathbb{R}^{2}$ of the end-effector (i.e., the
platform) of a planar HSA robot. We realize this strategy by first classifying
the motor imaginary signals into Cartesian-space movement directions (e.g.,
the active axis and sign of the movement). We use this information to adjust
the position of a task-space attractor iteratively (see Section II-B). Section
II-D describes how a model-based computational controller establishes this
attractor. Importantly, we preserve the soft robot’s compliance by shaping the
closed-loop system’s impedance in Cartesian space.

###  II-A Background: Motor Imagery-based BMI systems

Imagining the movement of body parts or limbs (e.g., hands, legs, tongue)
without moving it or the mental rehearsal of a motor act without overt
movement execution is termed Motor Imagery [17]. The neuronal activities
observable inside a frequency range of $8\text{\,}\mathrm{H}\mathrm{z}$ to
$12\text{\,}\mathrm{H}\mathrm{z}$ (Mu) and $12\text{\,}\mathrm{H}\mathrm{z}$
to $30\text{\,}\mathrm{H}\mathrm{z}$ (Beta) are associated with cortical areas
directly connected to the brain’s motor output (activating primary
sensorimotor areas that can be modulated with imaginary mental movement in
healthy as well people with neuromuscular disabilities).

The motor imagery BMI framework typically consists of four integral
components:

  1. 1.

Signal acquisition: The initial stage involves the recording of neural signals
while the person imagines the movements of the limbs, generally acquired using
noninvasive methodologies (e.g., EEG).

  2. 2.

Feature extraction: Following signal acquisition, signal processing techniques
are applied to extract salient features from the neural patterns associated
with specific cognitive processes or intentions.

  3. 3.

Feature translation: This translation phase interprets the user’s cognitive
intent, converting it into actionable instructions for external devices.

  4. 4.

Device output: The culmination of the BMI process is the application of the
interpreted commands to external devices.

As detailed further in Sec. III-B, we leverage the difference in signals when
imagining motor actions vs. rest state to control the sign of movement. The
active axis of movement can be switched by clenching the yaw.

###  II-B Planning attractors with brain signals

Our brain signal processing pipeline provides us with two pieces of
information at each time step $k$: i) the unit vector
$e_{\mathrm{a}}(k)\in\\{[1,0]^{\mathrm{T}},[0,1]^{\mathrm{T}}\\}$
corresponding to the current active axis of movement and ii) the sign of
movement $s(k)\in\\{-1,1\\}$. We use $e_{\mathrm{a}}(k)$ and $s(k)$ to
incrementally steer a virtual attractor defined in operational space
$x^{\mathrm{at}}\in\mathbb{R}^{2}$ as follows

$\small u(k)=s(k)\,e_{\mathrm{a}}(k)\in\mathbb{R}^{2},\quad
x^{\mathrm{at}}(k)=x^{\mathrm{at}}(k-1)+\Delta_{\mathrm{x}}\,u(k),$ (1)

where $\Delta_{\mathrm{x}}\in\mathbb{R}^{+}$ is a tunable constant influencing
the velocity of the attractor movement. Later, we will shape the potential
field with a computational controller such that the attractor becomes a
globally asymptotically stable equilibrium (see Section II-D).

Figure 2: EEG data processing pipeline: The EEG data is acquired in real-time,
pre-processed, and divided into episodes and subbands. Next, we extract power
features and pass them to two LDA classifiers: the first outputs the axis of
movement (for example, moving along the x- or y-axis), and the second provides
the sign of movement (for example, positive or negative movement along the
active axis). These commands are then used to move the attractor in Cartesian
space.

###  II-C Background: modeling planar HSA robots

Robots based on HSA rely on rods made of architected metamaterials to generate
motion. More specifically, twisting the rods along their handedness leads to
an elongation of the rod [18]. Combining multiple HSA in the setting of a
parallel robot and actuating them with servo motors allows us to generate
complex motion primitives and offer beneficial mechanical characteristics such
as a high stiffness-to-weight ratio [15, 18].

Prior work [16] has shown that the shape of planar HSA robots can be
approximated by one CS segment. Therefore, we define the configuration of the
system as
$q=\begin{bmatrix}\kappa_{\mathrm{be}}&\sigma_{\mathrm{sh}}&\sigma_{\mathrm{ax}}\end{bmatrix}^{\mathrm{T}}\in\mathbb{R}^{3}$.
We also have access to closed-form formulations of the forward kinematics
$\pi:q\rightarrow\chi$ and inverse kinematics $\varrho:\chi\rightarrow q$
where $\chi=\begin{bmatrix}x^{\mathrm{T}}&\theta\end{bmatrix}^{\mathrm{T}}\in
SE(2)$ is the pose in task space and $\theta$ represents the end-effector
orientation [16]. We use the notation $J(q)=\frac{\partial x}{\partial
q}\in\mathbb{R}^{2\times 3}$ to refer to the kinematic Jacobian.

We can then derive the dynamics of the planar HSA robot in Euler-Lagrangian
form

$\small
M(q)\ddot{q}+C(q,\dot{q})\dot{q}+G(q)+K\,(q-q^{0})+D\,\dot{q}=\alpha(q,\phi),$
(2)

where $M(q),C(q,\dot{q})\in\mathbb{R}^{3\times 3}$ captures the inertial and
Coriolis effects, $G(q)\in\mathbb{R}^{3}$ contributes the gravitational forces
and $K\in\mathbb{R}^{3\times 3}$ is the stiffness of the robot in its un-
actuated state $q^{0}$. Furthermore, $D\in\mathbb{R}^{3\times 3}$ is a
positive-definite damping matrix. Finally, for the planar case, two HSA rods
are assumed to be actuated by the motor/twist angle $\phi\in\mathbb{R}^{2}$.
As the handedness of the rods will be accounted for later, we state the
actuation bounds as $0\leq\phi_{i}\leq\phi_{\mathrm{max}}\>\forall
i\in\\{1,2\\}$. The auxetic trajectory [18] causes the motors to act through
the elasticity of the rods on the system and modify the axial rest length of
the rod as a function of the twist strain [15]. Furthermore, the stiffness of
the rod can be modeled to be an affine function with respect to the twist
strain [18, 15]. Both effects are captured in the actuation function
$\alpha(q,\phi)$, which is nonlinear with respect to the actuation coordinate
$\phi$ and affine in the configuration $q$. Although this has never been done
in the context of HSA robots, it is immediate to see that their dynamics (2)
can be projected into operational space yielding the form [19, 20]

$\small\Lambda(q)\,\ddot{x}+\mu(q,\dot{q})\dot{q}+J_{\mathrm{B}}^{+\mathrm{T}}(G(q)+K(q-q^{0})+D\,\dot{q})=J_{\mathrm{B}}^{+\mathrm{T}}\alpha(q,\phi),$
(3)

where
$J_{\mathrm{B}}^{+}(q)=B^{-1}J^{\mathrm{T}}(JB^{-1}J^{\mathrm{T}})^{-1}\in\mathbb{R}^{3\times
2}$ is the dynamically consistent pseudo-inverse,
$\Lambda(q)=(J\,B^{-1}J^{\mathrm{T}})^{-1}\in\mathbb{R}^{2\times 2}$ is the
inertia matrix in task space, and
$\mu(q,\dot{q})=\Lambda(q)\,(JB^{-1}C-\dot{J})\in\mathbb{R}^{2\times 3}$
collects the Cartesian Coriolis and centrifugal terms.

###  II-D Cartesian impedance controller

In previous work [16], we have devised a model-based control strategy for
regulating a planar HSA robot towards a desired position in task space. We
introduce below a novel control strategy that addresses some limitations of
our previous work that are critical for the BMI application. Namely, we (i)
avoid computationally demanding planning procedures, (ii) remove integral
terms that are unsafe for environment interaction, and (iii) enable impedance
shaping in operational space. This Cartesian-space impedance controller is
inspired by [21, 20], but specifically designed for and tailored to HSA
robots. Crucially, we need to overcome the challenges of underactuation and
the nonlinearity in the actuation - which were not present in that original
work.

####  II-D1 Proposed controller

We propose the following dynamic feedback law that renders $x^{\mathrm{at}}$
an attractor of the closed-loop system

$\small\begin{split}\tau=&J^{\mathrm{T}}(q)\,\left(K_{x}\,(x^{\mathrm{at}}-x)-D_{x}\,\dot{x}\right)+G(q)+K\,(q-q^{0})\\\
&+J^{\mathrm{T}}(q)\,J_{\mathrm{B}}^{+\mathrm{T}}(q)\,D\,\dot{q}+J^{\mathrm{T}}(q)\,\mu(q,\dot{q})\left(I_{3}-J_{\mathrm{B}}^{+}(q)J(q)\right)\dot{q}\end{split}$
(4)

where $\tau\in\mathbb{R}^{3}$ is the desired torque in configuration space,
$G(q)+K\,(q-q^{0})$ cancels the acting gravitational and elastic forces, and
$J^{\mathrm{T}}J_{\mathrm{B}}^{+\mathrm{T}}D\,\dot{q}$ removes the natural
dissipation in operational space. We emphasize that because the system is
underactuated, we need to cancel the stiffness directly in the configuration
instead of operational space as done in previous work [20]. We can shape our
desired impedance characteristics in Cartesian space with the PD term
$f_{\mathrm{PD}}=K_{x}\,(x^{\mathrm{at}}-x)-D_{x}\,\dot{x}$ which is then
projected into configuration space by premultiplying with $J^{\mathrm{T}}(q)$.

The term $\mu(q,\dot{q})\left(I_{3}-J_{\mathrm{B}}^{+}(q)J(q)\right)\dot{q}$
decouples the operational space dynamics from the residual of the null-space
dynamics [20][21, Ch. 4]. The identity
$\dot{q}=J_{\mathrm{B}}^{+}\,\dot{x}+Z^{\mathrm{T}}\,\nu_{\mathrm{N}}$, where
$Z^{\mathrm{T}}\in\mathbb{R}^{3\times 1}$ is the dynamically-consistent
pseudo-inverse of the null space, allows us to formulate $\dot{q}$ as a sum of
the task-space velocity $\dot{x}$ and the null-space velocity
$\nu_{\mathrm{N}}$. Leveraging this identity, the Coriolis and centrifugal
matrix $\mu(q,\dot{q})$ can be split into a term
$\mu_{x}(q,\dot{q})=\mu\,J_{\mathrm{B}}^{+}\in\mathbb{R}^{2\times 2}$ excited
by $x$ and the expression
$\mu_{\mathrm{N}}(q,\dot{q})=\mu\,Z^{\mathrm{T}}\in\mathbb{R}^{2\times 1}$
that is excited by the null-space coordinates resulting in
$\mu(q,\dot{q})\,\dot{q}=\mu_{x}(q,\dot{q})\,\dot{x}+\mu_{\mathrm{N}}(q,\dot{q})\,\nu_{\mathrm{N}}$.
This allows us to cancel the term
$\mu_{\mathrm{N}}(q,\dot{q})\,\nu_{\mathrm{N}}$ through
$\mu(q,\dot{q})\left(I_{3}-J_{\mathrm{B}}^{+}(q)J(q)\right)\dot{q}$ without
having to compute the null space explicitly.

In summary, the closed-loop dynamics in operational space can be stated as

$\small\Lambda(q)\,\ddot{x}+\mu(q,\dot{q})\,J_{\mathrm{B}}^{+}\,\dot{x}+K_{x}\,(x^{\mathrm{at}}-x)-D_{x}\,\dot{x}=0,$
(5)

which results in $x^{\mathrm{at}}$ being the globally asymptotically stable
equilibrium of the operational space dynamics.

(a) Operational workspace

(b) Experimental setup

Figure 3: Panel (a: Operational workspace of a HSA robot with attached end-
effector: the color displays the mean steady-state actuation
$\frac{\phi_{1}^{\mathrm{ss}}+\phi_{2}^{\mathrm{ss}}}{2}$ necessary for the
end-effector to remain at the position. Additionally, we visualize three
example shapes: the straight configuration with $\phi^{\mathrm{ss}}=(0,0)$
(blue), maximum clockwise bending with
$\phi^{\mathrm{ss}}=(3.49,0)\,$\mathrm{r}\mathrm{a}\mathrm{d}$$ (red), and
maximum counter-clockwise bending with
$\phi^{\mathrm{ss}}=(0,3.49)\,$\mathrm{r}\mathrm{a}\mathrm{d}$$ (green). Panel
(b): The HSA robot is mounted platform-down to a motion capture cage with 8x
Optitrack PrimeX 13 cameras, which track the 3D pose of the platform (i.e.,
the end-effector). A Dynamixel MX-28 servo actuates each of the four HSAs. We
project a rendering of the current (white dot) and desired (red dot) end-
effector position, the attractor (green square), and the operational workspace
(grey area) onto the black screen in the background. The study subject wears a
cap with the Neuroconcise FlexEEG sensor, and we acquire the data of three
electrodes connected to the motor cortex. Figure 4: ERD/S (overall average)
over a time period of $2.5\text{\,}\mathrm{s}$ of training data for right-hand
Imagination v/s rest state including the Alpha and Beta bands of the EEG
signals where the cue is presented at $0\text{\,}\mathrm{s}$. We plot the data
of three sensors (i.e., channels): FC3-CP3 (left), FCZ-CPZ (middle) and
FC4-CP4 (right).

(a) End-effector x-coordinate

(b) End-effector y-coordinate

(c) Configuration $q$

(d) Control input $\phi$

Figure 5: Experimental results for tracking a reference trajectory of nine
step functions with motor imagery. Panel (a) & (b): The x/y-coordinate of the
end-effector position with the solid line denoting the actual position, the
dotted line the attractor position, and the dashed line the reference (i.e.,
the setpoint). Panel (c): The evolution of the configuration. Panel(d): The
saturated planar control inputs.

####  II-D2 Mapping to Lagrangian forces

Now that we have formulated our control law $\tau$ in configuration space, we
need to identify a strategy to specify the motor angles
$\phi\in\mathbb{R}^{2}$ such that $\alpha(q,\phi)\approx\tau$. Note that, in
contrast to other continuum soft robots studied in literature [22], the
actuation term $\alpha(q,\phi)$ is not affine in control. In previous work
[16], we side-stepped this challenge by linearizing with respect to the
steady-state actuation $\phi^{\mathrm{ss}}$:
$A(q)=\lVert\frac{\partial\alpha}{\partial\phi}\rVert_{\phi=\phi^{\mathrm{ss}}}$
therefore recovering the usual scenario of an affine actuation function.
Unfortunately, this is not possible in the setting of this work as i) we do
not have access to such $\phi^{\mathrm{ss}}$, and ii) linearizing around
$\phi$ causes the closed-loop system to become unstable. We, therefore,
propose to formulate instead a nonlinear least-squares problem
$\phi^{\mathrm{d}}=\operatorname*{arg\\!\min}_{\phi}\frac{1}{2}\lVert\tau-\alpha(q,\phi)\rVert^{2}$
and solve it in real-time with a Levenberg Marquardt solver implemented in JAX
[23].

We note that this approach is not guaranteed to be valid for the general case
of an underactuated soft robot but for this particular structure of
$\alpha(q,\phi)\in\mathbb{R}^{3}$ with $\phi\in\mathbb{R}^{2}$ it is possible
to identify solutions $\phi$ with the Euclidean norm of the residual being
smaller than $0.001$. The source code of the controller is available on
GitHub333https://github.com/tud-phi/hsa-planar-control.

##  III Experiments

###  III-A Experimental setup

In the following, we detail the EEG data processing procedure (see also Fig.
2) and present our experimental setup, which is annotated in Fig. 3(b).

####  III-A1 EEG data processing

We integrate the 3-channel flexEEG Neuroconcise device with the OpenVibe
software to acquire the EEG data and process it in real-time. This
configuration facilitates data recording and cue presentation. We process the
EEG signals at a sampling frequency of $125\text{\,}\mathrm{H}\mathrm{z}$ with
a pipeline that involves three bi-polar channels around the motor cortex:
FC3-CP3, FCZ-CPZ, and FC4-CP4. After a notch filter of
$50\text{\,}\mathrm{H}\mathrm{z}$, we apply Independent Component Analysis
(ICA) to extract three independent components from the recorded EEG data,
which is represented by the equation $S(t)=W\,X(t)$, where $S(t)$ are the
extracted independent components and $W$ represents the unmixing matrix,
allowing us to separate eye blink artifacts in EEG signals, which is critical
for enhancing the accuracy. Subsequently, we apply a Butterworth filter bank
to isolate specific frequency bands of interest, including 8-15
$\mathrm{H}\mathrm{z}$ and 15-42 $\mathrm{H}\mathrm{z}$. This enhances the
ability to analyze EEG data by isolating and examining different frequency
band components within the EEG signals. Once the signals are filtered in sub-
bands and epoched with a duration of $2\text{\,}\mathrm{s}$ and time interval
of $0.065\text{\,}\mathrm{s}$, the features are extracted by the log of the
power: $L_{i}(t)=\log\left(P_{i}(t)\right)$, where the power
$P_{i}(t)=|E_{i}(t)|^{2}$ is represented by square of magnitude of the EEG
signal $E_{i}(t)$ at time instance $t$. These features are then provided to a
classifier, which we select as LDA due to its simplicity [24].

We implement a second classifier with the same pipeline, where jaw clenching
is provided as a muscle artifact that is classified v/s raw EEG data.

####  III-A2 Robotic system

We consider a robot consisting of four HSA rods, which were 3D printed via
digital projection lithography from the photopolymer resin Carbon FPU 50. Each
HSA rod is electrically actuated by a Dynamixel M-28 servo up to a maximum
twist angle of
$\phi_{\mathrm{max}}=$3.49\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$. The robot
is attached platform-down to a motion capture cage with eight Optitrack Prime
X13 cameras tracking at $200\text{\,}\mathrm{H}\mathrm{z}$ the pose of
reflective markers attached to the end-effector of the HSA robot. We estimate
the current Cartesian-space velocity of the end-effector with a Savitzky-Golay
filter. Subsequently, we leverage a closed-form expression of the inverse
kinematics of a CS model [16] to compute the current configuration $q$ of the
robot. We render an image of the current shape of the robot together with the
present end-effector position (white dot), the attractor planned by the user
(green square), the operational workspace (grey, see also Fig. 3(a)) and if
applicable, the goal position (red dot). We specify the currently active axis
of movement $e_{\mathrm{a}}$ with a double arrow and project the resulting
image onto a black screen in the background of the motion capture cage. The
robot is operated with a ROS2 software framework444https://github.com/tud-
phi/ros2-hsa. We receive the predicted and classified brain signals via TCP at
a frequency of $18\text{\,}\mathrm{H}\mathrm{z}$ and move the attractor
subsequently with $\Delta_{\mathrm{x}}=$0.2\text{\,}\mathrm{m}\mathrm{m}$$. We
evaluate the Cartesian impedance controller using the gains
$K_{\mathrm{p}}=$300\text{\,}\mathrm{N}\mathrm{/}\mathrm{m}$$,
$K_{\mathrm{d}}=$1.5\text{\,}\mathrm{N}\mathrm{s}\mathrm{/}\mathrm{m}$$ at a
frequency of $50\text{\,}\mathrm{H}\mathrm{z}$ and finally send the desired
motor positions to the servos.

The entire control pipeline from measuring EEG signals to sending the
actuation commands to the motors exhibits a maximum latency of (i.e., is
upper-bounded by) $52\text{\,}\mathrm{m}\mathrm{s}$.

###  III-B BMI protocol:

In the following, we will describe the protocol for collecting the motor
imagery dataset, training the EEG classifiers, and mapping classifier outputs
into actionable robot commands.

####  III-B1 Training protocol

The participant is given brief instructions about motor imagery signals. We
follow the Graz-BCI [25] paradigm, which assists with training, where the
display of the cue instructs the participant to perform imagination of
movements: when a left arrow appears, the participant is asked to rest and not
focus on motor movement. When the right arrow appears on the screen, the
participant is asked to imagine the motor activity from the dominant hand
(here, the right hand), such as grasping or squeezing an object. The training
protocol for right-hand motor imagery v/s rest demands fifty cues per class.
We similarly collect data for the second classifier by asking participants to
clench their jaw, which can be detected as muscular artifacts (vs. no muscular
movement) in the EEG signals. At the end of the trial, we train both
classifiers (see Sections II-A and III-A1 for more information) and repeat the
procedure until an accuracy of $75\text{\,}\mathrm{\char 37\relax}$ is
obtained for right-hand motor imagery v/s rest and accuracy of
$85\text{\,}\mathrm{\char 37\relax}$ for jaw clench artifact v/s raw EEG.

(a) End-effector x-coordinate

(b) End-effector y-coordinate

(c) Configuration $q$

(d) Control input $\phi$

Figure 6: Experimental results for tracking a reference trajectory of nine
step functions directly with the Cartesian impedance controller with access to
the privileged information $x^{\mathrm{d}}$. Panel (a) & (b): The
x/y-coordinate of the end-effector position with the solid line denoting the
actual position, the dotted line the attractor position, and the dashed line
the reference (i.e., the setpoint). Panel (c): The evolution of the
configuration. Panel(d): The saturated planar control inputs.

####  III-B2 Online robot control

Now, the participant operates the HSA robot in real time, with both
classifiers being executed online. Moreover, we map the outputs of the
classifier into actionable commands: we initialize the x-axis as the active
axis of movement (i.e., $e_{\mathrm{a}}=[1,0]$). When the first classifier
detects jaw clenching for at least $80\text{\,}\mathrm{\char 37\relax}$ of
samples over a duration of $2.8\text{\,}\mathrm{s}$, we switch to the y-axis:
$e_{\mathrm{a}}=[0,1]$ and vice-versa. If we do not detect any yaw clenching
artifacts, we evaluate the output of the second classifier in parallel: if
there is motor imagery activity identified in the EEG classifier, the
attractor will be moved in the positive direction (i.e., $s=1$) along
$e_{\mathrm{a}}$. In contrast, if the EEG signals are classified as the
participant being at rest, $s=-1$ (i.e., movement in the negative direction).

###  III-C Setpoint regulation

We randomly generate nine setpoints $x^{\mathrm{d}}(t)\in\mathbb{R}^{2}$
within the operational workspace of the robot (see Fig. 3(a) and display them
as a red circle to the user over a duration of $540\text{\,}\mathrm{s}$. The
user can freely move the attractor to reach and keep the end-effector at the
setpoint. Furthermore, we execute an experiment in which the computational
controller has access to normally privileged information: as we substitute
$x^{\mathrm{at}}$ with $x^{\mathrm{d}}$ in (4), the Cartesian impedance
controller now immediately regulates the robot towards the setpoint. By
excluding the BMI from the pipeline, this provides us with a reference of the
performance we can expect from the computational controller and, with that,
also represents a performance upper bound.

(a) $t=$0\text{\,}\mathrm{s}$$

(b) $t=$12\text{\,}\mathrm{s}$$

(c) $t=$24\text{\,}\mathrm{s}$$

(d) $t=$36\text{\,}\mathrm{s}$$

(e) $t=$48\text{\,}\mathrm{s}$$

(f) $t=$60\text{\,}\mathrm{s}$$

(g) $t=$72\text{\,}\mathrm{s}$$

(h) $t=$84\text{\,}\mathrm{s}$$

(i) $t=$96\text{\,}\mathrm{s}$$

(j) $t=$108\text{\,}\mathrm{s}$$

Figure 7: Sequence of stills for completing a basic Activity of Daily Living
(ADL) by controlling the robot with EEG-based motor imagery. _Note:_ Fig. 7(i)
is edited for improved contrast.

###  III-D Interacting with the environment on a real-world task

We consider the ADL task of releasing hairspray by actuating the button of its
container with the HSA robot’s end-effector. For successful execution, the
end-effector must be very stiff in the normal direction of the contact. On the
other hand, we might want to benefit from the physical intelligence of the
system by being relatively flexible in the tangential direction. Therefore, we
first define the perpendicular stiffness
$k_{\perp}=$500\text{\,}\mathrm{N}\mathrm{/}\mathrm{m}$$ and the tangential
stiffness as $k_{\parallel}=$50\text{\,}\mathrm{N}\mathrm{/}\mathrm{m}$$. We
assume that the normal direction of the contact can be described by the polar
angle $\theta_{\perp}$ (with respect to the x-axis). We envision that in the
future, the user can adjust such stiffness characteristics online via a BMI
system similar to [6]. In this work, however, we estimate by visual inspection
that $\theta_{\perp}=$1.31\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$. The
Cartesian stiffness matrix in global coordinates is then given by
$K_{x}=R(\theta_{\perp})\,\mathrm{diag}(k_{\perp},k_{\parallel})\,R(\theta_{\perp})^{\mathrm{T}}$
where $R(\theta_{\perp})\in SO(2)$ is the rotation matrix between global and
contact frames.

(a) End-effector $x$-coordinate

(b) End-effector $y$-coordinate

(c) Configuration $q$

(d) Control input $\phi$

Figure 8: Experimental results for completing a basic Activity of Daily Living
(ADL) by controlling the robot with EEG-based motor imagery. Panel (a) & (b):
The x/y-coordinate of the end-effector with the solid line and dotted lines
denoting the actual and attractor position, respectively. Panel (c): The
evolution of the configuration. Panel(d): The saturated planar control inputs.

###  III-E Evaluation metrics

In the following, we will introduce and define a few metrics that help us
assess the performance of the approach.

####  III-E1 Event-Related Synchronization/De-Synchronization:

We apply ERD / ERS [26] to demonstrate the difference between EEG signals when
the participant imagines right-hand movement vs. rest, i.e., no activity.
ERD/ERS corresponds to a shift in power during imagination with respect to a
baseline. It is defined by

$\mathrm{ERD/ERS}(t,f)=\frac{P(t,f)-P_{\mathrm{base}}(f)}{P_{\mathrm{base}}(f)},$
(6)

where $\mathrm{ERD/ERS}(t,f)$ represents the ERD or ERS at a specific time $t$
and frequency $f$, $P(t,f)$ stands for the power of brain activity during
imagination, and $P_{\mathrm{base}}(f)$ denotes the baseline power.

###  III-F Step response metrics

For the task of setpoint regulation, we analyze primarily two aspects: (a) is
the participant able to reach the proximity of setpoint within the
(generously) allotted time of $60\text{\,}\mathrm{s}$? We define the proximity
of the setpoints as $\lVert
x^{\mathrm{d}}-x(t)\rVert_{2}\leq$2\text{\,}\mathrm{m}\mathrm{m}$$. And (b)
what is the response time for reaching for the first time the proximity of the
setpoint?

##  IV Results and Discussion

First, we analyze the ERD/ERS behavior with respect to rest vs. motor
imaginations in Fig. 4. It is evident that the baseline of rest remained the
same in both scenarios when the participant did not perform motor imagery, but
as soon as the cue is presented at $0.0\text{\,}\mathrm{s}$, a shift in power
for the right-hand motor imagery with comparison to rest state is noticeable.

We present the results for setpoint regulation employing motor imagery in Fig.
5. We observe that the participant can reach the proximity of the setpoint
within the allotted time of $60\text{\,}\mathrm{s}$ six out of nine times
(i.e., $66.6\text{\,}\mathrm{\char 37\relax}$). For the successful steps, the
average response time is $21.5\text{\,}\mathrm{s}$. However, as our protocol
does not contain a command to let the attractor rest, it is challenging to
keep the end-effector at the setpoint and we observe oscillations,
particularly with respect to the x-coordinate. In our third experiment, we ask
the Cartesian impedance controller to track the setpoints directly. The fast
response time, a well-known characteristic of model-based control approaches,
is evident. However, the errors in the model (for example, caused by
hysteresis or unmodelled nonlinearities) [16], together with the lack of
integral action, lead to steady-state errors.

Finally, we consider the ADL task of releasing hair spray using the end-
effector of the HSA robot. We present a sequence of stills in Fig. 7 and plots
of the entire sequence in Fig. 8. Already during the first attempt, the
participant can steer the end-effector toward the button, apply force, and
release the fluid within $86\text{\,}\mathrm{s}$. The impedance of the
controller is clearly visible in Fig. 8(b) when the manipulator is in contact
with the object at time $74\text{\,}\mathrm{s}$ to $104\text{\,}\mathrm{s}$.
Also, we noticed that the end-effector does not need to be perfectly aligned
with the center of the button and can still complete the task successfully due
to the compliance of the closed-loop system in the tangential direction.

We noticed that the variability of setting up the EEG device on each study
participant and the EEG sensor noise caused by external factors (e.g., floor
vibrations) still pose a considerable challenge for deploying motor imagery-
based tools in practice. Furthermore, subject-specific factors such as the
ability to focus on imagining motor actions, mental tiredness, etc.,
significantly affected the performance (e.g., classification accuracy,
setpoint tracking error).

##  V Conclusion

In this paper, we proposed to combine motor imagery-based BMI systems with
continuum soft robots. This symbiosis promises the safe and compliant
operation of robots that can assist people with limb impairments in their
daily lives. As demonstrated in the ADL experiment, the physical intelligence
of the soft robot can compensate for errors and deviations in the output of
the BMI classifier. Furthermore, we introduced a Cartesian impedance
controller for planar HSA robots that can deal with the peculiar
characteristics of these robots (e.g., underactuation, non-affinity in
control, etc.), and allows for model-based control without interfering with
the structural compliance of the system.

## Acknowledgment

The authors would like to thank Dr. Fabien Lotte for his suggestions
concerning the protocol, Dr. Tomas Ward and the Neuroconcise team for their
support with the FlexEEG device, and J.K Balasubramanian for his assistance
with the EEG setup.

## References

  * [1] S. Liu, L. Wang, and R. X. Gao, “Cognitive neuroscience and robotics: Advancements and future research directions,” _Robotics and Computer-Integrated Manufacturing_ , vol. 85, p. 102610, 2024.
  * [2] S. M. Coyle, T. E. Ward, and C. M. Markham, “Brain–computer interface using a simplified functional near-infrared spectroscopy system,” _Journal of neural engineering_ , vol. 4, no. 3, p. 219, 2007.
  * [3] K. Lee, D. Liu, _et al._ , “A brain-controlled exoskeleton with cascaded event-related desynchronization classifiers,” _Robotics and Autonomous Systems_ , vol. 90, pp. 15–23, 2017.
  * [4] J. Van Erp, F. Lotte, and M. Tangermann, “Brain-computer interfaces: beyond medical applications,” _Computer_ , vol. 45, no. 4, pp. 26–34, 2012.
  * [5] M. A. Khan, R. Das, _et al._ , “Review on motor imagery based bci systems for upper limb post-stroke neurorehabilitation: From designing to application,” _Computers in biology and medicine_ , vol. 123, p. 103843, 2020.
  * [6] L. Schiatti, J. Tessadori, _et al._ , “Soft brain-machine interfaces for assistive robotics: A novel control approach,” in _2017 International Conference on Rehabilitation Robotics (ICORR)_. IEEE, 2017, pp. 863–869.
  * [7] S. Aldini, A. Akella, _et al._ , “Effect of mechanical resistance on cognitive conflict in physical human-robot collaboration,” in _2019 international conference on robotics and automation (ICRA)_. IEEE, 2019, pp. 6137–6143.
  * [8] R. Zhang, S. Lee, _et al._ , “Noir: Neural signal operated intelligent robots for everyday activities,” _arXiv preprint arXiv:2311.01454_ , 2023.
  * [9] P. Arpaia, D. Coyle, _et al._ , “Non-immersive versus immersive extended reality for motor imagery neurofeedback within a brain-computer interfaces,” in _International Conference on Extended Reality_. Springer, 2022, pp. 407–419.
  * [10] D. Rus and M. T. Tolley, “Design, fabrication and control of soft robots,” _Nature_ , vol. 521, no. 7553, pp. 467–475, 2015.
  * [11] C. Della Santina, M. G. Catalano, _et al._ , “Soft robots,” _Encyclopedia of Robotics_ , vol. 489, 2020.
  * [12] J. Zhang, B. Wang, _et al._ , “An eeg/emg/eog-based multimodal human-machine interface to real-time control of a soft robot hand,” _Frontiers in neurorobotics_ , vol. 13, p. 7, 2019.
  * [13] N. Tacca, J. Nassour, _et al._ , “Neuro-cognitive assessment of intentional control methods for a soft elbow exosuit using error-related potentials,” _Journal of NeuroEngineering and Rehabilitation_ , vol. 19, no. 1, p. 124, 2022.
  * [14] C. Della Santina, M. Bianchi, _et al._ , “Controlling soft robots: balancing feedback and feedforward elements,” _IEEE Robotics & Automation Magazine_, vol. 24, no. 3, pp. 75–83, 2017.
  * [15] M. Stölzle, L. Chin, _et al._ , “Modelling handed shearing auxetics: Selective piecewise constant strain kinematics and dynamic simulation,” in _2023 IEEE International Conference on Soft Robotics (RoboSoft)_. IEEE, 2023, pp. 1–8.
  * [16] M. Stölzle, D. Rus, and C. D. Santina, “An experimental study of model-based control for planar handed shearing auxetics robots,” in _Experimental Robotics: The 18th International Symposium_. Springer, 2023.
  * [17] M. Lotze and U. Halsband, “Motor imagery,” _Journal of Physiology-paris_ , vol. 99, no. 4-6, pp. 386–395, 2006.
  * [18] I. Good, T. Brown-Moore, _et al._ , “Expanding the design space for electrically-driven soft robots through handed shearing auxetics,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 10 951–10 957.
  * [19] C. Della Santina, L. Pallottino, _et al._ , “Exact task execution in highly under-actuated soft limbs: an operational space based approach,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 3, pp. 2508–2515, 2019.
  * [20] C. Della Santina, R. K. Katzschmann, _et al._ , “Model-based dynamic feedback control of a planar soft robot: trajectory tracking and interaction with the environment,” _The International Journal of Robotics Research_ , vol. 39, no. 4, pp. 490–513, 2020.
  * [21] C. Ott, _Cartesian impedance control of redundant and flexible-joint robots_. Springer, 2008.
  * [22] C. Della Santina, C. Duriez, and D. Rus, “Model-based control of soft robots: A survey of the state of the art and open challenges,” _IEEE Control Systems Magazine_ , vol. 43, no. 3, pp. 30–65, 2023.
  * [23] M. Blondel, Q. Berthet, _et al._ , “Efficient and modular implicit differentiation,” _arXiv preprint arXiv:2105.15183_ , 2021.
  * [24] F. Lotte, “A tutorial on eeg signal-processing techniques for mental-state recognition in brain–computer interfaces,” _Guide to brain-computer music interfacing_ , pp. 133–161, 2014.
  * [25] A. Roc, L. Pillette, _et al._ , “A review of user training methods in brain computer interfaces based on mental tasks,” _Journal of Neural Engineering_ , vol. 18, no. 1, p. 011002, 2021.
  * [26] G. Pfurtscheller and F. L. Da Silva, “Event-related eeg/meg synchronization and desynchronization: basic principles,” _Clinical neurophysiology_ , vol. 110, no. 11, pp. 1842–1857, 1999.