# Multimodal Visual-haptic pose estimation in the presence of transient
occlusion

Michael Zechmair1, Alban Bornet2 and Yannick Morel1 1Michael Zechmair and
Yannick Morel are with Faculty of Psychology and Neuroscience, Maastricht
University, Maastricht, The Netherlands<EMAIL_ADDRESS>Bornet is with with Department of Radiology and Medical Informatics,
University of Geneva, Geneva, Switzerland<EMAIL_ADDRESS>and was with
Laboratory of Psychophysics, Brain Mind Institute, EPFL, Lausanne,
SwitzerlandThis research has received funding from the European Union’s
Horizon 2020 Framework Programme for Research and Innovation under the
Specific Grant Agreement No. 785907 (Human Brain Project SGA3).

###### Abstract

Human-robot collaboration requires the establishment of methods to guarantee
the safety of participating operators. A necessary part of this process is
ensuring reliable human pose estimation. Established vision-based modalities
encounter problems when under conditions of occlusion. This article describes
the combination of two perception modalities for pose estimation in
environments containing such transient occlusion. We first introduce a vision-
based pose estimation method, based on a deep Predictive Coding (PC) model
featuring robustness to partial occlusion. Next, capacitive sensing hardware
capable of detecting various objects is introduced. The sensor is compact
enough to be mounted on the exterior of any given robotic system. The
technology is particularly well-suited to detection of capacitive material,
such as living tissue. Pose estimation from the two individual sensing
modalities is combined using a modified Luenberger observer model. We
demonstrate that the results offer better performance than either sensor
alone. The efficacy of the system is demonstrated on an environment containing
a robot arm and a human, showing the ability to estimate the pose of a human
forearm under varying levels of occlusion.

###### Index Terms:

##  I Introduction

In recent decades, robotic applications have become prevalent in industrial
processing and manufacturing lines. Robots have been used to automate tasks
and reduce human workloads. In particular, robots are well suited to executing
repeatable, well-defined processes. Recent research has focused on expanding
the capabilities of robotic applications with the aim of creating more
autonomous systems that can adapt to environmental factors that alter task
parameters ([1]). As these robots could be deployed in complex production
lines, more processes could be automated. To ensure reliable interactions with
their surroundings, robotic applications require robust methods of
environmental characterization. This includes detecting and localizing nearby
objects of interest, particularly those necessary to perform a given task. One
approach is to develop methods of human-robot collaboration, as the
capabilites of humans and robots synergize well. Humans are capable of
adapting to changes in established processes, and can adjust a robot’s
behavior accordingly. However, having humans and robots collaborate on a task
requires the establishment of safety guarantees to prevent injuring humans. As
such, precise, real-time localization is fundamental for robots to understand
human intentions, adapt to dynamic behaviors, avoid collisions, and
effectively collaborate with their human counterparts.

Current approaches to object and human localization heavily focus on vision-
based methods. This relies on segmenting a vision-based input, such as an RGB
camera, highlighting how various objects are distributed in an image. Most
modern computer vision methods use convolutional neural networks (CNNs) to
solve vision segmentation tasks ([2, 3, 4, 5, 6, 7]). Due to their
versatility, CNNs are also the most performant models for many other vision
tasks, including image recognition ([8, 9, 10]), image generation ([11, 12,
13, 14]), and scene rendering ([15, 16, 17]), among others. Because of this
flexibility, CNNs have proven useful for a wide range of image-based practical
applications such as medical imaging ([18, 19]), autonomous driving ([20]), or
manufacturing ([21, 22, 23]). However, it has been observed that CNNs face
problems when encountering visual occlusion ([24, 25, 26, 27]). In the case of
visual segmentation, image contours do not support the boundaries of an
occluded object, whereas the occluder’s contours may be mistaken for the
boundaries of a considered object. As discussed in [28], this problem is
particularly significant in collaborative robotic (“cobotic”) use cases, in
which humans and robots work together in close proximity. Such interactions
lead to strong and frequent occlusions, heavily impairing inferences made by
the network.

This challenge is very relevant to everyday human vision: despite encountering
elements that appear very rarely in isolation, human vision is notably robust
to occlusion ([29, 30, 31, 32, 33]). Hence, using insights from human vision
may grant the keys to increasing CNNs’ robustness to occlusions. More in
general, taking inspiration from the human visual cortex may help improve the
flexibility and generalization capabilities of neural networks. Importantly,
although CNNs reach human-level performance in many complex vision tasks and
are the best models of image-evoked population response in the primate visual
cortex ([34, 35, 36]), fundamental differences exist between humans and CNNs.
Indeed, human-like performance of CNNs does not necessarily imply human-like
computations. For example, the role of feedback connections in the human
visual cortex remains a matter of debate ([37, 38]). Understanding these
discrepancies is crucial to move towards more human-like models. To this end,
there exists a large corpus of results from vision psychophysics paradigms
that CNNs cannot explain. For example, visual crowding experiments have shown
that the human visual cortex integrates information across very large portions
of the visual field ([39, 40, 41]). These experiments suggest that, in human
vision, high-level context about the global configuration of the visual input
strongly affects local and low-level information processing ([42, 43, 44]). In
contrast, CNNs cannot reproduce these results because they are based on
feedforward and local operations only ([45, 46]). Identifying what is missing
from CNNs to account for global aspects of crowding is a good way to
understand differences to human-like computations. Recent studies showed that
the only models of human vision explaining the global aspects of visual
crowding include explicit recurrent grouping and segmentation processes ([47,
48]). For example, adding dynamic routing to CNNs (capsule networks) ([49]) or
illusory contour mechanics ([50]), both of which instantiate grouping and
segmentation, matched human behaviour in visual crowding paradigms ([47, 51,
52]). This line of evidence suggests that one computational function of
recurrent processing in the human visual cortex is to efficiently select which
features integrate (grouping) and which do not (segmentation). These processes
help the brain cope with complex input data (occlusions, reflections, noise,
etc.) and refine low-level information based on high-level context. In
computer vision for example, adding feedback processing to CNNs is helpful to
perform inferences from partial information ([29, 32, 53, 54]).

We built upon such psycho-physical insights to create a visual segmentation
model providing a degree of robustness to occlusion. More specifically, we
present a method able to learn to produce segmentation masks based on
Predictive Coding, and infer likeliest pose by comparing the segmentation mask
produced by the PC model to a range of candidate masks, reflecting different
candidate poses: ProcNet.

While vision-based localization has been extensively employed for its non-
intrusiveness and versatility, it faces significant challenges in environments
where occlusion is prevalent. Cluttered spaces or close interactions can
obstruct visual cues, leading to incomplete or inaccurate localization. Human-
robot collaboration in particular requires close interactions, leading to
occlusion. The limitations of vision-based methods necessitate the exploration
of multimodal localization approaches which address occlusion-dependent
inaccuracies and enhance overall localization accuracy. Embracing multimodal
approaches that fuse data from multiple sensors enables robots to compensate
for the limitations of individual sensors and achieve more comprehensive and
robust localization, particularly in challenging environments. Among various
sensing alternatives, capacitive sensing has emerged as an ideal additional
modality for enhancing human localization in robotic applications. By
leveraging changes in electrical fields, capacitive sensors can accurately
detect and track nearby objects, including humans. However, the sensor’s
limited range necessitates a complementary, broad-range sensing modality.
Therefore, combining capacitive and vision-based localization allows us to
cover the entire volume of a robot’s working environment. By placing
capacitive sensors along the robot’s hull, we can localization nearby objects
at close range, while vision-based sensing localizes objects in a broad area
in the workspace.

Capacitive sensors have been used to support detection of a broad range of
materials. In [55], the authors present a simple sensor design to measure the
level of liquid present in a container, requiring only two metal rods.
Alternatively, the water content in crude-oil can be accurately determined by
measuring sensor phase changes ([56]). Water flow rates can be measured by
similar techniques, as shown in [57]. A novel sensor electrode design intended
to improve sensitivity to humidity is presented by the authors of [58]. Very
precise distance measurements of metallic objects have been performed by
combining capacitive with inductive-based sensors ([59]). The technology can
also be used to for wearables, supporting the design of flexible sensors
allowing to quantify human activity. Variables measured include respiratory
and heart rate, as well as muscle deformation ([60, 61]). Capacitive sensors
have also been employed to measure a number of object properties. In [62], the
authors describe the implementation of a sensor on two rotors allowing the
determination of their relative rotation angle. The approach in [63] uses
three liquid electrodes, encased in elastic material, to measure both shear
and pressure deformation. As capacitive sensors mainly require two conductive
plates to function, they can even be directly integrated into production
processes by commercially available, multi-material 3D-printers, as discussed
in [64].

For the purpose of human-robot interaction and accident prevention, a number
of haptic-sensing based approaches have been investigated. In [65], various
methods of active touch sensing used by mammals are discussed, along with
their applications to robotics. In [66], a pressure-sensitive skin that can
detect human touch when mounted on a robot’s hull is proposed. While this
approach is based on resistance measurements, the authors in [67] demonstrate
an analogous sensor with the same touch-sensing capabilities, based on
capacitive sensing. They also argue that by mounting the sensors on all
segments of a robot arm, contact between robot and human can be detected and
the robot’s movements can be immediately stopped, preventing injury. This
concept is expanded on in [68], where a pressure-sensitive skin is created
that employs an array of small, 1$\rm{mm}^{2}$ electrodes on flexible circuit
boards, allowing precise measurements of applied forces. Expanding on the idea
of safety in cobotics, the authors in [69] demonstrate a sensor that is
capable of both contact-sensing as well as close-range human-presence
detection. The developed sensor is mounted on an anthropomorphic robot hand,
allowing distance-aware interactions between robot and human. In [70], a
capacitive sensor array that is directly integrated into workers’ clothing is
proposed. The sensors are capable of detecting nearby robots and can stop
those robot’s movements remotely. Conversely, a multi-electrode based
capacitive sensor is presented in both [71] and [72], intended to be mounted
on robotic segments. The latter also demonstrates a system which, by employing
their capacitive sensor, is capable of detecting and stopping a moving robot
arm in the presence of a human. Building on these capabilities, the author in
[73] employs a single capacitive sensor to pull a hospital gown over a human’s
arm, maintaining a safe distance in the process.

To combine the short-range capabilities of capacitive sensors with the long-
range abilities of vision-based systems, we use a modified Luenberger observer
model. Observers are generally used for determining an estimate of the
internal state of a system ([74]). Here, the relevant internal state is the
pose of the tracked object, which can be inferred from the sensing modality
measures. A similar approach to combining sensor measurements was demonstrated
by the authors in [75], where an observer model was used to estimate the pose
of a quadcopter based on visual and interial sensors. Similarly, in [76], an
extended Kalman filter combines sensor information received by odometric and
sonar sensors. Observers have been employed in a wide range of use-cases,
include sensor fusion for electrochemical applications ([77]), the charge
state of lithium-ion batteries ([78]), and to determine the pose of a robotic
gripper ([79]).

The contribution of this article is a multimodal sensing modality for object
pose estimation, containing novel vision- and haptic-based sensing modalities
aimed at minimizing the negative effects of occlusion. We present methods for
object localization based on both haptic sensing and vision. For haptic
sensing, we present a compact capacitive-based hardware system that can be
mounted on a robot arm, and a method to evaluate sensor output to estimate a
considered object’s pose. As stated above, ProcNet (our approach to vision-
based localization) is based on a predictive-coding segmentation method,
combined with a custom object pose estimation model that compares 3D-object
shapes to a segmentation mask. Results of the two pose estimation methods are
evaluated and combined via a modifed Luenberger-type observer model, enabling
occlusion-resistant localization. Lastly, we compare performance to another
established pose estimation model, NVIDIA’s PoseCNN ([80]).

The following article is organized as follows: Section II describes our
perception modalities, with section II-A focusing on active electric
perception, and section II-B on ProcNet. Section III details our method of
merging both approaches, and section IV provides an overview over results.
Lastly, section V concludes the article.

##  II Perception Modalities

In this section, we introduce the two previously discussed localization
methods based on, haptic sensing and vision. We present the two methods
separately, detailing their setup, evaluating their performance, and
determining their capabilities and limitations. For haptic sensing, we present
the capacitive-based hardware setup and corresponding pose estimation model.
For vision-based sensing, we utilize our ProcNet model for frame segmentation
along with our method for subsequent object localization.

###  II-A Active Electric Field Modality

In the following, we present the manner in which we are able to estimate the
pose of electrically sensitive objects in proximity to a haptic sensor.

####  II-A1 Perception Strategy

Haptic, capacitive-based sensors function by generating an electric field and
measuring how nearby objects interact with it. The sensors are composed of at
least two electrodes, one to emit an electric field, and one to measure it. An
example setup can be seen in Fig. 1. In general, a high-frequency field is
preferable, for two reasons. First, sensors that operate at higher frequencies
are more resistant to external noise, as most electric field noise occurs in
the lower frequency bands, at around 0-1kHz ([81]). The second benefit their
ability to generate strong electric fields with simple hardware components; in
general, an increase in field strength at the emitting electrode leads to a
larger signal at the measurement electrode, culminating in higher SNR (Signal-
to-Noise Ratio), and ultimately better perception of nearby objects.
Therefore, capacitive sensors should strive to generate strong electric
fields. However, the strength of an electric field is determined by the
voltage applied to the emitting electrode. Generating high voltages at DC
(direct current) is far more complex than at AC (alternating current).
Sensor’s operating at high frequencies can incorporate an inductor-capacitor
circuit component (also called an LC tank) into their design to convert a low-
voltage input into a high-voltage signal at their resonance frequency. This
circuit is easy to design and implement, making it an ideal driver for the
emitting electrode.

The sensor’s measurement electrode is connected to an amplifying circuit,
comprised of an opamp in gain configuration. The amplified signal is passed to
the ADC (analog-digital converter) of a microcontroller ([82]). We selected
this specific microcontroller due to its fast onboard ADC and high operating
frequencies, allowing us to process the incoming signals in real-time and
compute the respective signal strength using Fourier series decomposition at
operating frequency. At a sampling rate of 1kHz, our configuration can process
up to 40 measurement signals simultaneously. As each sensor has a sensing
range of around 15cm, we can cover a large surface area with minimal amount of
hardware (for additional details, see our previous work [83]). In our setup
(see Fig. 13), we were able to detect and localize objects near a robot arm by
attaching sensors on a large portion of the hull.

####  II-A2 Experimental Results

c
---
r
---
l
---
t
---
b
---
Figure 1: Integration of electrode board with signal processing hardware
(left) and diagram (right) consisting of one excitation electrode (middle,
light blue ring) and five measurement electrodes (dark blue areas, center,
left, right, top, bottom).

Figure 2: Haptic sensing measures recorded when considering a human hand
(left) and forearm (right). The sensor was moved across the horizontal plane,
with electrodes above the target at a constant height of 1cm.

To characterize the sensor’s performance, a series of tests were conducted. To
ensure repeatability of measures, we mounted the electrodes and electronics on
the end effector of a Franka Emika Panda 6 Degree-of-Freedom (DoF) robot arm
([84]). The setup allows us to systematically determine electrode poses in
relation to object pose when recording sensor responses. An example of
measures is shown in Fig. 2, where a human hand and forearm were sampled at a
constant height. Similarly, we obtained sample measures for a series of
objects at various distances, ranging from 1cm to 20cm, in increments of 5mm.
Sensor angles were aligned from 0° to 90° in relation to the horizontal
mounting plane, in increments of 10°. All gathered samples and data is
available online here. Gaussian regression was applied in post-processing to
the measures ([85]). Results indicate that the sensor provides a measurable
reaction to the presence of nearby objects even when they are not directly in
front of the electrodes. Accordingly, for the purpose of proximity detection,
a single sensor could be used to monitor a large volume of space around a
robot arm.

####  II-A3 Sensor Model

Figure 3: Measures obtained with the haptic sensors above various considered
objects. The robot arm was moved so that distance $d(t)$ between electrodes
and object ranged from 0.5 to 20cm. The smooth lines are generated by (1).

To evaluate sensor performance in various scenarii, we derived a model
reflecting sensor behavior. As discussed in section II-A2, measures were
gathered to determine the response to the presence of (and distance to)
various objects. Resulting measures describe the relation between relative
sensor-object distance and recorded voltage $v_{\text{m}}(t)$ (see Fig. 3).
The results show a predictable, well-behaved structure, which can be
approximated using

$v_{\text{d}}(t)=\frac{a_{1}}{1+a_{2}d^{2}(t)}+a_{3},\quad t\geqslant 0,$ (1)

where $v_{\text{d}}(t)$ is the estimated sensor measure in V,
$a_{1},a_{2},a_{3}\in\mathds{R}^{+}$ are parameters describing the sensor’s
behavior (in V, $\text{m}^{-2}$, and V, respectively), and $d(t)$ is the
distance (in m) between sensor center and the considered object’s center of
mass. Parameters depend on both sensor configuration (surface area) as well as
the object’s properties (its shape, and dielectric parameter). The SNR
(Signal-to-Noise Ratio) is computed as described in [86] as

$\sigma\left(v_{\text{d}}(t)\right)=10\log_{10}\frac{\Big{(}\max\left(v_{\text{d}}(t)\right)-\min\left(v_{\text{m}}(t)\right)\Big{)}^{2}}{2\text{Var}[n_{\text{d}}(t)]},$

where $\sigma(v_{\text{d}}(t))\in\mathds{R}$ is the SNR in dB,
$\text{Var}(x(t))\in\mathds{R}^{+}$ is the variance of $x(t)$, and
$n_{\text{d}}(t)$ is the noise present in $v_{\text{d}}(t)$ in V (with all
signal noise being treated as white noise). Note that $t$ spans the entire
considered measurement period. To determine $n_{\text{d}}(t)$, we measured the
electrode output with no object present. Noise variance was assessed to be
about $6.2\text{mV}^{2}$. The SNR can be used as an indicator of a sensor’s
detection range for various objects. For instance, the SNR sinks under 1dB at
a distance of around 15cm in the case of the human forearm. Such a detection
range is useful in supporting human-robot interactions, as shown in section
IV.

####  II-A4 Haptic-based Pose Estimation

Our haptic sensors are capable of estimating the relative position of the
closest point on the surface of a nearby object. By taking samples from
multiple sensors, we can determine various points along an objects hull. Given
enough samples, it is even possible to reconstruct the shape of an object. An
example is shown in Fig. 2, where a series of samples form the outline of a
human hand. The above figure was generated using a single sensor taking
multiple measures over the course of half an hour. For our use case, only a
few sensors will be in range of the object at a given time, resulting in a
limited number of samples. A complete reconstruction of an object’s shape is
not possible. However, a small number of samples are sufficient to estimate a
known object’s pose, defined as

$\displaystyle\nu_{\text{o}}(t)\triangleq\left[\begin{matrix}x_{\text{o}}(t)\\\
\eta_{\text{o}}(t)\end{matrix}\right],\quad t\geqslant 0,$ (2)

where $\nu_{\text{o}}(t)\in\mathds{R}^{6}$ is the pose of object $o$,
$x_{\text{o}}(t)\in\mathds{R}^{3}$ is the position of the object’s center of
mass in m, and $\eta_{\text{o}}(t)\in\mathds{R}^{3}$ the attitude using Euler
coordinates expressed in radians (axis transformations executed in RPY (Roll-
Pitch-Yaw) order). Each sensor delivers an estimated relative position of the
nearest point of an object, as

$x_{\text{r}s}(t)=x_{\text{o}}(t)-x_{s}(t)+e_{\text{x}s}(t),$ (3)

where $s\in\mathds{N}$ is the index of a single sensor,
$x_{s}(t)\in\mathds{R}^{3}$ is the sensor’s current position, and
$e_{\text{x}s}(t)\in\mathds{R}^{3}$ is the estimation error. All sensors are
mounted on a 6-DoF (degrees-of-freedom) robot arm ([84]), with joint angles
$\theta(t)\in\mathds{R}^{6}$ in radians. As the sensors are placed at fixed
positions along the hull, we can use the robot’s kinematics to determine
$x_{s}(t)$. However, computing an object’s pose from sensor and kinematics
information is not trivially possible; as the sensor samples are dependent on
an object’s surface, the computation requires surface analysis to determine
the relation between sensor measures and relative pose. We employed a neural
network, depicted in Fig. 5, to evaluate the relationship. The network takes
the robot’s joint angles $\theta(t)$ and sensor measures $x_{\text{r}s}(t)$ as
input, and outputs the estimated object pose $x_{\text{o}}(t)$ and
$\Omega_{\text{o}}(t)$.

Using experimental data and the model derived in section II-A3, we generated a
training dataset. In simulation, we mounted the sensors on the hull of a robot
and recorded their response to nearby human limbs along with the robot’s joint
angles. In combination with the object pose this dataset, we can use the
dataset to train a neural network on estimating pose according to sensor
measures. All relevant data can be found here. The resulting pose estimation
can be seen in Fig. 4. As shown, the sensor has an estimated sensing range of
around 15cm. At greater distances, the pose estimation error increases
exponentially.

Figure 4: Pose estimation error in relation to distance $d$ from haptic sensor
as RMSE (Root-Mean-Square-Error). $x_{0}(t)$
---
$x_{1}(t)$
---
$x_{n}(t)$
---
$\nu_{\text{o}}(t)$
---
$\theta(t)$
---

$256$
---

ReLU
---

$256$
---

ReLU
---

$256$
---

ReLU
---

$256$
---

ReLU
---

…
---

Measures
---

Sensor
---

Angles
---
Joint
---
Pose
---
Estimate
---
Figure 5: Neural network using robot joint angles and sensor measures to
generate object pose estimation. It contains four fully connected layers, each
comprised of 256 ReLU units.

###  II-B Visual Modality

The visual modality uses a captured camera image and computes the pose of the
considered object using its shape within the frame.

####  II-B1 Perception Strategy

Figure 6: Overview of the PredNet architecture. The width of each coloured box
indicates the spatial scale of the layer component (the larger, the more
resolution). Their height indicates how many feature maps are represented
(also shown by each number). $\Delta t$ indicates connections for which axonal
delays were implemented. Figure 7: Detailed view of the computations performed
at each PredNet layer. The numbers represent kernel size for hGRU and Conv
(convolution) operations, scaling factor for the Upspl (up-sample) and Pool
operations. Note that Conv and Pool operations are absent in the first layer
of PredNet.  Figure 8: Detailed view of how latent activity of the decoded
layers representation components, coming with different resolutions, are
combined in the segmentation prediction process.

Detecting an object’s pose in a camera frame requires two steps. First, we
determine the portion of the frame that pertains to the target object. We use
an approach derived from Predictive Coding methods to generate the
corresponding segmentation mask that identifies an object’s relevant data in
the image. Then, we use a 3D representation of the object to determine the
pose in fixed-frame coordinates. By translating and rotating the 3D
representation to match the object’s segmentation mask, we can estimate a pose
corresponding to the object’s true location. In the following, we will explain
the procedure in detail.

####  II-B2 PredNet Model

The initial segmentation mask is generated by a neural network based on
PredNet architecture (see Fig. 6, 7, and 8). A PredNet is composed of a stack
of layers that represent information and compute prediction error at their own
level. Each layer is composed of a representation component (yellow boxes) and
a prediction error computation component (blue boxes). The representation
components use top-down information (arrows pointing downwards) as well as the
error signal coming from their own layer (arrows pointing left) to generate an
accurate prediction of the incoming bottom-up input of their own layer. They
include recurrent units (circular arrows), in order to retain information for
long periods of time. Going downstream, max pooling operations decrease
spatial resolution, and convolutions increase the number of feature maps. The
prediction error computation components take the difference between the
prediction generated by the representation component at that layer (arrows
pointing right), and the input received from the upstream layer (arrows
pointing upwards). Their output constitutes the local prediction error signal
of that layer. Image frames are sent to the first layer of the network and the
prediction error signal is computed and propagated to the upstream layer,
while top-down activity is projected downstream, based on the activity of the
network related to the previous input frames. At every time step, the sum of
all layers’ prediction error is computed and constitutes the self-supervised
loss of the network. At the same time, the output of the representation
component of all layers except the first one (which is used for image
prediction), is sent to a segmentation decoding module (detailed computation
in Figure 1C). The difference between the decoded segmentation mask and the
true segmentation mask constitutes the supervised loss of the network. It is
computed as the sum of the Dice loss [79] and the Focal loss [80]. An array of
pixels representing the generated segmentation mask is

$m\in\mathds{N}^{h\times w},$ (4)

with $h,w\in\mathds{N}$ as the height and width of the camera image in pixels.
The values in $m$ correspond to the index of a detected object at the
individual pixel coordinates, with a value of $0$ indicating no detected
object.

####  II-B3 Vision-based Pose Estimation

Determining a fixed-frame pose from the segmentation mask is done with a
3D-representation of the object. As camera images are recorded at fixed time
intervals $\Delta t_{\text{c}}$, assume that we are currently looking at frame
$k\in\mathds{N}$ at time instance $t_{k}=k\Delta t_{\text{c}}$. We define

$\begin{matrix}x_{\text{o}k}&=x_{\text{o}}(t_{k}),\\\
\Omega_{\text{o}k}&=\Omega_{\text{o}}(t_{k}),\end{matrix}$ (5)

with $x_{\text{o}k}\in\mathds{R}^{3}$ and
$\Omega_{\text{o}k}\in\mathds{R}^{3}$ as the object pose and rotation at time
$t_{k}$ in m and radians, respectively. The current object pose can be
determined by finding the translation and rotation for which the object shape
most closely matches the segmentation mask $m_{k}$ at frame $k$. We use a 3D
renderer based on the Godot-game engine to to determine

$\hat{m}_{k}=f_{\text{g}}(\hat{x}_{\text{o}k},\hat{\Omega}_{\text{o}k}),$ (6)

where $\hat{x}_{\text{o}k},\hat{\Omega}_{\text{o}k}$ represent an object’s
candidate pose, and $\hat{m}_{k}\in\mathds{N}^{h\times w}$ is the
corresponding segmentation mask. The function $f_{\text{g}}(\centerdot)$ is
used by the renderer to generate candidate segmentation masks from object
poses. To determine the object’s pose at frame $k$, we compare the rendered
candidate mask with the segmentation mask determined by the PredNet as

$\displaystyle\begin{aligned} \left[\begin{matrix}x_{\text{o}k}\\\
\Omega_{\text{o}k}\end{matrix}\right]&=\operatorname*{\arg\\!\max}_{\hat{x}_{\text{o}k},\hat{\Omega}_{\text{o}k}}f_{\text{m}}(m_{k},\hat{m}_{k})\\\
&=\operatorname*{\arg\\!\max}_{\hat{x}_{\text{o}k},\hat{\Omega}_{\text{o}k}}f_{\text{m}}(m_{k},f_{\text{g}}(\hat{x}_{\text{o}k},\hat{\Omega}_{\text{o}k})),\end{aligned}$
(7)

where $f_{\text{m}}(\centerdot)$ is a comparison function quantifying the
overlap between two segmentation masks. For our use case, we assume the
segmentation mask identifies the shape of a single object. Thus, the $m$’s
entries are either 1 (if an object is present at the given pixel) or 0 (if no
object is present). We set

$f_{\text{m}}(m,\hat{m})=\sum_{i=1}^{h}\sum_{j=1}^{w}\frac{4(m_{ij}-\frac{1}{2})(\hat{m}_{ij}-\frac{1}{2})}{wh},$
(8)

with $m_{ij},\hat{m}_{ij}\in[0,1]$ being the $i,j$ coefficients of $m$ and
$\hat{m}$, respectively. This results in pixels where $m$ and $\hat{m}$ have
equal values being weighted as 1, and pixels with unequal values weighted as
-1. Therefore, the maximum value of $f_{\text{m}}(m,\hat{m})$ corresponds to
the masks with the most overlap. To decrease search times, we use information
from the previous frame to compute the current object pose. In particular,
starting from $x_{\text{o}k-1}$, $\Omega_{\text{o}k-1}$, a gradient ascent
search on $f_{\text{m}}(m_{k},\hat{m}_{k})$ is performed to find the maximum
overlap. As $f_{\text{m}}(m_{k},\hat{m}_{k})$ represents the results of
computing the shape of a non-trivial 3D-mesh in camera space, a direct
derivation is not possible. Instead, we estimate the Jacbobian of
$f_{\text{m}}(m_{k},\hat{m}_{k})$ in relation to $x_{\text{o}k}$ and
$\Omega_{\text{o}k}$ by taking multiple samples around a point of interest.
The Jacobian is then be used to follow the local gradient and determine the
object’s pose resulting in the most overlap between $m_{k}$ and $\hat{m}_{k}$.
As we started the gradient ascent from the previous object’s pose at $k-1$ and
object motion between camera frames is fairly low, it is very unlikely that
any local maxima (aside from the global one) are present in the search area.
Images depicting the results of the consecutive steps are shown in Fig. 10.
The resulting modality offers accurate performance (less than 2cm error on
average) up to a distance of around 5m when considering the entire object
pose. Note however that at larger distances the main contributing factor to
positional error is the relative distance to the camera; the modality can
detect an object’s relative pose up to a distance of around 10m when ignoring
the relative distance (see the dashed orange line in Fig. 10).

Figure 9: Pose estimation error in relation to distance $d$ from camera as
RMSE (Root-Mean-Square-Error). The subfigures show errors for the individual
axes. The lower left figure depicts the xy-axis errors (x solid, y dashed) and
RP-attitude. Figure 10: Segmentation and pose estimation of a human forearm.
The top left image shows the camera frame. Top right and bottom left show the
ground truth and the PredNet-generated segmentation masks, respectively. Each
color denotes a separately segmented part of the body. Bottom right shows an
example of pose estimation. The limb pose of the candidate mask that most
closely matches the segmentation is selected.

##  III Multimodal Localization

To combine the results of various sensors, multimodal localization requires
the quantification of the quality of a sensor’s performance. The pose returned
by each modality can be weighed accordingly, with more accurate results being
ranked higher than less precise ones.

###  III-A Modified Luenberger-type Observer

We use a modified Luenberger observer model to combine the two previously
introduced perception modalities. To estimate the pose of a given object, we
use a system defined as

$\displaystyle\dot{x}(t)$ $\displaystyle=f(x(t),u(t))+n_{\text{x}}(t),\quad
x(0)=x_{0},\quad t\geqslant 0,$ $\displaystyle y_{\rm v}(t)$
$\displaystyle=g_{\rm v}(x(t))+n_{\rm v}(t),$ (9) $\displaystyle y_{\rm h}(t)$
$\displaystyle=g_{\rm h}(x(t))+n_{\rm h}(t),$

with $x(t)\in\mathds{R}^{6}$ as the tracked object’s pose in m and rad,
$u(t)\in\mathds{R}^{6}$ as the input in m/s and rad/s,
$n_{\text{x}}(t)\in\mathds{R}^{6}$ as the velocity noise in m/s and rad/s,
$y_{\rm v}(t),y_{\rm h}(t)\in\mathds{R}^{6}$ as the pose in m and rad
retrieved from the visual and haptic sensors, respectively, and $n_{\rm
v}(t),n_{\rm h}(t)\in\mathds{R}^{6}$ as sensor noise in m and rad,
respectively. As the tracked object can move along arbitrary trajectories,
$f(x(t),u(t))$ is a nonlinear function. Due to the nature of our sensors,
$g_{\rm v}(x(t))$ and $g_{\rm h}(x(t))$ are likewise nonlinear. For ease of
exposition, we combine the two sensing modalities and define

$\displaystyle y(t)$ $\displaystyle\triangleq g(x(t))+n(t),$ (10)
$\displaystyle g(x)$ $\displaystyle=\begin{bmatrix}g_{\rm v}(x)^{\rm T},g_{\rm
h}(x)^{\rm T}\end{bmatrix}^{\rm T},$ $\displaystyle n(t)$
$\displaystyle=\begin{bmatrix}n_{\rm v}(t)^{\rm T},n_{\rm h}(t)^{\rm
T}\end{bmatrix}^{\rm T}.$

To merge vision- and haptic-based pose estimations, we implement a modified
Luenberger observer model as described in [74], enabling us to track the
object’s trajectory. We use the observer variables

$\displaystyle\begin{aligned}
\dot{\hat{x}}(t)&=f(\hat{x}(t),u(t))+G(t)z(t),&\hat{x}(0)&=\hat{x}_{0},\\\
z(t)&=y(t)-\hat{y}(t),&t&\geqslant 0,\\\
\hat{y}(t)&=g(\hat{x}(t))+n(t),&&\end{aligned}$ (11)

where $\dot{\hat{x}}(t)\in\mathds{R}^{6}$ is the estimated object pose,
$\hat{y}(t)\in\mathds{R}^{12}$ is the pose measured by our perception
modalities, $z(t)\in\mathds{R}^{6}$ is the pose error, and
$G(t)\in\mathds{R}^{6\times 12}$ is the nonlinear observer gain. As the
observer should track the object’s true pose, our goal is to minimize the
difference between actual pose $x(t)$ and estimated pose $\hat{x}(t)$, we
define the error $e(t)$ as

$\displaystyle\begin{aligned} e(t)&\triangleq x(t)-\hat{x}(t),\qquad
t\geqslant 0,\\\ \dot{e}(t)&=f(x(t),u(t))-f(x(t)-e(t))\\\
&\quad-G(t)\left(y(t)-g(x(t)-e(t)+n(t)\right).\end{aligned}$ (12)

where $e(t)\in\mathds{R}^{6}$ is the observation error in m and rad. A Taylor
series expansion of $\dot{e}(t)$ at steady state $e(t_{\rm s})=0$ is
performed, resulting in

$\displaystyle\dot{e}(t)$
$\displaystyle=\Big{(}A(t)-G(t)C(t)\Big{)}e(t)+e_{\rm r}(t),$ (13)

with

$\displaystyle A(t)=\frac{\partial
f(\hat{x}(t),u(t))}{\partial\hat{x}(t)},\qquad C(t)=\frac{\partial
g(\hat{x}(t)}{\partial\hat{x}(t)},$

where $A(t)\in\mathds{R}^{6\times 6}$ and $C(t)\in\mathds{R}^{12\times 6}$ are
the matrix linearizations of functions $f(\centerdot)$ and $g(\centerdot)$ at
steady state $e(t_{\rm s})=0$, respectively.

$\displaystyle K(t)=$ $\displaystyle P(t)C^{\rm T}(t)R^{-1}(t),$
$\displaystyle\dot{P}(t)=$ $\displaystyle A(t)P(t)+P(t)A^{\rm T}(t)+Q(t)$
$\displaystyle-P(t)C^{\rm T}(t)R^{-1}(t)C(t)P(t),$ (14) $\displaystyle R(t)=$
$\displaystyle\text{diag}(R_{\text{v}},R_{\text{c}}),$ $\displaystyle
P(t_{0})=$ $\displaystyle P_{0},$

where $Q(t)\in\mathds{R}^{6\times 6}$, $R(t)\in\mathds{R}^{12\times 12}$ are
the noise covariance matrices of $n(t)$ and $m(t)$, respectively.
$R_{\text{v}},R_{\text{c}}\in\mathds{R}^{6\times 6}$ are the noise covariance
matrices of the vision- and haptic-based perception modalities, respectively.
$R_{\text{v}}$ and $R_{\text{c}}$ are estimated while individually for the two
sensing modalities. In section III-B, we detail how they are computed.

In our use cases, we found that the trajectories of tracked objects are
smooth, with no external perturbations interrupting their movements.
Therefore, we implemented a canonical 2nd order low-pass filter with
$\omega_{\rm n}\in\mathds{R}^{+}$ as the angular cutoff frequency of the
filter in rad/s, and $\zeta\in\mathds{R}^{+}$ is the filter ramp.

###  III-B Integration of Perception Methods

Figure 11: Pose estimation error comparison between our multimodal approach
(orange) and NVIDIA’s PoseCNN (cyan).

Figure 12: Simulation environment for human limb tracking. The green forearm
silhouette designates the estimated pose, the blue pads are the haptic
sensors, and the illuminated areas represent the cameras field of view.

$R_{\text{v}}$ and $R_{\text{c}}$ are estimated individually for the two
sensing modalities. For haptic sensors, we measured the noise covariance of a
human forearm at various relative positions and orientations from the sensor,
within sensing range of the electrodes (relative distance between object and
electrode less than 15cm). We used Gaussian regression to determine a function
representing the relation between measured relative pose and noise covariance,
resulting in

$R_{\text{c}}(t)=f_{\text{c}}(\hat{x}(t),\theta(t)).$ (15)

For vision-based localization, a different approach was implemented. We
estimate $R_{\text{v}}(t)$ by determining the amount of occlusion; in general,
the more an object is hidden, the less reliable vision-based pose estimation
becomes, resulting in

$R_{\text{v}}(t)=f_{\text{v}}(\hat{m}(t),m(t)),$ (16)

with

$f_{\text{v}}(\hat{m}(t),m(t))=\frac{\sum_{i=1}^{h}\sum_{j=1}^{w}\hat{m}_{ij}(t)-m_{ij}(t)}{\sum_{i=1}^{h}\sum_{j=1}^{w}\hat{m}_{ij}(t)}W_{\text{v}},$
(17)

where $W_{\text{v}}\in\mathds{R}^{6\times 6}$ is a diagonal matrix that
determines the noise strength for each dimension.

Having computed $R_{\text{v}}$ and $R_{\text{c}}$, we can use them to
determine the influence that each sensing modality should have on the
estimated pose. For instance, if an object is close to the haptic sensors, the
sensor’s SNR is high (due to the fact that the emitted electric field is
heavily influenced by the close proximity of the scanned object). Therefore,
the pose estimated by the haptic sensor should be more reliable than if an
object were far away, and the sensor’s measures should be weighted higher.
This is reflected in how $R_{\text{c}}(t)$ influences the computation of
$\hat{x}(t)$ in (14) and subsequently (11). Similarly, the pose determined by
ProcNet is more reliable the less of an object is occluded, meaning its
estimated pose should be weighted higher the less occlusion is observed.
$f_{\text{v}}(\hat{m}(t),m(t))$ determines the level of occlusion by comparing
the segmentation mask computed by our PredNet architecture with the one
generated by the 3D-renderer. As the renderer ignores other objects in the
scene, we can use it’s segmentation mask to determine how much of an object is
obstructed from the camera’s view. We add an additional weight $W_{\text{v}}$
to designate how much occlusion influences the accuracy of vision-based pose
estimation.

Figure 13: Human limb detection and pose estimation with mounted haptic
sensors. The green silhouette designates estimated pose, and the green-blue
pads show the haptic sensors.

##  IV Numerical Simulation

Figure 14: Example of various levels of occlusion (Top left: no occlusion, top
right: light occlusion, bottom left: medium occlusion, bottom right: heavy
occlusion). Trajectories within range of haptic sensor
---
Camera Distance | Occlusion Level |  ProcNet ($e_{\rm v}$) |  Multimodal ProcNet ($e_{\rm m}$) |  PoseCNN ($e_{\rm r}$) |  Multimodal PoseCNN ($e_{\rm rm}$)
Short (0–3m) | Light (0–33%) |  1.73cm |  1.59cm |  1.62cm |  1.59cm
Medium (33–66%) |  2.48cm |  1.83cm |  4.83cm |  2.59cm
Heavy (66–100%) |  5.84cm |  3.27cm |  6.29cm |  3.37cm
Medium (3–6m) | Light (0–33%) |  1.81cm |  1.32cm |  1.57cm |  1.28cm
Medium (33–66%) |  2.59cm |  1.90cm |  5.02cm |  2.18cm
Heavy (66–100%) |  6.04cm |  3.54cm |  6.54cm |  4.53cm
Long (6–10m) | Light (0–33%) |  4.07cm |  3.06cm |  2.65cm |  2.43cm
Medium (33–66%) |  5.39cm |  4.32cm |  2.86cm |  2.90cm
Trajectories outside of range of haptic sensor
Camera Distance | Occlusion Level |  ProcNet ($e_{\rm v}$) |  Multimodal ProcNet ($e_{\rm m}$) |  PoseCNN ($e_{\rm r}$) |  Multimodal PoseCNN ($e_{\rm rm}$)
Short (0–3m) | Light (0–33%) |  1.71cm |  1.71cm |  1.61cm |  1.61cm
Medium (33–66%) |  2.45cm |  2.45cm |  4.79cm |  4.79cm
Heavy (66–100%) |  5.80cm |  5.80cm |  6.25cm |  6.25cm
Medium (3–6m) | Light (0–33%) |  1.83cm |  1.83cm |  1.56cm |  1.56cm
Medium (33–66%) |  2.60cm |  2.60cm |  5.03cm |  5.03cm
Heavy (66–100%) |  5.99cm |  5.99cm |  6.50cm |  6.50cm
Long (6–10m) | Light (0–33%) |  3.98cm |  3.98cm |  2.61cm |  2.61cm
Medium (33–66%) |  5.33cm |  5.33cm |  2.85cm |  2.85cm
Table 1: Average sensor error on relation to various scenarii. The pose errors
of each individual sensing modality (PredNet-based vision as $e_{\rm v}$, and
PoseCNN-based vision as $e_{\rm r}$) is listed. $e_{\rm m}$ and $e_{\rm rm}$
denote the pose errors of our multimodal method, using ProcNet and PoseCNN,
respectively.

A series of simulations were performed to determine the efficacy of our
method, both with and without the presence of occlusion. First, we verified
our setup in a straightforward scene; a human moves his forearm up and down
over an array of haptic sensors (see Fig. 12). An RGB camera is placed within
the scene to deliver vision-based pose estimates via ProcNet. The setup is
repeated multiple times, with varying levels of occlusion. The setup uses both
sensing modalities to determine the pose of the human forearm. If the arm is
too far away from the haptic sensors, pose estimation is performed by vision.
Similarly, if view of the arm is obstructed (and it is in range of the sensor
grid), pose estimation is performed by capacitive measures. In cases where
both modalities are generating valid pose estimates, our observer model
synthesizes the results. We compare our results to a pose estimation algorithm
developed by NVIDIA, PoseCNN ([80]). A second set of simulations was performed
with haptic sensors mounted on a robot arm (see Fig. 13). As previously
described, a moving robot arm occludes nearby objects, leading to difficulties
in pose estimation by vision-based methods alone. By mounting haptic sensors,
we ensure that nearby objects can still be localized via a second method.
Similarly to our previous scenario, we performed multiple tests with different
levels of occlusion. The results can be seen in Fig. 12. Model parameters for
all simulations are configured as
$W_{\text{v}}=\rm{diag}(8,8,8,16\pi,16\pi,16\pi)$, $\omega_{\rm n}=120\pi$ and
$\zeta=\frac{1}{\sqrt{2}}$.

Lastly, we designed a series of datasets to determine the system’s performance
under various degrees of occlusion and at varying distances from a camera (all
relevant data can be found online here). The level of occlusion is determined
by the amount of relevant pixels in the camera image that are obscured; here,
the relevant pixels are those describing the shape of the considered objects.
We therefore determine the degree of occlusion as the sum of all relevant yet
obscured pixels divided by the sum of all pixels describing a considered
object’s shape. Thus, a frame with a completely visible object is noted as
having 0% occlusion, while a frame with a fully obscured object has 100%
occlusion. When evaluating various sensor distances, the level of occlusion
was kept low, at less than 10% (see Figure 14 for examples). Figure 15 shows
an example of how individual modalities localize a human upper limb in our
setup. Haptic sensors have a small detection range (around 20cm, see our
previous work in [83]) as denoted in the top right image. However, within this
range, the sensor can provide sub-millimeter accuracy, as shown previously in
Figure 4. ProcNet provides a larger sensing range, but also with higher
errors. The bottom left image shows camera input, and the bottom right image
shows the resultant localized human pose. The results are shown in Table 1.
ProcNet outperforms PoseCNN when the camera is less than 6m away from the
tracked object, starting at medium occlusion levels. When considering robot
workcells, the setup is often constrained by similar parameters; a camera is
placed within 6m of the robot, which is close enough to monitor the entire
range of motion, but still far enough away to prevent collision. While moving,
the robot will also generate a moderate amount of occlusion, thus preventing
PoseCNN from performing accurate pose estimation. Another effect of occlusion
is the marked performance increase of multimodal pose estimation, with this
approach offering better results than the single-modality, vision-based pose
estimation. Note that while haptic-based pose estimation is very accurate, the
modality can only detect objects at close proximity.

##  V Conclusion

In this article, we described two sensing modalities, one based on vision, the
other on electric perception. Using a Luenberger observer model, we combined
the pose estimates generated by the individual sensors to deliver robust
results in the presence of occlusion. In the future, we aim to improve
usability of our setup. The model can be generalized to detect multiple
targets and determine their respective poses instead of being limited to
tracking single objects. Future work could also involve a more comprehensive
method of estimating object poses based on haptic sensor measures. By taking
the various surface points measured by individual sensors and interpreting
them as a point cloud, a less heuristic approach to pose estimation could be
considered.

ProcNet
---
Figure 15: Localization via visual modality. The left image shows an overview
of the scene, the middle image is the camera input, and the right image
depicts localized limb poses overlaid over the input image.

## References

  * [1] G. Schuh, R. Anderl, J. Gausemeier, M. Ten Hompel, and W. Wahlster, _Industrie 4.0 Maturity Index: Managing the Digital Transformation of Companies_. Utz, Herbert, 2017.
  * [2] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy _et al._ , “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [3] L. Chen, Y. Zhu, G. Papandreou _et al._ , “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision_ , 2018, pp. 801–818.
  * [4] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 580–587.
  * [5] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proc. of the IEEE Int. Conf. on Computer Vision_ , 2017, pp. 2961–2969.
  * [6] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3431–3440.
  * [7] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [10] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6105–6114.
  * [11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [12] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proc. of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [13] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [14] J. Zhu, T. Park, P. Isola, and A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [15] S. Eslami, D. Jimenez Rezende, F. Besse, F. Viola, A. Morcos, M. Garnelo, A. Ruderman, A. Rusu, I. Danihelka, K. Gregor _et al._ , “Neural scene representation and rendering,” _Science_ , vol. 360, no. 6394, pp. 1204–1210, 2018.
  * [16] C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” _International Journal of Computer Vision_ , vol. 126, pp. 973–992, 2018.
  * [17] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, “Unified perceptual parsing for scene understanding,” in _Proceedings of the European conference on computer vision_ , 2018, pp. 418–434.
  * [18] W. Shen, M. Zhou, F. Yang, C. Yang, and J. Tian, “Multi-scale convolutional neural networks for lung nodule classification,” in _Information Processing in Medical Imaging_ , 2015, pp. 588–599.
  * [19] H. Shin, H. R. Roth _et al._ , “Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, pp. 1285–1298, 2016.
  * [20] H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving models from large-scale video datasets,” in _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition_ , 2017, pp. 2174–2182.
  * [21] T. Han, D. Jiang _et al._ , “A novel adversarial learning framework in deep convolutional neural network for intelligent diagnosis of mechanical faults,” _Knowledge-based systems_ , vol. 165, pp. 474–487, 2019.
  * [22] G. Sateesh Babu, P. Zhao, and X. Li, “Deep convolutional neural network based regression approach for estimation of remaining useful life,” in _Database Systems for Advanced Applications: 21st International Conference, DASFAA 2016, Dallas, TX, USA, April 16-19, 2016, Proceedings, Part I 21_. Springer, 2016, pp. 214–228.
  * [23] L. Wen, X. Li, and L. Gao, “A transfer convolutional neural network for fault diagnosis based on resnet-50,” _Neural Computing and Applications_ , vol. 32, pp. 6111–6124, 2020.
  * [24] “Why uber’s self-driving car killed a pedestrian,” The Economist, 2017.
  * [25] A. Fawzi and P. Frossard, “Measuring the effect of nuisance variables on classifiers,” in _Proceedings of the British Machine Vision Conference_ , 2016, pp. 137–1.
  * [26] A. Kortylewski, Q. Liu, H. Wang, Z. Zhang, and A. Yuille, “Combining compositional models and deep networks for robust object classification under occlusion,” in _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2020, pp. 1333–1341.
  * [27] R. McAllister, Y. Gal, A. Kendall, M. Van Der Wilk, A. Shah, R. Cipolla, and A. Weller, “Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning,” in _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence_ , 2017.
  * [28] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li _et al._ , “Challenges and outlook in robotic manipulation of deformable objects,” _IEEE Robotics & Automation Magazine_, vol. 29, no. 3, pp. 67–77, 2022.
  * [29] K. Rajaei, Y. Mohsenzadeh, R. Ebrahimpour, and S.-M. Khaligh-Razavi, “Beyond core object recognition: Recurrent processes account for object recognition under occlusion,” _PLoS computational biology_ , vol. 15, no. 5, p. e1007001, 2019.
  * [30] H. Zhu, P. Tang, J. Park, S. Park, and A. Yuille, “Robustness of object recognition under extreme occlusion in humans and computational models,” _arXiv preprint arXiv:1905.04598_ , 2019.
  * [31] V. A. Lamme, K. Zipser, and H. Spekreijse, “Masking interrupts figure-ground signals in v1,” _Journal of cognitive neuroscience_ , vol. 14, no. 7, pp. 1044–1053, 2002.
  * [32] H. Tang, M. Schrimpf, G. Lotter, Kreiman _et al._ , “Recurrent computations for visual pattern completion,” _Proceedings of the National Academy of Sciences_ , vol. 115, no. 35, pp. 8835–8840, 2018.
  * [33] D. Wyatte, T. Curran, and R. O’Reilly, “The limits of feedforward vision: Recurrent processing promotes robust object recognition when objects are degraded,” _Journal of Cognitive Neuroscience_ , vol. 24, no. 11, pp. 2248–2261, 2012.
  * [34] N. Kriegeskorte, “Deep neural networks: a new framework for modeling biological vision and brain information processing,” _Annual review of vision science_ , vol. 1, pp. 417–446, 2015.
  * [35] M. Schrimpf, J. Kubilius, F. Geiger _et al._ , “Brain-score: Which artificial neural network for object recognition is most brain-like?” _BioRxiv_ , p. 407007, 2018.
  * [36] D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo, “Performance-optimized hierarchical models predict neural responses in higher visual cortex,” _Proceedings of the national academy of sciences_ , vol. 111, no. 23, pp. 8619–8624, 2014.
  * [37] B. A. Olshausen and D. J. Field, “What is the other 85 percent of v1 doing,” _L. van Hemmen, & T. Sejnowski_, vol. 23, pp. 182–211, 2006.
  * [38] R. S. van Bergen and N. Kriegeskorte, “Going in circles is the way forward: the role of recurrence in visual inference,” _Current Opinion in Neurobiology_ , vol. 65, pp. 176–193, 2020.
  * [39] M. Malania, M. H. Herzog, and G. Westheimer, “Grouping of contextual elements that affect vernier thresholds,” _Journal of Vision_ , vol. 7, no. 2, pp. 1–1, 2007.
  * [40] T. P. Saarela, B. Sayim, G. Westheimer, and M. H. Herzog, “Global stimulus configuration modulates crowding,” _Journal of Vision_ , vol. 9, no. 2, pp. 5–5, 2009.
  * [41] T. J. Vickery, W. M. Shim, R. Chakravarthi, Y. V. Jiang, and R. Luedeman, “Supercrowding: Weakly masking a target expands the range of crowding,” _Journal of Vision_ , vol. 9, no. 2, pp. 12–12, 2009.
  * [42] M. Manassi, B. Sayim, and M. H. Herzog, “Grouping, pooling, and when bigger is better in visual crowding,” _Journal of Vision_ , vol. 12, no. 10, pp. 13–13, 2012.
  * [43] M. Manassi, M. H. Herzog _et al._ , “When crowding of crowding leads to uncrowding,” _Journal of vision_ , vol. 13, no. 13, pp. 10–10, 2013.
  * [44] M. Manassi, S. Lonchampt, A. Clarke, and M. H. Herzog, “What crowding can tell us about object representations,” _Journal of Vision_ , vol. 16, no. 3, pp. 35–35, 2016.
  * [45] A. Bornet, O.-H. Choung, A. Doerig, D. Whitney, M. H. Herzog, and M. Manassi, “Global and high-level effects in crowding cannot be predicted by either high-dimensional pooling or target cueing,” _Journal of Vision_ , vol. 21, no. 12, pp. 10–10, 2021.
  * [46] A. Doerig, A. Bornet, O.-H. Choung, and M. H. Herzog, “Crowding reveals fundamental differences in local vs. global processing in humans and machines,” _Vision research_ , vol. 167, pp. 39–45, 2020.
  * [47] A. Bornet, A. Doerig, M. H. Herzog, E. Van der Burg _et al._ , “Shrinking bouma’s window: How to model crowding in dense displays,” _PLoS computational biology_ , vol. 17, no. 7, p. e1009187, 2021.
  * [48] A. Doerig, A. Bornet, R. Rosenholtz, G. Francis, M. H. Herzog _et al._ , “Beyond bouma’s window: How to explain global aspects of crowding?” _PLoS computational biology_ , vol. 15, no. 5, p. e1006580, 2019.
  * [49] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” _Adv. in neural information processing systems_ , vol. 30, 2017.
  * [50] G. Francis, M. Manassi, and M. H. Herzog, “Neural dynamics of grouping and segmentation explain properties of visual crowding.” _Psychological review_ , vol. 124, no. 4, p. 483, 2017.
  * [51] O. Choung, A. Bornet, A. Doerig, and M. H. Herzog, “Dissecting (un) crowding,” _Journal of Vision_ , vol. 21, no. 10, pp. 10–10, 2021.
  * [52] A. Doerig, L. Schmittwilken, B. Sayim, M. Manassi, and M. H. Herzog, “Capsule networks as recurrent models of grouping and segmentation,” _PLoS computational biology_ , vol. 16, no. 7, p. e1008017, 2020.
  * [53] D. Linsley, J. Kim, A. Ashok, and T. Serre, “Recurrent neural circuits for contour detection,” _arXiv preprint arXiv:2010.15314_ , 2020.
  * [54] T. U. Otto, H. Ögmen, and M. H. Herzog, “The flight path of the phoenix—the visible trace of invisible elements in human vision,” _Journal of Vision_ , vol. 6, no. 10, pp. 7–7, 2006.
  * [55] F. Reverter, X. Li, and G. C. Meijer, “Liquid-level measurement system based on a remote grounded capacitive sensor,” _Sensors and Actuators A: Physical_ , vol. 138, no. 1, pp. 1–8, 2007.
  * [56] M. Z. Aslam and T. B. Tang, “A high resolution capacitive sensing system for the measurement of water content in crude oil,” _Sensors_ , vol. 14, no. 7, pp. 11 351–11 361, 2014.
  * [57] C. Chiang and Y. Huang, “A semicylindrical capacitive sensor with interface circuit used for flow rate measurement,” _IEEE Sensors Journal_ , vol. 6, no. 6, pp. 1564–1570, 2006.
  * [58] A. Rivadeneyra, J. Fernández-Salmerón, J. Banqueri, J. A. López-Villanueva, L. F. Capitan-Vallvey, and A. J. Palma, “A novel electrode structure compared with interdigitated electrodes as capacitive sensor,” _Sensors and Actuators B: Chemical_ , vol. 204, pp. 552–560, 2014\.
  * [59] W. Kan, Y. Huang, X. Zeng, X. Guo, and P. Liu, “A dual-mode proximity sensor with combination of inductive and capacitive sensing units,” _Sensor Review_ , vol. 38, no. 2, pp. 199–206, 2018.
  * [60] J. Cheng, O. Amft, and P. Lukowicz, “Active capacitive sensing: Exploring a new wearable sensing modality for activity recognition,” in _Pervasive Computing: 8th Int. Conf._ , 2010, pp. 319–336.
  * [61] S. K. Kundu, S. Kumagai, and M. Sasaki, “A wearable capacitive sensor for monitoring human respiratory rate,” _Japanese Journal of Applied Physics_ , vol. 52, no. 4S, p. 04CL05, 2013.
  * [62] P. L. Fulmek, F. Wandling, W. Zdiarsky, G. Brasseur, and S. P. Cermak, “Capacitive sensor for relative angle measurement,” _IEEE Trans. on Instrumentation and Measurement_ , vol. 51, no. 6, pp. 1145–1149, 2002.
  * [63] P. Roberts, D. D. Damian, W. Shan, T. Lu, and C. Majidi, “Soft-matter capacitive sensor for measuring shear and pressure deformation,” in _Int. Conf. on Robotics and Automation_. IEEE, 2013, pp. 3529–3534.
  * [64] C. Shemelya, F. Cedillos, E. Aguilera, E. Maestas, J. Ramos, D. Espalin, D. Muse, R. Wicker, and E. MacDonald, “3d printed capacitive sensors,” in _IEEE Sensors Journal_ , 2013, pp. 1–4.
  * [65] T. Prescott, N. Lepora, B. Mitchinson, M. Pearson, U. Martinez-Hernandez, and R. Grant, “Active touch sensing in mammals and robots,” in _The Senses: A Comprehensive Reference_. Elsevier, 2020, pp. 79–109.
  * [66] X. Lamy, F. Colledani, F. Geffard, Y. Measson, and G. Morel, “Robotic skin structure and performances for industrial robot comanipulation,” in _IEEE ASME International Conference on Advanced Intelligent Mechatronics_ , 2009, pp. 427–432.
  * [67] B. Šekoranja, D. Bašić, M. Švaco, F. Šuligoj, and B. Jerbić, “Human-robot interaction based on use of capacitive sensors,” _Procedia Engineering_ , vol. 69, pp. 464–468, 2014.
  * [68] E. Pritchard, M. Mahfouz, B. Evans, S. Eliza, and M. Haider, “Flexible capacitive sensors for high resolution pressure measurement,” in _IEEE SENSORS_ , 2008, pp. 1484–1487.
  * [69] D. Göger, H. Alagi, and H. Wörn, “Tactile proximity sensors for robotic applications,” in _IEEE International Conference on Industrial Technology_ , 2013, pp. 978–983.
  * [70] F. Xia, F. Campi, and B. Bahreyni, “Tri-mode capacitive proximity detection towards improved safety in industrial robotics,” _IEEE Sensors Journal_ , vol. 18, no. 12, pp. 5058–5066, 2018.
  * [71] A. Hoffmann, A. Poeppel, A. Schierl, and W. Reif, “Environment-aware proximity detection with capacitive sensors for human-robot-interaction,” in _IEEE RSJ International Conference on Intelligent Robots and Systems_ , 2016, pp. 145–150.
  * [72] T. Schlegl, T. Kröger, A. Gaschler, O. Khatib, and H. Zangl, “Virtual whiskers-highly responsive robot collision avoidance,” in _IEEE RSJ Int. Conf. on Intelligent Robots and Systems_ , 2013.
  * [73] Z. Erickson, M. Collier, A. Kapusta _et al._ , “Tracking human pose during robot-assisted dressing using single-axis capacitive proximity sensing,” _IEEE Robotics and Automation_ , pp. 2245–2252, 2018.
  * [74] H. K. Khalil, “Nonlinear control,” 2015.
  * [75] A. Gómez-Casasola and H. Rodríguez-Cortés, “Sensor fusion for quadrotor autonomous navigation,” in _American Control Conference_. IEEE, 2020, pp. 5219–5224.
  * [76] G. G. Rigatos, “Extended kalman and particle filtering for sensor fusion in motion control of mobile robots,” _Mathematics and computers in simulation_ , vol. 81, no. 3, pp. 590–607, 2010.
  * [77] E. Ross, R. Wagterveld, J. Stigter, M. Mayer, and K. Keesman, “Sensor data fusion in electrochemical applications: An overview and its application to electrochlorination monitoring,” _Computers & Chemical Engineering_, p. 108128, 2023.
  * [78] X. Hu, F. Sun, and Y. Zou, “Estimation of state of charge of a lithium-ion battery pack for electric vehicles using an adaptive luenberger observer,” _Energies_ , vol. 3, no. 9, pp. 1586–1603, 2010.
  * [79] F. Celani, “A luenberger-style observer for robot manipulators with position measurements,” in _2006 14th Mediterranean Conference on Control and Automation_. IEEE, 2006, pp. 1–6.
  * [80] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” _arXiv preprint arXiv:1711.00199_ , 2017.
  * [81] J. Smith, “Electric field imaging,” Ph.D. dissertation, Massachusetts Institute of Technology, 1999.
  * [82] Microchip Technology Inc. PIC32MK0512MCJ064 - High Performance 32-bit MCU with FPU & DSP. [Online]. Available: https://www.microchip.com/en-us/product/PIC32MK0512MCJ064
  * [83] M. Zechmair and Y. Morel, “Active electric perception-based haptic modality with applications to robotics,” in _International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 5950–5957.
  * [84] Franka Emika GmbH. Robot and interface specifications. [Online]. Available: https://support.franka.de/docs/control_parameters.html
  * [85] C. E. Rasmussen and C. K. Williams, _Gaussian processes for machine learning_. Springer, 2006, vol. 1.
  * [86] J. Price and T. Goble, “10 - signals and noise,” in _Telecommunications Engineer’s Reference Book_ , 1993, pp. 10.1–10.15.