# Human Following in Mobile Platforms with Person Re-Identification

Mario Srouji†, Yao-Hung Hubert Tsai†, Hugues Thomas, Jian Zhang †equal
contribution. Apple, contact<EMAIL_ADDRESS>
###### Abstract

Human following is a crucial feature of human-robot interaction, yet it poses
numerous challenges to mobile agents in real-world scenarios. Some major
hurdles are that the target person may be in a crowd, obstructed by others, or
facing away from the agent. To tackle these challenges, we present a novel
person re-identification module composed of three parts: a 360-degree visual
registration, a neural-based person re-identification using human faces and
torsos, and a motion tracker that records and predicts the target person’s
future position. Our human-following system also addresses other challenges,
including identifying fast-moving targets with low latency, searching for
targets that move out of the camera’s sight, collision avoidance, and
adaptively choosing different following mechanisms based on the distance
between the target person and the mobile agent. Extensive experiments show
that our proposed person re-identification module significantly enhances the
human-following feature compared to other baseline variants.

##  I Introduction

Recent advancements in artificial intelligence have made it possible for
humans to cooperate with mobile agents. This cooperation includes autonomous
delivery platforms, service-based autonomous agents taking care of elders in
hospitals, household agents cleaning floors, and telepresence agents creating
new communication methods. These robots cooperate with humans through various
types of interactions, and we argue that the human following feature is one of
the most crucial.

The objective of this paper is to design a human following system that meets
the following requirements: 1) the agent can follow a targeted person, even if
they are moving quickly or away from the agent, 2) the agent can search for
the person if they move out of the camera’s sight, 3) the agent can avoid
obstacles while following the target, and 4) the agent can still identify the
target even when they are in a crowd. Our human following approach is able to
address all of the above requirements, and the core is a novel person re-
identification module.

Our person re-identification module consists of three main components. First,
we introduce a 360-degree registration process to capture different angles of
the target, which is an improvement over standard registration processes in
prior literature [5, 4, 1] that only take a targeted person’s front and back
appearance into account. This process greatly helps us re-identify the target
even when the target is side-facing the camera. Second, our method adopts
different identification models for different body parts (face and torso) to
re-identify the targeted person, unlike prior literature [17, 7, 8] that use
the entire body’s appearance. This approach combines the best of both worlds:
primarily using faces for re-identification, since faces carry the most
distinctive feature among body parts [28]; secondarily using torso for re-
identification when faces are occluded or not present in the camera’s sight.
Third, we consider a motion tracker (e.g., Kalman filter) [3] to predict the
target’s future position, which helps to improve tracking when the target is
occluded or far away. In addition, we show that the motion tracker can help us
reduce the latency of running the person re-identification module, allowing
our mobile agent to still track the target when they are moving quickly.

Our human-following system also consists of several other modules, including
collision avoidance with local planning and path planning, a searching
algorithm that locates the targeted person when out of the camera’s sight, and
a following mechanism based on a RGBD camera and a fish-eye camera. By
integrating all of these modules, our system controls the mobile agent to
follow the targeted person and keep them in the camera’s line of sight, while
also preventing collisions with obstacles. It is important to note that our
system runs directly on the agent’s mobile device for on-device mobile
computation. This is different from prior works [22, 25, 6, 26, 10, 5, 8, 4,
1] that run human following systems on a server and send control commands from
the server to the mobile agent.

As a summary, in this paper, we present a novel human-following system that
incorporates a person re-identification module. We conducted extensive
experiments and user studies to demonstrate that our system outperforms other
baseline approaches in two different conditions: when a targeted person is
walking to random markers in an environment and when a targeted person is
following a course. The environments contain obstacles, and there are always
two people in the environment: one is the target person, and the other is an
interferer. Our system demonstrates strong performance in various metrics,
including the average speed of the agent following the target person, the
average following distance between the agent and the target person, the
average distance to obstacles, the number of times the agent loses the target
person, and the number of times the agent follows the wrong person.

##  II Related Work

Earlier human following techniques [22, 25, 6, 26, 30, 10] can be understood
as performing detection and tracking for humans. Recent approaches [5, 8, 4,
1] consider person re-identification as an additional module on top of
detection and tracking. In the following, we first discuss the approaches
without person re-identification, and then we discuss the approaches with
person re-identification.

Figure 1:  We register a target person using our 360-degree registration
module. Our tracking and re-identification pipeline is then able to find the
targeted person and track them among other individuals. Whenever the target is
lost, our re-id module comes into action. It detects the torso and face (if
available) of each detected person and computes corresponding embeddings. The
target is identified by comparing these embeddings to the saved feature bank.
If the target cannot be found, the robot enters search mode.

As some of the earliest attempts, Nagumo and Ohya [20] asked the targeted
person to carry LED lights for the mobile agent to detect and track. Schlegal
et al. [21] proposed to use the human’s contour and color histogram as the
human following signal. Hirai and Mizoguchi [10] used the human back and
shoulder as the human following signal, and Hu et al. [16] used leg appearance
as the human following signal. Instead of using only cameras on the mobile
agent, Marioka et al. [19] considered external cameras to improve detection
and tracking. Nonetheless, using external cameras lowers the practical
applicability for the human following feature. Other than visual features, Han
and Jin [9] explored the usage of the audio signal for human following. Since
these approaches do not consider person re-identification, their human
following feature can easily fail when the targeted person 1) is occluded or
2) intersects with another individual.

For the approaches with person re-identification ability, Gupta et al. [8] and
Gross et al. [7] presented the use of template matching to identify the
targeted person. Koide et al. [17] used the edges, color, and texture of the
targeted person’s clothes as the identification features. Nonetheless, these
approaches use pre-deep-learning vision features such as SURF [2], which have
been shown to be less effective nowadays [18]. To address this concern, work
[5, 4, 1] presented the use of deep-learning features as identification
features. Our approach is most similar to these works, yet we consider 1) a
different registration process (360-degree registration), 2) person
identification models by multiple parts (faces and torsos), and 3) the
integration with a motion tracker.

##  III Proposed System

In this section, we will first describe the person re-identification module,
and then we will discuss our human following system with the remaining modules
and algorithms.

###  III-A Person Tracking and Re-Identification Pipeline

We propose a person tracking and re-identification pipeline that includes
three components: a 360-degree registration process, a body detection and
tracking pipeline, and a person identification model that uses multiple body
parts (face and torso). We depict this pipeline in Figure 1.

Body detection model. For every frame, our system first runs a body and pose
detector that provides the body bounding boxes and the poses of all humans in
the image. We leverage the open source software provided by
[33]111https://github.com/xingyizhou/CenterNet, which provides good detection
performances.

Motion Tracker. We use a standard open-source Kalman filter motion tracker
[3][27]222https://github.com/abewley/sort which tracks the position of the
detected body bounding boxes. Each tracked bounding box is assigned a
consistent ID across frames allowing tracking of the target even with noisy
detections or when the target is occasionally not detected for a few frames.

Face and Torso Identification Models. To distinguish the targeted individual
from others, we utilize three additional neural-based machine learning models:
a face detector that provides face bounding boxes in the current frame; a face
embedding model that provides a face embedding on a cropped face bounding box;
and a torso embedding model that provides a torso embedding on a cropped torso
bounding box.

Figure 2: Left: when human is facing to the camera. Right: when human is not
facing to the camera. When there is a face detected, our face and torso
identification model will return 1) human bounding box, 2) face embedding, and
3) torso embedding. When there is no face detected, our face and torso
identification model will only return 1) human bounding box and 2) torso
embedding.

We demonstrate the functionality of the face and torso identification models
in Figure 2. First, given the initially detected body poses, we estimate the
person’s face and torso bounding boxes. We draw a square bounding box on the
face with its width and height being the distance between the left and right
ears. Then, we draw a rectangular bounding box on the torso with its width and
height covering the left and right shoulders and hips. However, we have
observed that the pose-induced face bounding boxes may not always be accurate,
especially when the person is not facing the camera. In such cases, the pose-
induced face bounding boxes may be falsely detected. On the other hand, the
pose-induced torso bounding boxes are often accurate.

Then, we obtain better face bounding boxes for all individuals using the
unofficial open-source implementation
333https://github.com/chenjun2hao/CenterFace.pytorch of CenterFace [29]. Note
that if an individual is not facing the camera, their face may not be
detected, resulting in a lower number of face bounding boxes than body
bounding boxes. These detected face bounding boxes are matched with the pose-
induced face bounding boxes based on their intersection of union (IoU) values.
If the corresponding IoU value is greater than $0.75$, we assign the detected
face bounding boxes (from the face detection model) to a person.

Eventually, we produce the face and torso embeddings by passing the face and
torso bounding boxes to two different models: FaceNet [23] and Object-reID
[32]. Again, we use open-source repositories
444https://github.com/timesler/facenet-pytorch
555https://github.com/layumi/Person˙reID˙baseline˙pytorch for reproducibility.
Note that a person may not have a face embedding if their face is not present
in the image. In summary, our models for identifying faces and torsos will
provide the bounding box for each person, as well as the torso embedding.
Additionally, the face embedding may be provided as an optional feature.

360 Registration Process. Initially, we choose a target to follow, and collect
its face and torso embeddings using a 360-registration process as shown in
Figure 3. The targeted person is asked to turn around in front of our mobile
agent. The process usually takes about 20 seconds to complete, during which
our models process hundreds of images. We then randomly select 100 face
embeddings and 100 torso embeddings to form the feature bank.

Figure 3: 360-degree and standard registration process for registering a
targeted person’s face and torso embeddings. 360-degree registration process
captures the embeddings from different angles of the target, while standard
registration process captures only the front-facing and back-facing angels.

Re-identification module. At the initial steps, the tracker is not yet aware
of which ID corresponds to the target individual. The tracker might also lose
the target due to occlusions or the person going out of sight. In those cases,
the re-identification (re-id) module is called. It compares the target’s
embeddings stored in the feature bank with the face and torso embeddings of
all the detected individuals in the scene.

To perform the comparison, we initially compute the average of the embeddings
within the bank. Then, we measure the cosine similarity between the
individual’s face and torso embeddings and the calculated average bank
embeddings. A similarity score is computed as the maximum value between the
face and the torso cosine similarities. If the highest score is above a fixed
threshold ($sim>0.8$), we establish the corresponding person as the target.
Otherwise, our following system starts its searching behavior, as described in
the next section.

On our mobile platform, we can run the person re-id module at 8 fps. This low
frequency makes it hard to follow fast-moving targets. This is why we rely on
the motion tracker most of the time, and make calls to the re-id module only
when necessary. By doing so, we can match the camera frame rate up to 30 fps,
and significantly reduce the reaction time of our following pipeline for the
majority of the time.

###  III-B Proposed Human Following System

Our system for human following, illustrated in Figure 4, consists of our
motion tracking and re-id modules, in addition to other modules including a
dual camera setup, a local planner and collision avoidance module, and a
search behavior to retrieve missing targets. The goal of the system is to
follow a targeted person by controlling the mobile agent’s movements (which
include moving forward, moving back, rotating left, rotating right, etc).

Dual camera setup. The challenges for a following system are different
depending on the distance to its target. If the target is close to the system,
it may go out of sight easily by walking to the side of the mobile agent. If
it is far from the system, obstacles might be in the way, introducing
occlusions or blocked paths. Therefore we use a short-range fish-eye RGB
camera with a large field of view and a long-range RGBD camera with a more
narrow field of view, but providing depth information. As suggested by prior
research [8, 19, 1, 10, 5, 4, 30] depth information is very valuable to follow
humans as it allows for path planning. The switch between the two cameras is
triggered by the measured depth of the person when using the RGBD camera or by
the size of the bounding box when using the fish-eye camera. If the bounding
box height is smaller than $45\%$ of the image, we switch to the RGBD camera.
If the person is closer than $1.5$ meters to the camera, we switch to the
fish-eye camera.

Navigation. When using the fish-eye camera, we use a very simple visual
servoing method, which produces a control command based on the bounding box
position in the image, aiming at centering this bounding box, and maximizing
the height of the bounding box to a certain extent. When using the RGBD
camera, we are able to estimate the depth of the target and thus its $(x,y)$
position relative to the system. Therefore the control command can be produced
directly with the local planner described below, using this local goal.

Local Planner and Collision Avoidance. To ensure the mobile agent’s safety, we
take into account any obstacles that may be present between the agent and its
target. This is done through the use of the SAFER local planner and collision
avoidance algorithm [24]. This algorithm can either take a control command or
a local goal as input and it outputs a safe control command that avoids
obstacles.

Figure 4:  Our mobile agent uses a dual-camera setup. A fish-eye RGB camera
for short-range following using simple visual servoing and a RGBD camera for
longer-range following using a local navigation goal. We leverage SAFER, a
local planer and collision avoidance module [24] to handle safe navigation
towards the target. In case the robot is in search mode, it will turn in the
direction where the target was last seen.

Search behavior. Thanks to the fish-eye camera and visual servoing, it is rare
to lose the person when it walks to the side of the mobile agent.
Nevertheless, this still might happen, and occlusions might also lead to the
system losing its target. In this case, the system enters into search mode.
When the system is using the RGBD camera, the last known position of the
target is kept as the local goal for a short period of time (e.g. 2 seconds).
After that, or when using the fish-eye camera, our mobile agent will stop
moving forward and rotate in the direction where the target was last seen.
This allows our mobile agent to have a better chance at re-finding the target.

##  IV Experiments

Our system, designed to track humans, is installed in a mobile agent that
includes a base from Diablo Robotics [12], Nvidia Jetson Orin [14] compute, a
J5 Create JVCU360 camera [11], a Zed camera [15], and a Livox Mid-360 lidar
sensor [13]. We conducted experiments in indoor office environments where the
mobile agent followed a target person. We tested two situations: the target
person walking to random markers and the target person following a course. The
environments included obstacles, and always had two people present, one being
the target person and the other being the interferer.

###  IV-A Comparison Approaches

For our experiments, we began by considering human-following systems without
person re-identification. We first looked at the variant that only tracks the
human, which is similar to the prior method [31]. Next, we looked at variants
that use face-only or torso-only identification models. It’s worth noting that
the variant that only considers torso-only identification models is similar to
the prior method [4], which uses a full-body identification module (e.g.,
clothing color, height, gait) for human tracking.

We then considered our method without using a motion tracker, and finally, we
looked at our method that uses only visual servo or only path planning to
drive the mobile agent. Our goal in designing these baseline variants was to
show the importance of 1) person re-identification in the human-following
feature and 2) the individual parts of our person re-identification system
(e.g., face-torso identification models, motion tracker, mobile agent driving
by visual servo or path planning). See Table I for our summary.

Variants | components in our system
---|---
Motion Model | Torso-id Model | Face-id Model | Action by Visual Servo | Action by Path Planning
Ours_w/o_reid [31] | ✓ | ✗ | ✗ | ✓ | ✓
Ours_w/o_motion | ✗ | ✓ | ✓ | ✓ | ✓
Ours_w/o_torso | ✓ | ✗ | ✓ | ✓ | ✓
Ours_w/o_face [4] | ✓ | ✓ | ✗ | ✓ | ✓
Ours_w/o_visualservo | ✓ | ✓ | ✓ | ✗ | ✓
Ours_w/o_pathplanning | ✓ | ✓ | ✓ | ✓ | ✗
Ours | ✓ | ✓ | ✓ | ✓ | ✓
TABLE I:  Baseline methods studied in the paper for human following. We dissect our system and get rid of certain components to construct the variants. Variants | Avg. Speed (m/s) | Avg. Following | Avg. Distance | # of Losing Target | # of Following
---|---|---|---|---|---
Distance (m) | to Obstacles (m) | Wrong Person
Participants walking to random markers ($75$ trials)
Ours_w/o_reid [31] | 0.87 | 1.3 | 0.86 | 12 / 75 | 7 / 75
Ours_w/o_motion | 0.63 | 1.4 | 0.81 | 5 / 75 | 2 / 75
Ours_w/o_torso | 0.79 | 1.9 | 0.79 | 9 / 75 | 2 / 75
Ours_w/o_face [4] | 0.93 | 1.3 | 0.77 | 5 / 75 | 5 / 75
Ours_w/o_visualservo | 0.71 | 1.5 | 1.15 | 4 / 75 | 1 / 75
Ours_w/o_pathplanning | 0.97 | 1.7 | 0.48 | 6 / 75 | 1 / 75
Ours | 0.92 | 1.2 | 0.83 | 3 / 75 | 1 / 75
Participants following courses ($15$ trials)
Ours_w/o_reid [31] | 1.16 | 1.6 | 0.82 | 7 / 15 | 5 / 15
Ours_w/o_motion | 0.83 | 2.3 | 0.78 | 4 / 15 | 1 / 15
Ours_w/o_torso | 0.92 | 2.1 | 0.69 | 5 / 15 | 3 / 15
Ours_w/o_face [4] | 1.19 | 1.4 | 0.77 | 3 / 15 | 2 / 15
Ours_w/o_visualservo | 1.03 | 1.7 | 0.91 | 3 / 15 | 0 / 15
Ours_w/o_pathplanning | 0.81 | 1.9 | 0.39 | 4 / 15 | 1 / 15
Ours | 1.24 | 1.5 | 0.74 | 2 / 15 | 0 / 15
TABLE II:  Quantitative results on 1) the average speed of agents when
following human, 2) the average following distance of the agent, 3) the
average distance to obstacles, 4) the number of losing the target person, and
5) the number of following a wrong person. We consider two conditions: 1) $5$
participants walking to $15$ random markers and 2) $5$ participants following
$3$ different courses. For each trial, its environment contains obstacles,
such as chairs, desks, and walls, and an interferer (trying to interfere the
agent following the target person).

###  IV-B Metrics

To determine how to successfully follow a human, we conducted user studies to
establish metrics. Our experiments involved 5 participants walking to 15
random markers (75 trials in total) and 5 participants following 3 different
courses (15 trials in total). These trials were conducted in various office
environments, each containing obstacles such as chairs, desks, cabinets, or
walls, as well as an interferer who randomly walked in the same environment to
interrupt the agent following the target person.

To measure success, we report 1) the average speed of the agent when following
the target person, 2) the average distance between the agent and followed
person, 3) the average distance to obstacles according to the Livox Mid-360
lidar [13], 4) the number of times the agent loses the target person, and 5)
the number of times the agent follows the wrong person. Additionally, we
created a questionnaire and asked participants to rate their experience on a
scale from 1 to 10, reporting the average score for each question.

The questions are

  1. Q1

[Safety], how safe did you feel while the robot was following you? (With 1
being not safe at all and 10 being very safe)

  2. Q2

[Collision to Participants], how often did you feel the robot might collide
with you? (With 1 being never and 10 being every time)

  3. Q3

[Collision to other Objects], how often did you feel the robot might collide
with another object? (With 1 being never and 10 being every time)

  4. Q4

[Following Accuracy], how accurately did the robot follow you? (With 1 being
not accurate at all and 10 being very accurate)

  5. Q5

[Navigation Smoothness], how smoothly did the robot navigate to follow you?
(With 1 being not smoothly at all and 10 being at very smoothly)

TABLE III:  Quantitative results on questions for participants (see Section IV-B). Each question is rated from $1$ to $10$, and $\uparrow$ means the higher the better; $\downarrow$ means the lower the better. Variants | Q1($\uparrow$) | Q2($\downarrow$) | Q3($\downarrow$) | Q4($\uparrow$) | Q5($\uparrow$)
---|---|---|---|---|---
Participants walking to random markers ($75$ trials)
Ours_w/o_reid [31] | 8.0 | 2.0 | 2.0 | 4.0 | 7.0
Ours_w/o_motion | 9.3 | 2.0 | 2.1 | 7.2 | 7.0
Ours_w/o_torso | 8.0 | 2.0 | 2.0 | 5.2 | 7.4
Ours_w/o_face [4] | 8.0 | 2.0 | 3.4 | 6.2 | 7.1
Ours_w/o_visualservo | 7.0 | 2.1 | 2.0 | 6.0 | 7.0
Ours_w/o_pathplanning | 5.1 | 4.0 | 6.0 | 5.3 | 4.6
Ours | 8.3 | 2.1 | 3.4 | 8.2 | 7.5
Participants following courses ($15$ trials)
Ours_w/o_reid [31] | 9.0 | 2.0 | 2.2 | 3.4 | 8.0
Ours_w/o_motion | 9.0 | 2.0 | 2.0 | 6.0 | 7.0
Ours_w/o_torso | 9.0 | 2.3 | 2.0 | 4.0 | 6.0
Ours_w/o_face [4] | 9.2 | 2.4 | 2.0 | 7.0 | 7.0
Ours_w/o_visualservo | 9.0 | 2.0 | 3.0 | 8.4 | 9.0
Ours_w/o_pathplanning | 6.0 | 5.0 | 5.2 | 4.4 | 5.1
Ours | 9.1 | 2.0 | 2.0 | 7.0 | 8.3

###  IV-C Quantitative evaluation

The quantitative results of our experiments are compiled in Table II. First,
we compared methods with (Ours) and without re-identification (re-id) models
(Ours_w/o_reid). We found that using re-id models can improve the speed of the
agent, reduce the following distance to the person being followed, and
significantly decrease the number of lost targets and incorrect following.
This observation suggests that using re-id models can greatly improve the
human following feature.

Similarly, when comparing methods with (Ours) and without the motion tracker
(Ours_w/o_motion), we observe that the speed of the agent when following a
person is greatly improved after using the motion tracker. We argue that this
is because, without using the motion tracker, the agent must perform re-
identification with re-id models most of the time, resulting in high latency.

Then, we discuss methods using different re-id models. Comparing
Ours_w/o_torso and Ours_w/o_face, we find that using face re-id models leads
to higher average speed of the agent, lower average following distance, and
far less following of the wrong person. This can be seen when comparing
Ours_w/o_torso, which results in a higher number of lost targets, but a lower
number for following an incorrect person when the agent is walking to a random
marker. Ours_w/o_face has less overall lost targets, but all of the lost
targets resulted in an incorrect following. We argue that this is because face
is a stronger feature for person identification, however since many times the
person may be facing away from the agent, the torso may be the only
identification feature for the agent. Combining the best of both worlds, Ours
improves over Ours_w/o_torso and Ours_w/o_face, suggesting that both torso and
face re-id models are crucial for human following.

Lastly, we discuss methods using different approaches (visual servo, path
planning, or both) for the agent to take actions. Comparing
Ours_w/o_visualservo and Ours_w/o_pathplanning, the main difference is that
the average distance to obstacles is much larger when using path planning
only, and the average speed is lower. In other words, path planning is a more
conservative approach for collision avoidance than visual servo. On the other
hand, Ours determines when to use visual servo or path planning depending on
the distance of the target person to the agent. We find this to be a good
combination of both approaches.

###  IV-D User study

In addition to raw measurements, we designed a user study to evaluate how user
would feel around our following system. The questions listed in Section IV-B
aim at evaluating different aspects of the following experience.

Questions 1, 2, and 3 are overall focusing on the feeling of safety that
participants got from the system. Overall, the scores reflect a pretty good
feeling of safety for all of the ablated variants of our method except for
Ours_w/o_pathplanning. This observation suggests that using path planning for
the agent’s action is crucial for avoiding collisions. The best safety scores
are obtained without the motion tracker, which is not surprising as it is the
slowest variant.

Question 4 tells us how well participants believe the following system
performed. Our full system has the highest score of all, showing that every
component of our system is crucial for the following accuracy. In particular,
we notice how crucial our re-id module is as the lowest score is obtained by
Ours_w/o_reid. We also observe the importance of using path planning and the
benefits of combining both face and torso re-id models.

Finally, the users rated the smoothness of our system navigation in question
5. The low score obtained by Ours_w/o_pathplanning suggests that using path
planning helps the system to navigate more smoothly.

##  V Conclusion

This paper addresses the issue of human following for mobile agents. We have
observed that person re-identification is crucial for human following,
particularly when the targeted individual is in a crowd or interacting with
other people. To tackle this problem, we have developed a new person re-
identification module that consists of three core components:
360-registration, identification models that use both face and torso, and
motion tracker. We conducted a series of experiments and found that our
approach outperforms previous methods, highlighting the effectiveness of each
component in our person re-identification module. We believe that our work can
help to improve the development of human following mobile agents and
contribute to the advancement of artificial intelligence.

## References

  * [1] Redhwan Algabri and Mun-Taek Choi.  Deep-learning-based indoor human following of mobile robot using color feature.  Sensors, 20(9):2699, 2020.
  * [2] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9, pages 404–417. Springer, 2006.
  * [3] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 3464–3468, 2016.
  * [4] Bao Xin Chen, Raghavender Sahdev, and John K Tsotsos.  Integrating stereo vision with a cnn tracker for a person-following robot.  In Computer Vision Systems: 11th International Conference, ICVS 2017, Shenzhen, China, July 10-13, 2017, Revised Selected Papers 11, pages 300–313. Springer, 2017.
  * [5] Bao Xin Chen, Raghavender Sahdev, and John K Tsotsos.  Person following robot using selected online ada-boosting with stereo camera.  In 2017 14th conference on computer and robot vision (CRV), pages 48–55. IEEE, 2017.
  * [6] Jennifer Goetz, Sara Kiesler, and Aaron Powers.  Matching robot appearance and behavior to tasks to improve human-robot cooperation.  In The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003., pages 55–60. Ieee, 2003.
  * [7] Horst-Michael Gross, Andrea Scheidig, Klaus Debes, Erik Einhorn, Markus Eisenbach, Steffen Mueller, Thomas Schmiedel, Thanh Q Trinh, Christoph Weinrich, Tim Wengefeld, et al.  Roreas: robot coach for walking and orientation training in clinical post-stroke rehabilitation—prototype implementation and evaluation in field trials.  Autonomous Robots, 41:679–698, 2017.
  * [8] Meenakshi Gupta, Swagat Kumar, Laxmidhar Behera, and Venkatesh K Subramanian.  A novel vision-based tracking algorithm for a human-following mobile robot.  IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(7):1415–1427, 2016.
  * [9] Jongho Han and Taeseok Jin.  Sound source based mobile robot control for human following in a networked intelligent space.  International Journal of Control and Automation, 8(7):67–74, 2015\.
  * [10] Nobutaka Hirai and Hiroshi Mizoguchi.  Visual tracking of human back and shoulder for person following robot.  In Proceedings 2003 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2003), volume 1, pages 527–532. IEEE, 2003.
  * [11] https://j5create.com.  j5create, 2023.
  * [12] https://shop.directdrive.com.  Diablo robotics, 2023.
  * [13] https://www.livoxtech.com/mid 360.  livox, 2023.
  * [14] https://www.nvidia.com/en us/.  Nvidia, 2023.
  * [15] https://www.stereolabs.com/zed 2/.  Zed camera, 2023.
  * [16] Jwu-Sheng Hu, Jyun-Ji Wang, and Daniel Minare Ho.  Design of sensing system and anticipative behavior for human following of mobile robots.  IEEE Transactions on Industrial Electronics, 61(4):1916–1927, 2013\.
  * [17] Kenji Koide and Jun Miura.  Identification of a specific person using color, height, and gait features for a person following robot.  Robotics and Autonomous Systems, 84:76–87, 2016.
  * [18] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * [19] Kazuyuki Morioka, Joo-Ho Lee, and Hideki Hashimoto.  Human-following mobile robot in a distributed intelligent sensor network.  IEEE Transactions on Industrial Electronics, 51(1):229–237, 2004\.
  * [20] Yousuke Nagumo and Akihisa Ohya.  Human following behavior of an autonomous mobile robot using light-emitting device.  In Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No. 01TH8591), pages 225–230. IEEE, 2001.
  * [21] Christian Schlegel, Jörg Illmann, Heiko Jaberg, Matthias Schuster, and Robert Wörz.  Vision based person tracking with a mobile robot.  In BMVC, pages 1–10. Citeseer, 1998.
  * [22] Jean Scholtz.  Theory and evaluation of human robot interactions.  In 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the, pages 10–pp. IEEE, 2003.
  * [23] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [24] Mario Srouji, Hugues Thomas, Hubert Tsai, Ali Farhadi, and Jian Zhang.  Safer: Safe collision avoidance using focused and efficient trajectory search with reinforcement learning.  arXiv preprint arXiv:2209.11789, 2022.
  * [25] Aaron Steinfeld, Terrence Fong, David Kaber, Michael Lewis, Jean Scholtz, Alan Schultz, and Michael Goodrich.  Common metrics for human-robot interaction.  In Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, pages 33–40, 2006.
  * [26] Stefan Waldherr, Roseli Romero, and Sebastian Thrun.  A gesture based interface for human-robot interaction.  Autonomous Robots, 9:151–173, 2000.
  * [27] Greg Welch, Gary Bishop, et al.  An introduction to the kalman filter.  1995\.
  * [28] Di Wu, Si-Jia Zheng, Xiao-Ping Zhang, Chang-An Yuan, Fei Cheng, Yang Zhao, Yong-Jun Lin, Zhong-Qiu Zhao, Yong-Li Jiang, and De-Shuang Huang.  Deep learning-based methods for person re-identification: A comprehensive review.  Neurocomputing, 337:354–371, 2019.
  * [29] Yuanyuan Xu, Wan Yan, Haixin Sun, Genke Yang, and Jiliang Luo.  Centerface: Joint face detection and alignment using face as point.  In arXiv:1911.03599, 2019.
  * [30] Ningshi Yao, Emily Anaya, Qiuyang Tao, Sungjin Cho, Hongrui Zheng, and Fumin Zhang.  Monocular vision-based human following on miniature robotic blimp.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3244–3249. IEEE, 2017.
  * [31] Takashi Yoshimi, Manabu Nishiyama, Takafumi Sonoura, Hideichi Nakamoto, Seiji Tokura, Hirokazu Sato, Fumio Ozaki, Nobuto Matsuhira, and Hiroshi Mizoguchi.  Development of a person following robot with vision based target detection.  In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5286–5291. IEEE, 2006.
  * [32] Zhedong Zheng, Xiaodong Yang, Zhiding Yu, Liang Zheng, Yi Yang, and Jan Kautz.  Joint discriminative and generative learning for person re-identification.  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [33] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  In arXiv preprint arXiv:1904.07850, 2019.


tiny[table]font=small [table]capposition=top (eccv) Package eccv Warning:
Package ‘hyperref’ is loaded with option ‘pagebackref’, which is *not*
recommended for camera-ready version

11institutetext: 1University of Bristol, 2Google Zurich, 3NVIDIA

#  Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for
Reconstructing Challenging Surfaces

Mauro Comi 11 Alessio Tonioni 22 Max Yang 11 Jonathan Tremblay 33 Valts Blukis
33 Yijiong Lin 11 Nathan F. Lepora∗ 11 Laurence Aitchison∗ 11

###### Abstract

Touch and vision go hand in hand, mutually enhancing our ability to understand
the world. From a research perspective, the problem of mixing touch and vision
together is underexplored and presents interesting challenges. To this end, we
propose Tactile-Informed 3DGS, a novel approach that incorporates touch data
(local depth maps) with multi-view vision data to achieve surface
reconstruction and novel view synthesis. Our method optimises 3D Gaussian
primitives to accurately model the object’s geometry at points of contact. By
creating a framework that decreases the transmittance at touch locations, we
achieve a refined surface reconstruction, ensuring a uniformly smooth depth
map. Touch is particularly useful when considering non-Lambertian objects
(e.g. shiny or reflective surfaces) since contemporary methods tend to fail to
reconstruct with fidelity specular highlights. By combining vision and tactile
sensing, we achieve more accurate geometry reconstructions with fewer images
than prior methods. We conduct evaluation on objects with glossy and
reflective surfaces and demonstrate the effectiveness of our approach,
offering significant improvements in reconstruction quality.

Figure 1:  We combine multi-view visual data and tactile sensing information
within a 3D Gaussian Splatting framework for accurate geometry reconstruction
and novel view synthesis of challenging surfaces. On the left, a robotic arm
is equipped with tactile sensors. On the right, we show results on novel view
synthesis and depth reconstruction, using a five minimal-view setting and
tactile input (first row) – touches not only improve surface reconstruction,
but also strengthen novel view synthesis.

##  1 Introduction

Humans perceive the world through a multitude of senses, and often, visual
perception alone is not sufficient to fully grasp the intricacy of the world.
When only using vision, it is possible to misunderstand shapes, materials,
scale, etc. of the world and its components. Tactile sensing, however,
provides consistent geometric information that complements visual perception,
particularly in scenarios where visual data may be insufficient. Thus, in this
paper we are interested in exploring the problem of combining two sources of
sensing, vision and touch, for novel view synthesis and shape reconstruction
from a set of views and touches (see Figure 1). While non trivial, the task of
combining vision and touch for accurate 3D reconstruction presents numerous
advantages. Current vision-based methods often struggle with non-Lambertian
surfaces [13], as well as scenarios where only a limited number of views are
available. These scenarios are common in real-world settings, such as robotics
manipulation, VR/AR, and 3D modeling, where physical constraints or occlusions
can restrict the number of accessible viewpoints. Unlike visual data, tactile
sensing provides consistent geometric information regardless of light-
dependent effects, and having data from an additional modality can support 3D
reconstruction in the context of both limited (5 views) and dense visual data.
Conversely, while tactile sensing provides valuable geometric information, it
is limited by the fact that it only captures partial information about an
object’s surface - e.g., when grasping an object, the hand rarely touches the
entire surface. Therefore, the ability to seamlessly combine sparse
observations across modalities and scenarios with varying viewpoint
availability is what we aim to achieve in this work.

We believe to be the first to explore the problem of solely using RGB and 3D
touches to reconstruct an object and allow for novel view synthesis. Prior
work has looked at the problem of mixing vision and touch signals [17, 18] for
point cloud retrieval. Other methods have focused on reconstructing object
shapes using touch signals [16, 2]. While these methods tend to generate
valuable object surfaces, it is rare for them to perform novel view synthesis
from touch only; and even if they do, generating novel views is not their main
objective [21]. Ultimately, these methods are applied to Lambertian objects,
and recently, vision-only based methods have shown tremendous success in novel
view synthesis and 3D reconstructions for this setting [13, 8]. These
advancements are quite astonishing; however, because they leverage volumetric
rendering, their ability to reconstruct surfaces is limited, especially in the
presence of non-Lambertian surfaces. Moreover, depth sensors tend to fail in
this setting as the highly reflective surfaces can create discontinuities in
depth measurements [22]. As such, the problem of recovering textured non-
Lambertian objects from multiple views has been studied recently leveraging
inverse rendering [11]. This approach can produce very accurate results when
the method is allowed an extensive amount of views and unbounded compute time
(24+ hours) [11, 24].

Motivated by Liu et al.’s [11] research on non-Lambertian 3D reconstructions,
we propose Tactile-Informed 3DGS, a novel multimodal interaction approach that
integrates tactile sensing and vision within 3D Gaussian Splatting for
challenging object reconstruction. Differently from prior work, we streamlined
this problem by reducing the required number of viewpoints and computation
time through the incorporation of sparse 3D touch observations on the object’s
surface. Unlike visual data, tactile sensing is not affected by the presence
of specular highlights, reflections, and translucenct materials. A key insight
is that, by regularising the 3D transmittance of Gaussians around the touch
locations, our approach directly guides the optimisation procedure with
precise, direct information regarding object geometry. Acknowledging the
sparse nature of tactile observations, we further refine our method by
introducing an unsupervised regularisation term to smooth the predicted camera
depth maps.

By leveraging multimodal sensing, Tactile-Informed 3DGS provides three main
contributions: (1) state-of-the-art geometry reconstruction on reflective and
glossy surfaces, (2) faster scene reconstruction 10x over prior arts, and (3)
improved performance in geometry reconstruction and novel view synthesis with
minimal views. We evaluate our method on object-centric datasets containing
glossy and reflective surfaces, namely Shiny Blender [24] and Glossy Synthetic
[11] datasets. We also propose a real-world dataset of a highly shiny object
(metallic toaster) from which we capture a multiview set of images as well as
real robot touches. Our results show that multimodal sensing yields equal or
better geometry reconstruction than methods that incorporate notion of light
scattering, while still being an order of magnitude faster. Moreover, by
introducing touch as an additional sensing modality, our method increases
robustness and mitigates performance degradation when working with fewer
viewpoints.

##  2 Literature Review

###  2.1 Tactile sensing for 3D object reconstruction

Tactile sensing for object reconstruction is an emerging research field that
integrates techniques from computer vision, graphics, and robotics with high-
resolution, optical-based tactile sensors [26, 9, 10]. The study of affordable
and open-source tactile sensors has contributed to advancements in 3D shape
reconstruction through direct object interactions. These sensors, which
capture local depth maps or marker-based images, can be leveraged to collect
information about the surface at touch location.

Recently we have seen the introduction of datasets that incorporate touch,
such as ObjectFolder [4]. This dataset, however, does not include multiview
images of the objects and it does not include non-Lambertian objects, whereas
our method focuses on mixing touch and vision. As such, we propose a dataset
that includes multiple views of a metallic object. Suresh et al. [20]
introduced a dataset that both mixes an external depth sensor and touch
sensing of objects, although they do not provide a vision-based multiview
system.

Wang et al. [25, 26] proposed to utilise tactile exploration for object
reconstruction through voxel-based representations. Smith et al. [17, 18]
introduced a novel approach to decouple vision and tactile sensing processes,
employing a Convolutional Neural Network (CNN) to map tactile readings into
local contact surface representations. The reconstruction was achieved by
combining geometrical features extracted with tactile sensors with feature
embeddings collected using RGB cameras. Their experiments focused mainly on
synthetic data containing objects with simple material proprieties (single
color and only Lambertian materials) and in the end, their method does not
allow for novel view synthesis.

###  2.2 Novel-view synthesis on reflective surfaces

Novel-view synthesis (NVS) from sparse observations is a key area of research
within the domain of computer graphics, 3D computer vision, and robotics.
Methods based on Neural Radiance Fields (NeRF) are extremely successful at
synthesizing photorealistic images by modeling the volumetric density and
color of light rays within a scene with a Multi-Layer Perceptron (MLP) [13,
14, 1, 15]. 3D Gaussian Splatting (3DGS) [8] has been proposed as an
alternative to NeRF. Instead of modelling the radiance field implicitly
through a neural network, it is modelled explicitly by a large set of 3D
Gaussians. This has the benefit of a simpler explicit model that offers faster
training and rendering than even highly optimized NeRF implementations [14].
In this work we propose to extend the 3D Gaussian splatting framework to
reason about non-Lambertian surfaces and touch data.

The application of volumetric rendering equations causes difficulties for both
NeRFs and 3DGS in accurately representing specular and glossy surfaces. This
is because it is challenging to interpolate the outgoing radiance across
different viewpoints as it tends to vary significantly around specular
highlights [24]. Ref-NeRF[24] was proposed to address this limitation by
replacing NeRF’s view-dependent radiance parameterisation with a
representation of reflected radiance, leveraging spatially varying scene
properties and a regulariser on normal vectors. NeRFReN [5] models reflections
by fitting an additional radiance field for the reflected world beyond the
mirror, however this method supports reflections on planar surfaces only. Ref-
NeRF focuses on direct light, thus struggling with indirect lights from nearby
light emitters. NeRO [11], on the other hand, proposes to model direct and
indirect illumination separately using the split-sum approximation [7]. This
method divides the equation for surface reflectance into two components: one
that pre-integrates the lighting and another that integrates the
characteristics of the material. By directly approximating indirect specular
lighting, NeRO achieves improved geometry reconstruction. The results they
obtained for material retrieval and surface reconstruction are impressive, but
as we show in this work, when the method does not have access to an extensive
dataset of views, it tends to generate unconvincing results (see experiments
section).

##  3 Preliminaries

###  3.1 3D Gaussian Splatting

3D Gaussian Splatting [8] is an explicit radiance field technique for modeling
3D scenes as a large set of 3D Gaussians. Specifically, a scene is
parameterised as a set of $N$ anisotropic 3D Gaussians, each defined by a mean
vector $\boldsymbol{\mu}\in\mathbb{R}^{3}$, a covariance matrix
$\mathbf{\Sigma}\in\mathbb{R}^{3}$, an opacity $\alpha\in\mathbb{R}^{1}$ , and
a view-dependent color vector $\mathbf{c}\in\mathbb{R}^{m}$ encoded as
spherical harmonics. These parameters are optimised during the training
process through backpropagation. The influence of a Gaussian on a point
$\mathbf{x}\in\mathbb{R}^{3}$ is determined by:

$f(\mathbf{x};\mathbf{\Sigma},\boldsymbol{\mu})=\exp(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^{T}\mathbf{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})).$
(1)

During the optimisation process, the scene is rendered to a 2D plane by
calculating each Gaussian’s projected covariance
$\mathbf{\Sigma^{\prime}}\in\mathbb{R}^{2}$ and position
$\boldsymbol{\mu}^{\prime}\in\mathbb{R}^{2}$, following:

$\mathbf{\Sigma^{\prime}}=\mathbf{J}\mathbf{W}\mathbf{\Sigma}\mathbf{W}^{T}\mathbf{J}^{T},$
(2)

where $\mathbf{W}$ is the projective transformation and $\mathbf{J}$ is the
Jacobian of the affine transformation. The covariance matrix
$\mathbf{\Sigma}$, along with other parameters, is optimised to ensure a
positive semi-definite matrix through decomposition:

$\mathbf{\Sigma}=\mathbf{R}\mathbf{S}\mathbf{S}^{T}\mathbf{R}^{T}.$ (3)

The 2D mean vector $\boldsymbol{\mu}^{\prime}$ results from perspective
projection $\text{Proj}(\boldsymbol{\mu}|\mathbf{E},\mathbf{K})$, with
$\mathbf{E}$ and $\mathbf{K}$ being the camera’s extrinsic and intrinsic
matrices. Consequently, the pixel color $I(\mathbf{p})$ in the image space is
computed by summing the contributions of all Gaussians, and accounting for
their color, opacity, and influence:

$I(p)=\sum_{i=1}^{N}f^{2D}(\textbf{p};\boldsymbol{\mu^{\prime}}_{i},\mathbf{\Sigma^{\prime}}_{i})\
\alpha_{i}\
\mathbf{c}_{i}\prod_{j=1}^{i-1}1-f^{2D}(\textbf{p};\boldsymbol{\mu^{\prime}}_{j},\mathbf{\Sigma^{\prime}}_{j})\
\alpha_{j}$ (4)

Similarly, the depth is computed as:

$D(p)=\sum_{i=1}^{N}f^{2D}(\textbf{p};\boldsymbol{\mu^{\prime}}_{i},\mathbf{\Sigma^{\prime}}_{i})\
\alpha_{i}\
d_{i}\prod_{j=1}^{i-1}1-f^{2D}(\textbf{p};\boldsymbol{\mu^{\prime}}_{j},\mathbf{\Sigma^{\prime}}_{j})\
\alpha_{j}$ (5)

where $d_{i}$ is the depth of the $i\text{-th}$ Gaussian obtained by
projecting $\boldsymbol{\mu}_{i}$ onto the image frame.

The optimisation process starts by initialising the 3D Gaussians using
Structure-from-Motion (SfM) point clouds. It then proceeds to optimise the
learnable parameters by minimising the photometric loss between predicted and
ground truth images. During the entire procedure, Gaussians are adaptively
densified or pruned based on heuristics on their opacity and mean gradient
changes.

##  4 Methodology

This section outlines our approach to reconstructing 3D objects by integrating
visual and tactile data through 3D Gaussian Splatting. Our methodology follows
two primary stages. The first stage involves the generation of initial point
clouds from local depth maps, while the second step consists of optimising and
regularising a Gaussian representation to accurately model the object’s
surface.

###  4.1 Gaussian initialisation and optimisation

The optimisation process starts with the extraction of a pointcloud from
COLMAP, where each point’s position and associated color serve as the initial
mean and color attributes for a first set of Gaussians $G_{c}$. The view-
dependent color is encoded using spherical-harmonics, which are essential in
modeling the specular and glossy highlights. This initial point cloud is
combined with a further set of points $P_{t}$ collected with an optical
tactile sensor, on which a second set of Gaussians $G_{t}$ is initialised on
tactile data. Specifically, each point in $P_{t}$ serves as the mean for one
of the Gaussians in $G_{t}$, while their color and scale attributes are
initialised randomly. A uniform opacity value is assigned across all Gaussians
in both $G_{t}$ and $G_{c}$. Both sets of Gaussians can be cloned, split, and
removed according to the heuristics outlined by Kerbl et al. [8]. The newly
generated Gaussians that result from this process are assigned to the same set
of their originals.

The optimisation of these Gaussians involves minimising the photometric loss
between predicted and ground truth RGB image. This loss is further regularised
by the edge-aware smoothness loss described in 4.2.2. In addition to the
photometric and edge-aware smoothness loss, Gaussians $G_{t}$ are regularised
by the 3D transmittance loss, which we discuss in Sect. 4.2.1.

###  4.2 Regularisation

####  4.2.1 3D transmittance

The transmittance quantifies the degree to which light penetrates through a
medium. Given $N$ Gaussians and a point $\mathbf{p}\in\mathbb{R}^{3}$, where
$\mathbf{p}$ belongs to the touch surface, we define the average 3D
transmittance at $\mathbf{p}$ as:

$\hat{T}(\mathbf{p})=\dfrac{1}{N}\sum_{j=1}^{N}1-f(\textbf{p};\boldsymbol{\mu}_{j},\mathbf{\Sigma}_{j})\
\alpha_{j}$ (6)

where $\boldsymbol{\mu}_{i}\in\mathbb{R}^{3}$ and
$\mathbf{\Sigma_{i}}\in\mathbb{R}^{3}$ are the mean and covariance of the
unprojected 3D Gaussians. To manage the computational load and prioritise the
optimisation of the Gaussians around the touch locations, we limit the number
of Gaussians considered per point to those with the highest spatial influence
on $p$. Moreover, we incorporate a distance-based filtering criterion designed
to exclude Gaussians beyond a certain threshold distance from $\mathbf{p}$.
This is because, in the process of selecting Gaussians based on their spatial
impact, it is possible to include Gaussians that, while influential, are
positioned at considerable distances from the target point. Given that the
Gaussian rasteriser subdivides the frustum into discrete tiles, thus
calculating a point’s transmittance solely from Gaussians within the same
tile, distant Gaussians — even those with potential impact — would not account
for the final transmittance at $\mathbf{p}$. We define our loss
$\mathcal{L}_{T}$ as the average over all the $\mathbf{p}\in P_{t}$ points
collected on the object surfaces:

$\mathcal{L}_{T}=-\dfrac{1}{P}\sum_{i=1}^{P}\hat{T}(\mathbf{p_{i}})$ (7)

Minimising the average transmittance around the touch location effectively
brings the Gaussians closer to the touched surfaces and increases their
opacity, thereby implicitly forcing them to model the real underlying surface.

Figure 2: Proximity-Based Mask’s impact on the gradients computed by the edge-
aware smoothness loss. Left: points collected on the object’s surface
alongside the derived proximity mask. Centre: averaged horizontal and vertical
gradients as determined by the smoothness loss, where lighter shades
correspond to higher gradients. Right: integration of the smoothness loss with
the proximity mask using two distinct approaches: (a) implementation of a
distance-based Gaussian decay within the proximity mask, and (b) masking based
on a discrete threshold from the contact surface.

####  4.2.2 Edge-Aware Smoothness with Proximity-Based Masking

Given the sparsity of touch readings, we leverage an edge-aware smoothness
loss $\mathcal{L}_{S}$ [6] to refine the surface reconstruction further. This
regularisation is based on the intuition that depth transitions within an
image are often aligned with intensity changes in the RGB space. The edge-
aware smoothness function defined as:

$\mathcal{L}_{S}=\frac{1}{N}\sum_{i,j}\left(|\partial_{x}D_{i,j}|e^{-\beta|\partial_{x}I_{i,j}|}+|\partial_{y}D_{i,j}|e^{-\beta|\partial_{y}I_{i,j}|}\right)$
(8)

where $D$ is the rasterised depth map and $I$ is the ground truth image for a
given view. The horizontal and vertical gradients are calculated by applying
convolution operations with 5x5 Sobel kernels [23]. In contrast to the smaller
kernel size commonly adopted by edge detectors, the filters we adopt capture
more contextual information, leading to a smoother gradient estimation and
thus less instability around specular highlights.

While the edge-aware smoothness loss positively contributes to the refinement
of surface reconstruction, its integration with the transmittance loss
presents challenges. Specifically, the smoothness loss interferes with the
transmittance loss around touch locations — areas where we aim to prioritise
transmittance loss to accurately reconstruct contact surfaces. This is because
the smoothness loss operates on a global scale, affecting the continuity and
appearance of the entire scene or large parts of it. In contrast, the
transmittance loss has a more localised effect, focusing on areas around the
known points. When the smoothness loss acts on areas close to the known
geometry, it may counteract the localized enhancements intended by the
transmittance loss - for example, by minimising opacity in order to decrease
gradients in the depth image. To address this, we introduce a Proximity-Based
Masking strategy. This approach involves generating a proximity mask via a
Gaussian decay function, centered around the projected 3D touch locations.
This mask serves to delineate regions of higher confidence based on depth
information and visibility, thereby modulating the edge-aware smoothness loss
according to the proximity to touch locations.

The creation of a proximity-based mask is achieved by calculating the 2D
Euclidean distance transform [3] between every pixel in the rasterised depth
map and its nearest projected 3D point. Then, a Gaussian decay is applied
based on the calculated distance, producing the actual proximity mask.
Finally, the edge-aware smoothness loss, which aims to align the gradients of
the predicted depth map with those of the corresponding RGB image, is
modulated by the inverse of this proximity mask, effectively diminishing the
loss’s influence in areas close to the contact surfaces and preserving edge
details as the distance from touch locations increases. The integration of
transmittance loss with our edge-aware smoothness loss with proximity-based
masking ensures a comprehensive reconstruction quality: while the
transmittance loss facilitates the reconstruction of the contact surfaces, the
smoothness loss refines the overall surface by mitigating inaccuracies in less
explored regions.

##  5 Results

###  5.1 Datasets & Evaluation

We train and evaluate our method on two object-centric datasets containing
glossy and reflective surface: Shiny Blender [24] and Glossy Reflective [11].
The Shiny Blender [24] datasets consists of 6 objects characterised by non-
Lambertian materials. For each object, the dataset provides 100 training
images and 200 test images all accompanied by camera poses. The ball object
was excluded in our evaluation due to the absence of ground truth depth maps,
which makes the assessment of the reconstructed geometry infeasible. Although
ground truth depth maps and normal maps are available, our model relies solely
on RGB images and local depth maps collected on the object surface by
simulating a touch sensor, and we never leverage ground truth depth maps for
supervision.

The Glossy Synthetic [11] datasets consist of 8 objects, each represented by
128 images at a resolution of 800x800. The dataset is divided into 32 images
for training and 96 images for testing. This dataset ranges from objects with
large smooth surfaces displaying specular effects (e.g., Bell, Cat) to those
with complex geometries (e.g., Angel, Luyu).

The evaluation metric employed to asses geometry reconstruction is the Chamfer
Distance (CD)[19], which measures the similarity between predicted and ground
truth point clouds. The Glossy Synthetic dataset provides the ground truth
point cloud for CD calculation. For the Shiny Blender dataset, we create
ground truth point clouds for CD calculation by projecting the ground truth
depth maps in the three-dimensional space. This method ensures that we only
consider areas visible to the camera, thereby accounting for occlusions and
ensuring a fair comparison. The decision against using point clouds sampled
directly from the ground truth meshes is aimed at avoiding discrepancies
caused by occluded regions which are not comparable across different methods.
For the predicted point clouds, we apply the method adopted on the Glossy
Synthetic dataset. In addition, we provide a qualitative comparison between
standard 3DGS and our proposed method on a dataset collected in the real-world
(Sect. 5.4).

###  5.2 Glossy Synthetic

Table 1: Evaluation of the Chamfer Distance (CD) on the Glossy Synthetic
dataset [11] dataset. Our method is the best in recovering the object geometry
from glossy surfaces, followed by NeRO. Glossy Synthetic [11], 100 views
---
Method | Angel | Bell | Cat | Horse | Luyu | Potion | Tbell | Teapot | Avg.
CD($\downarrow$)
Ref-NeRF | 0.0291 | 0.0137 | 0.0201 | 0.0071 | 0.0141 | 0.0131 | 0.0216 | 0.0143 | 0.0241
NeRO | 0.0034 | 0.0032 | 0.0044 | 0.0049 | 0.0054 | 0.0053 | 0.0035 | 0.0037 | 0.0042
3DGS | 0.0002 | 0.0132 | 0.0073 | 0.0005 | 0.0011 | 0.0143 | 0.0133 | 0.0098 | 0.0075
Ours[5 grasps] | 0.0002 | 0.0068 | 0.0027 | 0.0004 | 0.0009 | 0.0038 | 0.0072 | 0.0059 | 0.0034

Full-views: Table 1 benchmarks our method against 3D Gaussian Splatting
(3DGS), NeRO, and Ref-NeRF in the settings where all methods have full access
to all the 100 training views. We chose these methods as they represent the
state-of-the-art in terms of quality reconstruction for challenging objects.
Additionally, our method uses 5 grasps per object, whereas a grasp consists of
“five” fingers touching the object, resulting in a total of 25 tactile
readings. To obtain the results of 3D Gaussian Splatting we trained using the
official 3DGS implementation, while for NeRO and Ref-NeRF we directly report
the available results from [11], although perceptual metrics (PSNR and SSIM)
are not provided. To compute the CD for 3D Gaussian Splatting-related methods,
the predicted point cloud is derived from projecting rasterised depth maps to
3D, using the 96 test cameras. This results in an average of 500K points per
object.

Our analysis demonstrates that our method significantly outperforms the
leading competitor, 3DGS, in 3D reconstruction quality. With just five grasps,
our approach achieves a 50% decrease in the average Chamfer Distance (CD),
resulting in 0.0034 compared to 3DGS’s 0.0075. This performance places our
3DGS-based method on par with, but in some cases ahead of, current NeRF-based
state-of-the-art methods such as NeRO and Ref-NeRF. Morevoer, radiance field
approaches such as NeRO and Ref-NeRF are considerably more computationally
demanding. For reference, while NeRO requires 25 hours of training [11], our
method takes 1 hour on average to reconstruct an object with 5 grasps.

Table 2: Evaluation of the Chamfer Distance (CD) and PSNR on 5-views on the
Glossy Synthetic dataset [11]. In a minimal view setting, our method performs
better than 3DGS and NeRO both in terms of geometry reconstruction and novel-
view synthesis. Glossy Synthetic [11], 5 views
---
Method | Angel | Bell | Cat | Horse | Luyu | Potion | Tbell | Teapot | Avg.
CD($\downarrow$)
3DGS | 0.0004 | 0.0220 | 0.0295 | 0.0007 | 0.0013 | 0.0122 | 0.0145 | 0.0079 | 0.0111
NeRO | 0.0893 | 0.0398 | 0.0230 | 0.1817 | 0.0170 | 0.0043 | 0.0859 | 0.0282 | 0.0586
Ours[5 grasps] | 0.0003 | 0.0082 | 0.0021 | 0.0004 | 0.0008 | 0.0013 | 0.0057 | 0.0022 | 0.0026
PSNR($\uparrow$)
3DGS | 20.98 | 19.60 | 22.77 | 18.99 | 21.24 | 21.33 | 17.97 | 17.80 | 20.08
NeRO | 10.93 | 17.33 | 15.53 | 9.76 | 13.58 | 17.44 | 12.54 | 13.13 | 13.78
Ours[5 grasps] | 21.02 | 19.52 | 23.06 | 21.94 | 20.95 | 21.64 | 18.25 | 17.94 | 20.55

Minimal-views: We explore the performance of the different methods in a
“minimal views” regime where models are provided with only 5 input views. In
this setting we evaluate the top 3 performing methods from the previous
experiment: 3DGS, NeRO and Ours. We used NeRO’s official implementation to
retrain the model. After training, we computed the CD for NeRO models as
described in the original paper and extracted meshes using the Marching Cubes
algorithm [12]. To ensure consistency in evaluation, we sampled around 500,000
points on the generated meshes, aligning with the previously detailed
evaluation procedure for 3DGS. Figure 3 and 4 show that in this setting the
addition of grasps significantly improves the quality of the reconstructed
geometry and multiview rendering. Table 2 reports quantitative results and
shows how NeRO and 3DGS quality of 3D reconstruction drops dramatically when
trained with few views, while our method can retain and, for some models, even
improve. In this setting, our method achieves over 4 times improvement over
the second best model in terms of geometry accuracy (0.0026 Ours vs 0.0111
3DGS), clearly showing the effectiveness of our formulation for high quality
3D reconstruction. The improved reconstruction contributes to enhanced quality
in novel view synthesis, as demonstrated by the PSNR scores.

Figure 3: Surface reconstruction qualitative results from 5 training views on
the Glossy Synthetic dataset

Figure 4: Novel-view synthesis qualitative results from 5 training views on
the Glossy Synthetic dataset Table 3: Evaluation of SSIM, PSNR, and CD results
on the Shiny Blender dataset [24]. Our full method includes both touches and
our proposed smoothness loss. Our method considerably improves the geometry
reconstruction, while achieving comparable levels of image fidelity. Shiny
Blender[24]
---
Method | Car | Coffee | Helmet | Teapot | Toaster | Avg.
CD($\downarrow$)
3DGS | 0.0027 | 0.0018 | 0.0068 | 0.0003 | 0.0069 | 0.0037
3DGS + S | 0.0014 | 0.0022 | 0.0024 | 0.0007 | 0.0077 | 0.0028
3DGS + T[5 grasps] | 0.0028 | 0.0011 | 0.0043 | 0.0002 | 0.0029 | 0.0022
Ours[5 grasps] | 0.0004 | 0.0017 | 0.0005 | 0.0002 | 0.0038 | 0.0013
PSNR($\uparrow$)
3DGS | 27.40 | 32.81 | 27.56 | 45.48 | 21.10 | 30.87
3DGS + S | 27.46 | 32.90 | 27.69 | 45.45 | 21.12 | 30.92
3DGS + T[5 grasps] | 27.36 | 32.88 | 27.60 | 45.51 | 21.16 | 30.90
Ours[5 grasps] | 27.43 | 32.91 | 27.48 | 45.10 | 21.19 | 30.82
SSIM($\uparrow$)
3DGS | 0.930 | 0.972 | 0.950 | 0.997 | 0.896 | 0.949
3DGS + S | 0.932 | 0.972 | 0.952 | 0.997 | 0.898 | 0.950
3DGS + T[5 grasps] | 0.930 | 0.971 | 0.948 | 0.996 | 0.896 | 0.948
Ours[5 grasps] | 0.932 | 0.972 | 0.951 | 0.997 | 0.899 | 0.950

###  5.3 Shiny Blender

We continue the evaluation on the Shiny Blender dataset [24] where we focus on
methods based on 3D Gaussian Splatting. We are interested in exploring the
impact of the different design choices and as such we compare our method
against a baseline defined by vanilla 3DGS, a method that only uses the edge-
aware smoothness regularisation introduced in Section 4.2.2 (3DGS+S), and a
method that only uses 3DGS and tactile information (3DGS + T).

Figure 5: Rasterised depth maps of the Toaster object. Our method results in a
smoother, accurate reconstruction compared to 3DGS and 3DGS with additional
regularisation on the smoothness loss.

Table 3 reports quantitative results comparing the four variants of Gaussian
Splatting-based methods according to both the quality of the 3D reconstruction
(CD) and of novel view synthesis (PSNR and SSIM). Our method does not
sacrifice the quality of the rendering as shown by neutral metrics in terms of
PSNR and SSIM compared to 3DGS, while it significantly improves the geometry
of the reconstructions as measured by Avg. CD (1.3 Ours vs 3.72 3DGS). The gap
is more pronounced on models with relatively large shiny surfaces like toaster
and helmet. Please also refer to Figure 5 for qualitative visualisations of
the difference in reconstructions. The results of 3DGS+S also show how the
smoothness loss is effective to improve the quality of 3D reconstruction, but
it still falls short compared to our full method where constraints given by
the surfaces reconstructed by the grasps and the smoothness regularisation act
in synergy to further boost performance. Similarly, while the addition of
touches in 3DGS+T results in better reconstruction quality over the standalone
3DGS model, the absence of smoothness regularisation limits its ability to
refine the reconstruction beyond the areas of contact.

We study the Chamfer Distance (CD) as a function of the number of touches used
for initialising and optimising the Gaussians in our scene representation
(Figure 6). Our results reveal consistent improvement in reconstruction
accuracy with an increase in physical interactions with the objects. Given the
stochastic nature of our sampling process, we conducted five independent runs
per number of touches to ensure the reliability of our results. This
validation highlights the robustness of our findings, confirming a trend of
improved reconstruction quality up to a point of convergence.

Figure 6: Correlation between the number of touches and its impact on surface
reconstruction for the Glossy Synthetic dataset. For this experiment we kept
the number of views constant to the full training dataset.

Figure 7: Qualitative reconstructions in the real-world. Highlighted areas
show that the introduction of touches leads to smoother and more accurate
surface reconstruction.

###  5.4 Real world experiment

In addition to the experiments in simulated environments, we conduct a
qualitative comparison on real world reconstruction. We employed a robotic
hand (Allegro Hand) equipped with four tactile sensors and an in-hand camera
to acquire multiple views and grasps of a real object, aiming to reconstruct
its geometry using both a standard 3D Gaussian Splatting model (3DGS) and our
novel approach. Specifically, we collected 25 views and 20 tactile readings to
capture the object’s geometry. The qualitative results presented here (Fig. 7)
are part of an evaluation to demonstrate the improvements our method offers in
real-world geometric reconstruction of challenging surfaces. Due to the
unavailability of ground truth geometry, the calculation of the Chamfer
Distance is not feasibile. For clearer visualisation and to emphasise the
quality of object reconstruction, we presented cropped images and 3D Gaussian
representations focused on the object, though training utilised the original,
uncropped images.

Our results highlight regions where our approach produces smoother surfaces or
eliminates holes. These findings align with outcomes from simulated
experiments, indicating that the inclusion of local depth maps obtained from
tactile sensors improves significantly reconstruction quality. For more
information on the data collection procedure employed in this experiment,
please refer to the supplementary material.

##  6 Limitations and Conclusion

This work introduces a novel approach for incorporating touch sensing with
vision. We believe to be the first to have explored the problem of
reconstruction and novel view synthesis of an object that is both seen and
touched. We presented convincing results both qualitatively and quantitatively
on two complex datasets, and we showed that our method outperforms baselines,
especially when a set of minimal views of the object are used.

Despite these promising results, there are some limitations that highlight
areas for future research. The current methodology employs random touch
sampling to collect contact surfaces for scene reconstruction. This strategy
may not always be efficient and could be extended by an adaptive sampling
method, which can select sampling locations to complement visual data. Future
work could focus on investigating how multimodal interaction can further
improve reconstruction of transparent objects within the scene. We believe
incorporating ideas like surface modelling and surface normals can also
improve upon our results.

## Acknowledgements

We would like to thank Alex Zook and Stan Birchfield for their feedback
regarding the scope and contributions of our research, which helped improve
the quality of our work.

## References

  * [1] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5855–5864 (2021)
  * [2] Comi, M., Lin, Y., Church, A., Tonioni, A., Aitchison, L., Lepora, N.F.: Touchsdf: A deepsdf approach for 3d shape reconstruction using vision-based tactile sensing. arXiv preprint arXiv:2311.12602 (2023)
  * [3] Fabbri, R., Costa, L.D.F., Torelli, J.C., Bruno, O.M.: 2d euclidean distance transform algorithms: A comparative survey. ACM Computing Surveys (CSUR) 40(1), 1–44 (2008)
  * [4] Gao, R., Dou, Y., Li, H., Agarwal, T., Bohg, J., Li, Y., Fei-Fei, L., Wu, J.: The objectfolder benchmark: Multisensory learning with neural and real objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17276–17286 (2023)
  * [5] Guo, Y.C., Kang, D., Bao, L., He, Y., Zhang, S.H.: Nerfren: Neural radiance fields with reflections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18409–18418 (2022)
  * [6] Heise, P., Klose, S., Jensen, B., Knoll, A.: Pm-huber: Patchmatch with huber regularization for stereo matching. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2360–2367 (2013)
  * [7] Karis, B., Games, E.: Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice 4(3), 1 (2013)
  * [8] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (July 2023), https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [9] Lambeta, M., Chou, P.W., Tian, S., Yang, B., Maloon, B., Most, V.R., Stroud, D., Santos, R., Byagowi, A., Kammerer, G., et al.: Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters 5(3), 3838–3845 (2020)
  * [10] Lepora, N.F.: Soft biomimetic optical tactile sensing with the tactip: A review. IEEE Sensors Journal 21(19), 21131–21143 (2021)
  * [11] Liu, Y., Wang, P., Lin, C., Long, X., Wang, J., Liu, L., Komura, T., Wang, W.: Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. In: SIGGRAPH (2023)
  * [12] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Seminal graphics: pioneering efforts that shaped the field, pp. 347–353 (1998)
  * [13] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [14] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * [15] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5865–5874 (2021)
  * [16] Qi, H., Yi, B., Suresh, S., Lambeta, M., Ma, Y., Calandra, R., Malik, J.: General in-hand object rotation with vision and touch. In: Conference on Robot Learning. pp. 2549–2564. PMLR (2023)
  * [17] Smith, E., Calandra, R., Romero, A., Gkioxari, G., Meger, D., Malik, J., Drozdzal, M.: 3d shape reconstruction from vision and touch. Advances in Neural Information Processing Systems 33, 14193–14206 (2020)
  * [18] Smith, E., Meger, D., Pineda, L., Calandra, R., Malik, J., Romero Soriano, A., Drozdzal, M.: Active 3d shape reconstruction from vision and touch. Advances in Neural Information Processing Systems 34, 16064–16078 (2021)
  * [19] Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T., Tenenbaum, J.B., Freeman, W.T.: Pix3d: Dataset and methods for single-image 3d shape modeling. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2974–2983 (2018)
  * [20] Suresh, S., Si, Z., Mangelson, J., Yuan, W., Kaess, M.: ShapeMap 3-D: Efficient shape mapping through dense touch and vision. In: Proc. IEEE Intl. Conf. on Robotics and Automation, ICRA. Philadelphia, PA, USA (May 2022)
  * [21] Suresh, S., Qi, H., Wu, T., Fan, T., Pineda, L., Lambeta, M., Malik, J., Kalakrishnan, M., Calandra, R., Kaess, M., et al.: Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. arXiv preprint arXiv:2312.13469 (2023)
  * [22] Tao, M.W., Su, J.C., Wang, T.C., Malik, J., Ramamoorthi, R.: Depth estimation and specular removal for glossy surfaces using point and line consistency with light-field cameras. IEEE transactions on pattern analysis and machine intelligence 38(6), 1155–1169 (2015)
  * [23] Vairalkar, M.K., Nimbhorkar, S.: Edge detection of images using sobel operator. International Journal of Emerging Technology and Advanced Engineering 2(1), 291–293 (2012)
  * [24] Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-nerf: Structured view-dependent appearance for neural radiance fields. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5481–5490. IEEE (2022)
  * [25] Wang, S., Wu, J., Sun, X., Yuan, W., Freeman, W.T., Tenenbaum, J.B., Adelson, E.H.: 3d shape perception from monocular vision, touch, and shape priors. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1606–1613. IEEE. https://doi.org/10.1109/IROS.2018.8593430, https://ieeexplore.ieee.org/document/8593430/
  * [26] Yuan, W., Dong, S., Adelson, E.H.: Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors 17(12), 2762 (2017)