# GDTM: An Indoor Geospatial Tracking Dataset with Distributed Multimodal
Sensors

Ho Lyun Jeong<EMAIL_ADDRESS>University of California, Los
AngelesLos AngelesCAUSA90095 ,  Ziqi Wang<EMAIL_ADDRESS>University of
California, Los AngelesLos AngelesCAUSA90095 ,  Colin Samplawski
<EMAIL_ADDRESS>University of Massachusetts AmherstAmherstMAUSA01003
,  Jason Wu<EMAIL_ADDRESS>University of California, Los AngelesLos
AngelesCAUSA90095 ,  Shiwei Fang<EMAIL_ADDRESS>University of
Massachusetts AmherstAmherstMAUSA01003 ,  Lance M. Kaplan
<EMAIL_ADDRESS>US DEVCOM Army Research LaboratoryUSA ,  Deepak
Ganesan<EMAIL_ADDRESS>University of Massachusetts
AmherstAmherstMAUSA01003 ,  Benjamin Marlin<EMAIL_ADDRESS>University of
Massachusetts AmherstAmherstMAUSA01003 and  Mani Srivastava<EMAIL_ADDRESS>University of California, Los Angelesand AmazonUSA90095

(2024)

###### Abstract.

Constantly locating moving objects, i.e., geospatial tracking, is essential
for autonomous building infrastructure. Accurate and robust geospatial
tracking often leverages multimodal sensor fusion algorithms, which require
large datasets with time-aligned, synchronized data from various sensor types.
However, such datasets are not readily available. Hence, we propose GDTM, a
nine-hour dataset for multimodal object tracking with distributed multimodal
sensors and reconfigurable sensor node placements. Our dataset enables the
exploration of several research problems, such as optimizing architectures for
processing multimodal data, and investigating models’ robustness to adverse
sensing conditions and sensor placement variances. A GitHub repository
containing the code, sample data, and checkpoints of this work is available at
https://github.com/nesl/GDTM.

††copyright: none††journalyear: 2024††copyright: none††conference: Paper Under
Submission; Feb 23–27, 2024; Los Angeles, CA, USA††booktitle: Under
Submission††price: 15.00††doi: 00.0000/0000000.0000000††isbn:
000-0-0000-0000-0/00/00

##  1\. Introduction

Geospatial tracking has been an essential and exciting problem for autonomous
building infrastructure. The task of geospatial tracking is to constantly
detect and locate objects moving across a scene, providing critical
localization and tracking services for mobile health, building management,
surveillance, etc (Zafari et al., 2019). For example, smart building
infrastructure can track the whereabouts of delivery robots to assist them
with complex tasks such as elevator control that would otherwise be unfeasible
for the robot to perform. First responders can also benefit from better
situational awareness provided by advanced target recognition and localization
services in unfamiliar indoor scenes.

Multimodal sensor networks are particularly useful in geospatial tracking
because of the GPS-denied nature of indoor environments, the complexity of
layouts, and limited cooperation from the sensing targets (i.e., human beings
or robots). Multimodal sensor fusion methods aggregate information from
heterogeneous sensors to construct perceptions of a scene. Research has shown
that multimodal sensor fusion helps improve the accuracy and reliability of
sensing applications in the following ways (Duan et al., 2022). _First_ ,
multimodal sensor fusion makes systems more robust to adversarial sensing
conditions, such as poor lighting conditions (Cui et al., 2023) or bad weather
(Bijelic et al., 2020). _Second_ , multimodal sensor fusion methods have
demonstrated increased resilience to unstable sources of data, i.e., missing
sensor readings (John and Kawanishi, 2022; Piechocki et al., 2023). _Third_ ,
the complementary sensing capability of multiple sensors also better captures
the complicated nature of a scene, providing knowledge not available to
unimodal sensing systems (Ulrich et al., 2018). Despite these benefits,
multimodal sensor fusion often relies on a data-driven approach, whose success
depends on time-synchronized large datasets collected with a network of
heterogeneous sensors.

The autonomous driving community has embraced multimodal sensor fusion
approaches, producing an abundance of algorithms (Huang et al., 2022) and
datasets (Geiger et al., 2013; Caesar et al., 2020; Mao et al., 2021; Sun et
al., 2020), where cameras and LiDARs are the primary choice of sensors.
Serving autonomous driving applications, these multimodal datasets are often
collected in large outdoor spaces with mobile robots and vehicles. However,
similar datasets for indoor infrastructures have been largely overlooked
despite being an equally critical asset. Therefore, we focus on providing a
nine-hour dataset collected with a network of multimodal sensors for the
geospatial tracking problem in order to benefit the design of autonomous
systems operating in indoor environments.

The sensor set in GDTM covers most commonly seen indoor modalities, including
stereo vision camera, LiDAR camera, mmWave radar, and microphone arrays. These
sensors possess complementary sensing capabilities and sufficient
redundancies. We created a sensor network consisting of three sensor nodes,
each equipped with the aforementioned sensor set. We also designed software
tools and scripts that synchronize the sensor network using Network Time
Protocol (NTP), control the start and the end of data collection sessions, as
well as compress and stream the data to storage devices. During the data
collection, the sensor nodes are placed around an indoor race track with
remote-controlled cars running on top. The sensor nodes’ field-of-views are
partially overlapped. GDTM’s multi-hour data collection experiments cover the
cases of a single tracking subject, multiple tracking subjects, and poor
illumination. We leverage an OptiTrack motion capture system to provide mm-
level ground truth for the location and orientation of the sensing target and
the sensor nodes. Another highlight of GDTM is that the dataset is collected
with a variety of different sensor placements. In most existing datasets (Han
et al., 2023), the sensors are placed at fixed positions in each scene. Models
trained on such data overfit to a particular viewpoint and generalize poorly
during deployments. We showcase such failures later in Fig 7. GDTM covers more
than 20 different sets of sensor viewpoints to provide sufficient training and
testing cases to enable the development of algorithms robust to sensor
placement variations.

Our GDTM dataset enables the investigation of a number of research questions.
For example, one may use GDTM to explore architectures for high-performance
multimodal sensor fusion trackers that are robust to poor illumination
conditions or missing sensor data. GDTM can also benefit research aiming to
create easily deployable sensing models resilient to the placement locations
and orientations of its sensor nodes. We also expect GDTM to benefit research
of multi-object tracking, resource scheduling of sensing applications, complex
event detection, and so on. In the next few sections of the paper, we first
review some current literature in Sec. 2, and then cover some details on the
sensing nodes, the experiments, and statistics of GDTM in Sec. 3. Next, we use
two baseline experiments to explore a subset of the research questions raised
by GDTM in Sec. 4.1 and Sec. 4.2, in order to demonstrate the quality and the
usefulness of GDTM. Finally, we discuss the limitations and future direction
to wrap the paper in Sec. 5.

##  2\. Related Work

Egocentric multimodal geospatial tracking datasets. A number of multimodal
datasets have been proposed focusing on autonomous driving applications. RGB-D
SLAM Dataset (Sturm et al., 2012) places RGB-D cameras on a robot moving in
the scene and uses an OptiTrack motion capture system to provide ground truth.
Indoor Location Competition 2.0 Dataset (Shu et al., 2021) includes IMU, Wi-
Fi, and Bluetooth data collected from a user-held cellphone and uses pre-
defined waypoints as the ground truth for indoor location sensing tasks.
StreetAware (Piadyk et al., 2023) combines video, audio, and LiDAR to
construct a multimodal urban scene dataset. Other datasets like KITTI (Geiger
et al., 2013), nuScenes (Caesar et al., 2020), ONCE (Mao et al., 2021), and
Waymo Open (Sun et al., 2020) also fall into this category, focusing on large
outdoor spaces for developing self-driving algorithms. These datasets
typically operate with mobile vehicles and sense the environment in an ego-
centric manner. Our GDTM dataset, on the other hand, employs a network of
indoor multimodal sensors to sense non-cooperative targets to investigate the
research opportunities provided by smart building infrastructures.

Datasets collected with sensor networks. Meanwhile, there exists a number of
datasets that focus on sensor networks where perception happens across
different sensors and viewpoints. The VIRAT video dataset (Oh et al., 2011)
employs multiple video surveillance cameras to provide benchmarks for human
activity recognition. A follow-up dataset work, MEVA (Corona et al., 2021),
extends the dataset to more camera views, including some infrared cameras and
drone-mounted cameras, and uses GPS to provide the ground truth of different
person’s location. MMPTRACK (Han et al., 2023) aims to create a large-scale
tracking dataset using a multi-camera tracking system for multiple human
subjects. Radio-frequency wireless signals are also commonly used in indoor
environments since they are less susceptible to light conditions and
occlusions. OperaNet (Bocus et al., 2022) combines Kinect visual sensors with
multiple radio-frequency wireless sensing modalities (e.g., Wi-Fi channel
state information) to construct a multimodal activity recognition dataset.
Compared with existing dataset work, GDTM differs both in the richness of the
sensors and the diversity of viewpoints. Sensor nodes of GDTM cover vision,
depth, radar, and audio, which are the most commonly seen indoor sensing
modalities. Moreover, GDTM provides multiple sets of sensor placements,
encouraging the development of fusion models robust to the domain shift caused
by changed sensor perspectives.

##  3\. Dataset Collection

###  3.1. Sensing Platform

We designed and assembled a multimodal sensing platform consisting of three
sensor nodes. We showcase one of the sensor nodes in Figure 1. Each node is
equipped with a complementary sensor set, including a RealSense L515 LiDAR
camera, a ZED 2i stereo vision camera, an IWR1443 mmWave Radar, and a
ReSpeaker microphone array. To cater to the possibility of outdoor operation,
one of the sensing nodes contains a commercial LiDAR, and all three nodes have
a GPS-RTK board. An NVIDIA Jetson Xavier NX serves as the main controller of
each sensing node, responsible for controlling sensor operations, maintaining
node health information, temporarily storing the data, and listening to the
instructions from a command node.

Figure 1. One of our multimodal sensor data collection nodes.

.

Figure 2. Sample data collected by one of the 3 sensing nodes. Figure 3. Data
collection session setup: (a) racetrack dimensions (b) experiment site (c) an
exemplar circular trajectory (d) an exemplar random trajectory.

A small NUC Mini computer serves as the command node. The command node and
three sensor nodes are placed under the same Wi-Fi network and time-
synchronized using the NTP protocol. During the experiment, the command node
sends instructions to the sensor nodes to control the start and the end of
sensor data collection. The command node also acquires each sensing node’s
health status during the experiments to ensure data continuity. At the end of
each experiment session, the acquired data will be compressed and streamed to
the command node.

Sample data from one of the sensor nodes is shown in Figure 2. In GDTM, we
provide both left and right images, as well as the calculated depth from the
stereo vision camera at 15 Hz with $1920\times 1080$ resolution. From LiDAR
cameras, we obtain RGB frames captured at 15 Hz with $1920\times 1080$ and
corresponding depth frames streamed at 15 Hz with $640\times 480$ resolution.
From the mmWave radars, range azimuth heatmap, range doppler heatmap, range
profile, noise profile, and point cloud are recorded at 4 Hz. The six audio
channels of ReSpeaker (raw data from the four microphones, merged playback,
and processed audio for automatic speech recognition, sampled at 16 kHz) are
encoded in FLAC format. While not being a main focus of this paper, Ouster
LiDAR scans are also included in GDTM. The ground truth contains mm-level pose
information (position and orientation) for each node and each car on the track
collected by the OptiTrack motion capture system, which tracks the reflective
balls mounted on the tracked subjects and the sensor nodes using infrared
light.

###  3.2. Experiment Setup

Experiment Site. The experiments are conducted in an indoor race track that
spans an area of $3\textrm{ m}\times 3.6\textrm{ m}$. The race track is
delineated with white tape on top of dark gray foam mats. The detailed
measurements of the race track are depicted in Figure 3(a). Six OptiTrack
cameras are set up around the room to collect ground truth. Sensor nodes are
placed around the race track during each experiment. Figure 3(b) shows the
experiment site.

Experiment Settings. We pick remote-controlled cars (RC cars) as targets for
geospatial tracking in GDTM since RC cars can easily create arbitrary
trajectories we desire. For each experiment, we drive a red RC car, a green RC
car, or both around the race track. The trajectory of each car mostly follows
the lanes on the race track, with behaviors like changing lanes, turning
around, or backing up. To avoid creating monotonous circular trajectories, we
also drove RC cars in random or erratic patterns. This random behavior allows
for verification that the tracking system is not simply predicting repetitive
motion. Figure 3(c) shows a circular trajectory, and Figure 3(d) demonstrates
a random trajectory. Each trajectory is plotted for a duration of 200 seconds.
We quantify the motion patterns by dividing the trajectories into chunks of 10
seconds and classifying them based on whether the chunk dominantly lies within
the racetrack lanes. We found that 80.8% of our trajectories are circular
patterns, and the remaining 19.2% are random patterns. Note that even in the
circular patterns, the trajectories are not simply repeating themselves but
are actually “donut-shaped”, covering a width of 60-80 cm, allowing for
greater diversity. Thus, combining both random and circular patterns, we have
good coverage of the testing site.

GDTM also covers the case of multi-object tracking by including over one hour
of data with both cars on the race track, featuring complex dynamics such as
overtakes and collisions. Meanwhile, GDTM contains around three hours of data
under poor lighting conditions. This setting challenges the robustness of the
tracking system as the utility of certain modalities, such as RGB and stereo
vision, is vastly reduced. Furthermore, GDTM features numerous positional
configurations (positions and orientations) of the three collection nodes,
including over 15 different views for experiments with each car in good
lighting. This prevents the tracking system from overfitting to a fixed
viewpoint, encouraging it to learn a representation more generalizable to
different deployments. A detailed breakdown of the dataset statistics can be
found in Table 1.

| Good Lighting Condition | Poor Lighting Condition
---|---|---
Red Car | Green Car | Both | Red Car | Green Car
# of Viewpoints | 18 | 15 | 6 | 7 | 2
Minutes | 200 | 165 | 95 | 145 | 35
Raw Data Size (GB) | 278 | 235 | 134 | 202 | 45

Table 1. Dataset Statistics.

##  4\. Benchmarking Experiments

Fusion Model (Sensors) | Good Light Condition | Poor Light Condition
---|---|---
View 1 | View 2 | View 3 | View 1 | View 2 | View 3
AD | NLL | AD | NLL | AD | NLL | AD | NLL | AD | NLL | AD | NLL
| Early
---
(Camera)
25.85 | 7.81 | 9.11 | 5.79 | 14.69 | 7.03 | 117.14 | 15.49 | 118.91 | 16.4 | 115.94 | 15
| Early
---
(All)
16.26 | 7.29 | 8.85 | 6.08 | 6.25 | 5.26 | 19.9 | 8.14 | 19.89 | 8.13 | 16.21 | 7.26
| Late
---
(Camera)
23.79 | 7.63 | 5.92 | 5.36 | 10.16 | 6.25 | 115.2 | 163.27 | 45.94 | 10.83 | 32.73 | 8.3
| Late
---
(All)
31.93 | 19.86 | 12.14 | 9.7 | 9.22 | 6.25 | 47.18 | 34.95 | 16.08 | 8.33 | 24.97 | 10.28

Table 2. Comparison of four variants of multimodal sensor fusion architectures
in both good and poor illuminations.

We designed two benchmarking experiments to showcase two of many research
questions arising from the GDTM dataset. Specifically, we focused on improving
geospatial tracking accuracy by investigating model architectures for sensor
fusion, and increasing tracking robustness by creating models robust to the
domain shift caused by sensor perspective changes.

###  4.1. Baseline 1: Exploring Multimodal Sensor Fusion Architectures

In this section, we compare an early fusion and a late fusion approach by
evaluating their tracking performance in different lighting conditions.

####  4.1.1. Model Architectures:

Figure 4. Multimodal sensor fusion architecture.

Late Fusion Model Architecture. Video-based object tracking has been a well-
investigated area where mature architectures such as mmtracking (Contributors,
2020) have been proposed. In this first experiment, we aim to extend the
video-based tracking architectures to multimodal sensor data. Specifically, we
adopt the heteroskedastic geospatial detector (HGD) structure proposed in
(Samplawski et al., 2023), as shown in Figure 4(a). HGD consists of a backbone
that extracts features from raw sensor data, an adapter (a fully-connected
layer) that maps the extracted features to a fix-sized ($1\times 256$) shared
embedding, and an output head that converts the latent encoding into the
object’s mean location and covariance using another fully-connected layer. HGD
then uses a Kalman filter to reconcile the predictions from different branches
and output a final distribution of the objects’ predicted locations. In HGD,
the backbone is shared within each modality, and the output head is shared by
all nodes and all modalities. Adapters, however, are unique to the combination
of nodes and modalities. With 3 nodes and 4 modalities, we have four
backbones, 12 adapters, and one output head.

We extend the HGD architecture by designing backbones for all modalities. For
vision-based modalities such as RGB and depth, we use ResNet-50 pre-trained on
ImageNet as the backbone. For radar, we use an MLP with two fully connected
layers interleaved with GELU non-linearity layers. For audio, we stack a 2D
convolution, a BatchNorm layer, and finally, a ConvNext layer (Liu et al.,
2022) as the backbone. The benefit of using HGD is that it accounts for
prediction uncertainties by outputting a multivariate normal distribution
rather than a single location prediction.

Early Fusion Model Architecture. In the late fusion model above, each modality
generates its own predictions that are later reconciled by a Kalman filter. To
facilitate early information exchange across sensors, we propose an early
fusion approach where the fusion happens at the intermediate feature level
(see Figure 4(b)) The inputs are first processed by the same backbones used in
the late fusion model.

Then, the adapter (with positional encoding) flattens the backbone features
into $D_{k}\times 256$ feature vectors, where $D_{k}$ is related to the
original size of the $k_{th}$ input branch. Later in the model, we will use an
attention mechanism that ignores the order of the features, which results in
the loss of spatial information. For example, the camera backbone extracts
features from the input image, where each feature represents a pixel region in
the original image. These features are ordered sequentially to preserve the
spatial information. The attention mechanism, however, is agnostic to feature
ordering and cannot retain spatial information. To address this issue, we
apply sigmoid positional encodings proposed by (Carion et al., 2020). We
calculate an embedding vector $\mathbf{E}$ of size 256 representing the
spatial order of each feature vector $\mathbf{F}$, and then append the
embedding $\mathbf{E}$ to $\mathbf{F}$, increasing the feature dimension from
256 to 512. The appending operation is inspired by (Meng et al., 2021) to
reduce the likelihood that the model will learn to ignore the positional
encodings.

The outputs of these adapters are concatenated into a latent feature vector
$\mathbf{V}\in\mathrm{R}^{D\times 512}$, where $D=\sum_{k}d_{k}$. This is
where the fusion begins. The latent feature vector is then fed to a six-layer
cross-attention module. Each layer is an 8-head cross-attention accepting
three sets of vectors (query, key, and value) as inputs. Similar to (Carion et
al., 2020), we instantiate a single $512$-dim query embedding $\mathbf{Q}$ for
the first cross-attention layer. This embedding is a set of learnable
parameters that encode the latent prediction of the object’s location. We
update the embedding by applying repeated layers of cross-attention. Here, the
query and key of the attention come from $\mathbf{Q}$, and the value is
calculated from the latent feature vector $\mathbf{V}$ introduced above. The
output of each cross-attention layer is still $1\times 512$, which becomes the
$\mathbf{Q}$ for the next attention layer. The final embedding is then decoded
into a Gaussian distribution using the output head of the model, followed by a
Kalman filter similar to the one used in the late fusion architecture. To
support multi-object tracking in the future, we can instantiate more
$\mathbf{Q}$ vectors, with each of them being responsible for one object.

In both models, the loss is defined by the negative log-likelihood (NLL) of
the ground truth position of the object under the distribution specified by
the output head. After training, some additional data is required to perform a
post-hoc model recalibration as described in (Samplawski et al., 2023) to
better capture model prediction uncertainties. More specifically, we apply an
affine transformation $\Sigma^{\prime}=a\Sigma+bI$ to the output covariance
matrix $\Sigma$ with parameters $a$ and $b$ that minimizes the calibration
data’s NLL. Our models process the sensor data and make predictions frame-by-
frame. The new predictions are reconciled with each other and with the
historical predictions using a Kalman tracker.

####  4.1.2. Evaluation metrics:

Average Distance (AD). We use the average distance (AD) metric to quantify the
discrepancy between the model prediction and the ground truth trajectory of
the object. The average distance is computed by taking the mean of the
distance between the predicted and the ground truth location of the object at
each frame. A lower AD value indicates a higher accuracy and a closer
alignment between the predicted and the actual trajectories.

Negative Log-Likelihood (NLL). We use the negative log-likelihood (NLL) metric
to quantify the ability of the models to probabilistically predict the
trajectory of the object with a multivariate Gaussian distribution. A lower
NLL value indicates a higher similarity between the model’s predicted
distribution and the actual ground truth distribution.

####  4.1.3. Experiments and Results

Both the early and late fusion models are trained, calibrated, and tested on a
total of six different sets of data selected from GDTM, three sets under good
lighting conditions and three under poor lighting conditions. Each set
consists of approximately 10 minutes (9000 frames) for training, 1 minute (900
frames) for calibration, and 3 minutes (2700 frames) for testing. The training
of each set takes about six hours on a single RTX A6000 GPU. Note that, in
this section, the training and testing are conducted using data from different
sessions but from the same viewpoints (sensor placements).

Figure 5. Average tracking error of the four model variants under different
lighting conditions. Figure 6. Cumulative distribution function (CDF) plot of
different models’ tracking error.

We conduct tests on the two models, each with two configurations: RGB camera
only and all modalities combined. We investigate the effect of environmental
challenges on single- and multi-modality models by subjecting them to poor
lighting conditions (see results in Table 2). Generally, the all-modality
version of the early fusion model outperforms other variants. We then average
the results across experiments to generate Figure 5. We can see that in good
lighting conditions, both the camera-only and all-modality models perform well
since the camera feed contains sufficient information about the car’s
location. However, our camera-only models struggle under poor lighting
conditions, with average tracking errors (AD) exceeding 100 cm for early
fusion and AD=64.62 cm for late fusion. When additional modalities are
incorporated, the performance improves significantly, reducing the AD to
approximately 18 cm (using early fusion). Additionally, we show the cumulative
distribution function (CDF) plots of different models’ positioning errors in
Figure 6. The CDF plots clearly show that multimodal fusion methods outperform
their camera-only counterparts, especially early fusion architecture in poor
illumination conditions. These findings emphasize the role of multimodality in
improving model robustness under environmental constraints, and highlight the
usefulness of the GDTM dataset for evaluating multimodal models.

Enabled Embeddings | Good Light Condition | Poor Light Condition
---|---|---
AD↓ | NLL↓ | AD↓ | NLL↓
Camera Only | 16.15 | 7.79 | 116.27 | 37.04
Depth Only | 118.21 | 40.69 | 117.06 | 37.48
Doppler Only | 25.85 | 22.84 | 17.06 | 7.03
Audio Only | 89.8 | 30.07 | 100.76 | 25.19
All Modalities | 6.25 | 5.26 | 16.21 | 7.26

Table 3. Early fusion model performance when the embeddings of all-but-one
modality are nullified.

Furthermore, we investigate each modality’s contribution toward the final
prediction in the early fusion model by nullifying other modalities’
embedding. In Figure 4(b), the latent embedding is constructed by
concatenating the adapter outputs of each modality. We first train and freeze
the early fusion model, set the non-target modalities’ embedding to zero, and
then evaluate the model performance. In this case, only the target modality
contributes to the final predictions. The result (see Table 3) shows that the
early fusion model “pays more attention” to the camera and Doppler information
under good illuminations and focuses on mmWave Doppler information when
lighting is poor. It also reveals that early fusion models with attention tend
to rely on a small subset of modalities that provide sufficient information,
necessitating techniques such as regularization to ensure model redundancy.

###  4.2. Baseline 2: Developing Models Robust to Sensor Placements

| View1 | View2 | View3 | View4 | View5 | View6 | Average
---|---|---|---|---|---|---|---
| Model
---
(Sensors)
AD | NLL | AD | NLL | AD | NLL | AD | NLL | AD | NLL | AD | NLL | AD | NLL
| Late3D
---
(Camera)
124.06 | 94.73 | 114.98 | 79.6 | 136.16 | 115.65 | 133.28 | 78 | 117.34 | 61.96 | 119.95 | 87.62 | 124.30 | 86.26
| Late3D
---
(Camera,
Depth)
42.53 | 15.72 | 54.64 | 17.92 | 85.15 | 50.84 | 53.72 | 31.89 | 74.18 | 48.81 | 54.83 | 34.85 | 60.84 | 33.34
| Late3D
---
(All)
45.00 | 31.32 | 33.71 | 17.64 | 61.89 | 41.11 | 34.07 | 19.27 | 67.45 | 82.97 | 42.77 | 36.05 | 47.48 | 38.06

Table 4. Performances of models trained and tested on a variety of sensor
placements.

In this section, we showcase GDTM’s potential to be used for solving the
domain adaptation problem caused by changed sensor perspectives during
deployment. Domain adaptation refers to leveraging labeled data in one or more
related source domains to learn a model for unseen or unlabeled data in a
target domain (Csurka, 2017). In our application scenarios, a neural network
can be developed and trained with a number of camera/sensor perspectives.
However, during deployment, the cameras/sensors’ locations and poses are
different from that of the training data, and the network needs to be tuned or
adapted before it can work properly. In this case, the source and target
domain consist of sensor data coming from different perspectives.

We use the models developed in Section 4.1 to show the domain shift problem
intuitively. In Figure 7(a), the green dots stand for ground-truth target
locations, black circles represent the early fusion model’s predictions, and
the blue ovals represent the final Kalman tracker’s predictions. The center of
the ovals is the predicted target location, and the size of the ovals stands
for prediction covariance (uncertainties). We find that the model accurately
tracks the targets with data from the viewpoint on which the model was trained
(i.e., source domain data). However, in Figure 7(b), we find that the model
fails catastrophically when trying to track the object using target domain
data (i.e., data from a sensor viewpoint that the model has never seen
before).

Figure 7. Model performance on (a) trained viewpoint (b) unseen viewpoint.

In Section 4.1, the models were trained on a set of _global_ coordinates,
where the models learn the correspondences between sensor readings and
objects’ locations. However, these correspondences break up when the sensors
move to a new perspective. To provide a preliminary baseline addressing this
challenge, we instead train nodes to localize the object within _its own local
coordinate system_ and transform the predictions to global coordinates using
sensor node pose information.

Late Fusion Model Modifications: The late fusion model is more suitable for
this task as we need to make predictions for each node and transform the
result separately. A few modifications need to be made to the model for this
new task. Firstly, we need to enable 3D predictions for the model. In Sec.
4.1, the cars move on the ground plane ($xz$-plane, $y=0$), so the coordinates
are two-dimensional. These coordinates become 3D when transformed into the
nodes’ local coordinates because sensor nodes are not placed parallel to the
ground plane. Thus, we must modify the _Output Head_ in Fig. 4 by increasing
the output dimension from 5 (2 for mean and 3 for covariance matrix) to 9 (3
for mean and 6 for covariance matrix).

We also modify the _adapter_ from being independent for each node to being
_shared_ across nodes. These adapters previously generated embeddings that
allowed each node to predict the targets’ global coordinates, so they were
independent as each node learned a different mapping. However, each node is
now predicting in its own local coordinate, so the adapter can be shared
because all the nodes are built from the same blueprint and share the same
layout.

Coordinate Transformations: GDTM utilizes the OptiTrack motion capture system
to record the object’s global coordinates while also recording the
transformation matrix between each node’s local coordinate system to the
global. To compute the loss for training, we must provide ground truth local
coordinates by transforming the global ground truth data into each node’s
local coordinate system. Furthermore, at inference time, we also must
transform the predicted local coordinates into global coordinates for the
Kalman Filter to create one unifying prediction in the global coordinate
system. The transformation between global distributions
$\mathbf{\mu_{g}}\in\mathbb{R}^{3\times
1},\mathbf{\Sigma_{g}}\in\mathbb{R}^{3\times 3}$ and local distributions
$\mathbf{\mu_{l}},\mathbf{\Sigma_{l}}$ is implemented using a translation
(node position) $\mathbf{T}$ and a rotation $\mathbf{R}$ (node orientation):

(1)
$\mathbf{\mu_{g}}=\mathbf{R}(\mathbf{\mu_{l}}-\mathbf{T}),\mathbf{\Sigma_{g}}=\mathbf{R}\mathbf{\Sigma_{l}}\mathbf{R}^{T}.$

Experiments and Results The model is trained on 83 minutes of data with 13
different viewpoints, calibrated with 5 minutes of data with three different
viewpoints, and tested on 6 minutes of data with six different views. The
model is trained and tested with three configurations: RGB only, RGB + depth,
and all-modalities. The training takes roughly 48-72 hours on a single RTX
4090 GPU, where including more modalities leads to a longer training time. We
aim to provide a baseline model for object tracking that is generalizable to
different sensor placements without retraining. The results are shown in Table
4. We find that our model fails to produce a meaningful result using RGB
images only, as the AD exceeds 120 cm within an area of $3\mathrm{m}\times
3.6\mathrm{m}$. However, when the depth modality is incorporated, we observe a
significant improvement. The AD drops to approximately 60 cm, improving by
50%. This demonstrates the potential of leveraging multimodality to overcome
the limitations of relying solely on RGB images. The addition of mmWave and
acoustic modalities further pushed the mean tracking error down to 47.48 cm.
We can see that the inclusion of more sensing modalities generally makes the
model more robust to the domain shift caused by sensor perspective changes.
Despite the performance improvement brought by multimodal sensor fusion, an AD
around 45 cm falls short of the desired accuracy. By releasing GDTM to the
research community, we hope the dataset will contribute to the development of
multimodal sensor fusion models more robust to sensor perspective shifts.

##  5\. Limitations, Future Work, and Conclusions

In this paper, we introduced GDTM, a multimodal, multiview dataset for indoor
geospatial tracking using networked sensors. We showed baseline experiments
where we could use GDTM to develop robust algorithms for adversarial sensing
conditions such as poor illumination. We also demonstrated the potential to
construct models agnostic to sensor placement variances. Here, we discuss the
limitations and possible future extensions of GDTM.

Extension to other indoor layouts and scenarios. Limited by the scale of the
experiment site and the lack of IRB approval for collecting data in public
space, GDTM is limited to a racetrack setting in a single room and does not
cover complicated indoor layouts with obstacles and occlusions. We hope future
studies will diversify GDTM by extending it to various realistic indoor
scenarios.

Better multimodal sensor fusion architectures. While we show that the GDTM
dataset enables the exploration of several research problems, our baseline
results are still preliminary. In the single-view baseline, our early fusion
model leans heavily towards only 1-2 modalities, limiting its robustness to
missing sensor feeds. Our solution to the multiview baseline is also less
satisfying in terms of absolute tracking error. Furthermore, our baselines do
not cover the case of multiple tracking targets. These questions can be
addressed in the future with more efforts towards better neural model designs.

Incidents and accidents during data collections. During the collection of
GDTM, we have recorded many incidents such as car collisions, vehicles running
off track, and humans rescuing stuck cars. We included a field called
$valid\\_range$ in the metadata where we manually inspected the data and
marked the data void before and after each session. If any incident causes the
car to stay still for a long period, the corresponding part will be clipped
out by applying the $valid\\_range$ mask. Meanwhile, we see these incidents as
an opportunity to investigate other problems. For example, a learning-enabled
agent may perform Q&A about the experiment data, such as asking how many times
the car has run out of lane or when a car collision occurred. Of course, doing
so requires labeling incidents separately.

In conclusion, we hope GDTM opens up the opportunity to investigate widespread
robustness issues in multimodal sensor fusion and to create lightweight,
deployable models for indoor geospatial tracking, which can benefit the design
and implementation of autonomous systems in smart building infrastructures.

###### Acknowledgements.

The research reported in this paper was sponsored in part by: the IoBT REIGN
Collaborative Research Alliance funded by the Army Research Laboratory (ARL)
under Cooperative Agreement W911NF-17-2-0196; the National Science Foundation
(NSF) under award #1822935; and the Air Force Office of Scientific Research
(AFOSR) under award FA9550-22-1-0193. The views and conclusions contained in
this document are those of the authors and should not be interpreted as
representing the official policies, either expressed or implied, of the
funding agencies.

## References

  * (1)
  * Bijelic et al. (2020) Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide. 2020.  Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In _Proceedings of the IEEE/CVF CVPR_. 11682–11692.
  * Bocus et al. (2022) Mohammud J Bocus, Wenda Li, Shelly Vishwakarma, Roget Kou, Chong Tang, Karl Woodbridge, Ian Craddock, Ryan McConville, Raul Santos-Rodriguez, Kevin Chetty, et al. 2022\.  OPERAnet, a multimodal activity recognition dataset acquired from radio frequency and vision-based sensors.  _Scientific data_ 9, 1 (2022), 474.
  * Caesar et al. (2020) Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020.  nuscenes: A multimodal dataset for autonomous driving. In _Proceedings of the IEEE/CVF CVPR_. 11621–11631.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-end object detection with transformers. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_. Springer, 213–229.
  * Contributors (2020) MMTracking Contributors. 2020.  MMTracking: OpenMMLab video perception toolbox and benchmark.  https://github.com/open-mmlab/mmtracking.
  * Corona et al. (2021) Kellie Corona, Katie Osterdahl, Roderic Collins, and Anthony Hoogs. 2021.  MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. 1060–1068.
  * Csurka (2017) Gabriela Csurka. 2017.  Domain adaptation for visual applications: A comprehensive survey.  _arXiv preprint arXiv:1702.05374_ (2017).
  * Cui et al. (2023) Can Cui, Yunsheng Ma, Juanwu Lu, and Ziran Wang. 2023.  Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusion.  _arXiv preprint arXiv:2305.17318_ (2023).
  * Duan et al. (2022) Shengshun Duan, Qiongfeng Shi, and Jun Wu. 2022.  Multimodal Sensors and ML-Based Data Fusion for Advanced Robots.  _Advanced Intelligent Systems_ (2022), 2200213.
  * Geiger et al. (2013) Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ 32, 11 (2013), 1231–1237.
  * Han et al. (2023) Xiaotian Han, Quanzeng You, Chunyu Wang, Zhizheng Zhang, Peng Chu, Houdong Hu, Jiang Wang, and Zicheng Liu. 2023.  Mmptrack: Large-scale densely annotated multi-camera multiple people tracking benchmark. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 4860–4869.
  * Huang et al. (2022) Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. 2022.  Multi-modal sensor fusion for auto driving perception: A survey.  _arXiv preprint arXiv:2202.02703_ (2022).
  * John and Kawanishi (2022) Vijay John and Yasutomo Kawanishi. 2022.  A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition. In _Proceedings of the 4th ACM International Conference on Multimedia in Asia_. 1–5.
  * Liu et al. (2022) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022.  A ConvNet for the 2020s.  _Proceedings of the IEEE/CVF CVPR (CVPR)_ (2022).
  * Mao et al. (2021) Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. 2021\.  One million scenes for autonomous driving: Once dataset.  _arXiv:2106.11037_ (2021).
  * Meng et al. (2021) Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. 2021.  Conditional detr for fast training convergence. In _Proceedings of the IEEE/CVF ICCV_. 3651–3660.
  * Oh et al. (2011) Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al. 2011\.  A large-scale benchmark dataset for event recognition in surveillance video. In _CVPR 2011_. IEEE, 3153–3160.
  * Piadyk et al. (2023) Yurii Piadyk, Joao Rulff, Ethan Brewer, Maryam Hosseini, Kaan Ozbay, Murugan Sankaradas, Srimat Chakradhar, and Claudio Silva. 2023.  StreetAware: A High-Resolution Synchronized Multimodal Urban Scene Dataset.  _Sensors_ 23, 7 (2023), 3710.
  * Piechocki et al. (2023) Robert J Piechocki, Xiaoyang Wang, and Mohammud J Bocus. 2023.  Multimodal sensor fusion in the latent representation space.  _Scientific Reports_ 13, 1 (2023), 2005.
  * Samplawski et al. (2023) Colin Samplawski, Shiwei Fang, Ziqi Wang, Deepak Ganesan, Mani Srivastava, and Benjamin Marlin. 2023.  Heteroskedastic Geospatial Tracking with Distributed Camera Networks. In _Uncertainty in Artificial Intelligence_. PMLR.
  * Shu et al. (2021) Yuanchao Shu, Qiang Xu, Jie Liu, Romit Roy Choudhury, Niki Trigoni, and Victor Bahl. 2021.  Indoor Location Competition 2.0 Dataset.  https://www.microsoft.com/en-us/research/publication/indoor-location-competition-2-0-dataset/
  * Sturm et al. (2012) J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. 2012.  A Benchmark for the Evaluation of RGB-D SLAM Systems. In _Proc. of the International Conference on Intelligent Robot Systems (IROS)_.
  * Sun et al. (2020) Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. 2020\.  Scalability in perception for autonomous driving: Waymo open dataset. In _Proceedings of the IEEE/CVF CVPR_. 2446–2454.
  * Ulrich et al. (2018) Michael Ulrich, Thomas Hess, Sherif Abdulatif, and Bin Yang. 2018.  Person recognition based on micro-Doppler and thermal infrared camera fusion for firefighting. In _2018 21st International Conference on Information Fusion (FUSION)_. IEEE, 919–926.
  * Zafari et al. (2019) Faheem Zafari, Athanasios Gkelias, and Kin K Leung. 2019.  A survey of indoor localization systems and technologies.  _IEEE Communications Surveys & Tutorials_ 21, 3 (2019), 2568–2599.