# Dialectal Speech Recognition and Translation of
Swiss German Speech to Standard German Text:
Microsoft’s Submission to SwissText 2021

Yuriy Arabskyy, Aashish Agarwal, Subhadeep Dey, Oscar Koller
Microsoft - Munich, Germany
{yuarabsk, t-aagarwal, subde, oskoller} @microsoft.com

###### Abstract

This paper describes the winning approach in the Shared Task 3 at SwissText
2021 on Swiss German Speech to Standard German Text, a public competition on
dialect recognition and translation. Swiss German refers to the multitude of
Alemannic dialects spoken in the German-speaking parts of Switzerland. Swiss
German differs significantly from standard German in pronunciation, word
inventory and grammar. It is mostly incomprehensible to native German
speakers. Moreover, it lacks a standardized written script. To solve the
challenging task, we propose a hybrid automatic speech recognition system with
a lexicon that incorporates translations, a 1st pass language model that deals
with Swiss German particularities, a transfer-learned acoustic model and a
strong neural language model for 2nd pass rescoring. Our submission reaches
46.04% BLEU on a blind conversational test set and outperforms the second best
competitor by a 12% relative margin.

##  1 Introduction

While general speech recognition has matured to a point where it surpasses
human performance on specific datasets Xiong et al. (2016), dialectal
recognition as in the case of Swiss German Nigmatulina et al. (2020) or Arabic
dialects Ali et al. (2021); Hussein et al. (2021); Ali (2018) still represents
a major challenge. Swiss German refers to the multitude of Alemannic dialects
spoken in the German-speaking parts of Switzerland. It hence represents
dialects that differ significantly from standard or high German in
pronunciation, word inventory and grammar. Moreover, it lacks a standardized
writing system. High German is commonly used for the large majority of written
communication between people and in the media of German-speaking Switzerland,
while in rather informal chats and short messaging Swiss people may use
transliterated non-standardized Swiss dialect. The process of transcribing
Swiss German into high German text therefore requires speech recognition with
an inherent translation step. Moreover, the task can be considered low-
resource as available data remains extremely scarce.

In previous studies, Garner et al. (2014) tackled this challenge by training
hybrid models (HMM-GMM, HMM-DNN, and KL-HMM) to transcribe Walliserdeutsch, a
Swiss-German dialect spoken in the south-western alpine canton of Switzerland
and further used a phrase-based machine translation model to translate it to
standard German. Following this, other researchers explored techniques to add
the translation step in the lexicon by directly mapping Swiss-German
pronunciation to standard German. Stadtschnitzer and Schmidt (2018) estimated
Swiss German pronunciations from a standard German speech recognition model
using a data-driven technique, and trained stronger TDNN-LSTM based acoustic
models. Kew et al. (2020) and Nigmatulina et al. (2020) trained transformer-
based G2P models from standard German to Swiss pronunciations and trained a
Kaldi-based TDNN+ivector system using the WSJ
recipe111https://github.com/kaldi-asr/kaldi/tree/master/egs/wsj. Yet a third
approach is to directly apply end-to-end deep learning models. Büchi et al.
(2020) and Agarwal and Zesch (2020) at SwissText 2020 used the Jasper
architecture Li et al. (2019) and Mozilla DeepSpeech Hannun et al. (2014),
respectively. In both cases the system was first trained on high German data
and then transfer-learned to Swiss German.

In this paper, we describe our proposal to solve the challenging task of
transcribing Swiss German speech to standard German text. It won the
competition at SwissText 2021 with a large margin to other competing systems.

##  2 System Overview

In this section, we propose our changes to a conventional hybrid Bourlard and
Dupont (1996) automatic speech recognition (ASR) system, which relies on
lexicon and alignments for good performance. We present details in order to
enable it for dialect speech recognition and translation.

###  2.1 Data

To train our proposed model, we utilized a selection of publicly available and
internal datasets. Our starting point was the Swiss Parliament Corpus V2
dataset Plüss et al. (2020) shared as part of the SwissText 2021 competition.
It covers 293 hours and contains recordings from the local parliament of the
Kanton Bern. Its transcripts are in standard German while the audio covers
Swiss German (predominantly in the Bernese dialect). The dataset has been
preprocessed by the publishers with the purpose of cleaning its annotations
and ensuring a good match between audio content and transcription. It is
provided with a choice of different preprocessing flavors. We used the
train_all split. In addition, we used a 493-hour internal dataset representing
a media domain encompassing conversational speech from interviews,
discussions, podcasts and others. A subset (around 50 hours) of the data is
annotated with both Swiss transliterations as well as standard German. The
remaining data has only been annotated with standard German. Additionally, we
used an internal high German dataset encompassing around 10k hours to pre-
train our model.

In terms of test data, the SwissText 2021 competition was accompanied by a 13
hours conversational test set covering Swiss German speakers from all German-
speaking parts of Switzerland. The encountered dialectal distribution is
claimed to closely match the real distribution in Switzerland. The set was not
disclosed to the participants. Hence, for the analysis in this paper, we
report our numbers on a publicly available test set part of the dataset from
the Bernese parliament Plüss et al. (2020). It comprises 6 hours of dialectal
speech.

###  2.2 Lexicon

2nd person plural | 2nd person sing | Diminuation | Shortening | Translation | Variability
---|---|---|---|---|---
fragt | fragst | erdmännchen | gymnasium | kopf | kannst
_f hr a_ g ax t_ | _f hr a_ k sh_ | _e_r t m eh n l i__ | _g ih m i__ | _g hr ih n t_ | _k a sh_
riecht | riechst | gläschen | schwimmbad | kneipe | zweites
_sh m oe k c ax t_ | _sh m oe k sh_ | _g l e_ s l i__ | _b a_ d ih_ | _b ai ts_ | _ts v ai t_
Table 1: Example words and pronunciations from each G2P test condition

We propose to incorporate the translation from Swiss German to standard German
as part of the lexicon. However, this leads to a complex and often ambiguous
mapping from phoneme to grapheme sequences, which is very different from
languages with a direct relation between writing scheme and pronunciation
(e.g. English or standard German). Subsequently, statistical models that map
graphemes to phonemes (G2P) trained on Swiss German data incorporating such
translations yield much noisier output with significantly higher phone error
rates as compared to G2Ps for standard languages. To mitigate this problem, we
construct the lexicon in several stages.

In a first step, we make use of parallel corpora encompassing Swiss and
standard German annotations to extract word mappings between Swiss and
standard German. Sophisticated filtering methods help to ensure a high quality
of these mappings. We opt for frequency filtering and filtering based on
vicinity in a word embedding space Bojanowski et al. (2017) of Swiss German
words taking the most frequent mapping as center point.

In a second step, a standard German G2P model is applied to convert Swiss
German transliterations into corresponding phone sequences. This results in a
dictionary that maps standard German words to Swiss pronunciations. Jointly
with existing Swiss German lexicon resources Schmidt et al. (2020), the
previously generated mappings are then used to train a dedicated Swiss German
G2P model.

We evaluate the quality of the resulting G2P model on a manually labeled test
set. Those cover mappings from standard German words to Swiss German phone
sequences and encompass a variety of relevant categories such as diminuation,
shortening or translation. Refer to Table 1 for samples of the assessed
categories.

The Swiss G2P model allows to find suitable pronunciations for the relevant
word inventory present in the acoustic and language model training corpora.
However, to further increase the quality of the given pronunciations, data-
driven lexicon learning techniques Zhang et al. (2017) are applied. Those help
to identify and correct noisy lexicon entries.

###  2.3 Language Model

Incorporating the translation from Swiss German to standard German as part of
the lexicon introduces significant ambiguity in the decoding process. To
counteract, we suggest using a strong standard German language model (LM)
which helps to produce accurate hypotheses. We employ a first pass count-based
LM to output up to 100 sentence hypotheses and a second pass neural LSTM (long
short-term memory) LM Sundermeyer et al. (2012) for rescoring Deoras et al.
(2011). The first pass model is a 5-gram LM trained on large amounts of
standard German text corpora totalling to over 100 billion words. We apply
Kneser-Ney smoothing Kneser and Ney (1995).

Furthermore, we make some adjustments to better deal with Swiss German
particularities, as described in the following paragraphs.

#### Compounds:

German is a compounding language and tends to compose words (particularly
nouns) of several smaller subwords. The resulting chains of word stems can
lead to an infinitely large vocabulary size with words that occur very
infrequently throughout the corpus. This spreading of probability mass weakens
the LM. We hence decompound all compounded words in the training corpus and
split them into subwords.

#### Clitics:

Swiss German tends to merge words beyond compounding, not preserving word
stems Hollenstein and Aepli (2014). For instance, the Swiss German ‘ _hemmer_
’ is the translation of ‘ _haben wir_ ’ in standard German (English: ‘ _have
we_ ’). We identified approximately $8000$ clitics in our corpus. We
incorporate them in the decoding process by updating lexicon and LM. Following
the example above the translated clitic ‘ _haben#wir_ ‘ with the corresponding
Swiss pronunciation is added to the lexicon. As for the LM, we merge
occurrences of relevant word pairs and interpolate with the unmerged LM.

###  2.4 Acoustic model

The acoustic model is trained with 80 dimensional log-mel filterbank features,
with a processing window of 25ms and 10ms frame shift. The feature vector from
the previous frame is concatenated with the current frame to obtain a 160
dimensional vector. We used a LC-BLSTM (latency controlled bidirectional long
short-term memory) based acoustic model, that is popularly applied in speech
recognition for controlling decoding latency to a few frames Chen and Huo
(2016). The model was trained with alignments from a feed-forward network with
context-dependent tied states (senones). The model has $\sim$9k senone units.
The LC-BLSTM is trained with 6 hidden layers with 512 units each. The hidden
vectors from the forward and backward propagation were concatenated and then
projected to a 512 dimensional vector. The model is trained with a cross
entropy loss function. The decoding lexicon is extended with Swiss German
words for training. The transliterations are used during forced alignment
whenever possible. This helps to reduce the pronunciation ambiguity in the
alignment phase and is especially helpful in the early training phases when no
strong model is available for alignment.

The results are reported in terms of BLEU Papineni et al. (2002) and word
error rate (WER) on the Swiss Parliament test set described in Section 2.1.

##  3 Results and Discussions

Row | Description | WER | BLEU
---|---|---|---
1 | Swiss Parliament | 45.93 | 33.16
2 |  \+ Transfer Learning | 42.09 | 37.53
3 |  \+ Internal Data | 41.10 | 38.49
4 |  \+ 2nd Pass Rescoring | 38.71 | 41.91
Table 2: Performance in [%] of different system configurations evaluated on
the Swiss Parliament test set.

An ablation study of the proposed approaches is presented in Table 2. All of
the performance gains in this section will be reported as relative percentage
improvements, while aforementioned table contains absolute numbers.

We first evaluate the effect of transfer learning on the results with the
Swiss Parliament training set. It can be observed that it significantly helps
to improve both WER and BLEU. In particular, the transfer-learned model (row
2, Table 2) improves over the model trained from scratch (row 1, Table 2) by
$8.4\%$ WER and $11.6\%$ BLEU. The result shows that a well-trained German
model can effectively boost the limited resources of Swiss German.

Further adding additional internal training data shows additional gains in
performance. As such, we observe that the WER improves by $2.3\%$ and BLEU by
$2.5\%$.

Finally, 2nd Pass rescoring is applied as described in Section 2.3 to reorder
the top 100 hypotheses. It can be observed from row 4, Table 2 that rescoring
helps to improve the performance by $5.8\%$ WER and $8.9\%$ BLEU.

Our submission to SwissText 2021 achieves $46.04\%$ BLEU on the official
SwissText blind test set. This leads to a $12\%$ relative margin in BLEU with
respect to the second best competitor which was $40.99\%$.

The acoustic models have been trained using 8 GPUs for 25 epochs. This results
in a total training time of around 400 GPU-hours when training on Swiss
Parliament only and about 1200 GPU-hours when adding the internal data.

##  4 Conclusion and Future Work

In this paper, we described a speech recognition system that achieves strong
results on the task of recognizing Swiss German dialect and translating it
into standard German text. We proposed a hybrid ASR system with a lexicon that
incorporates translations, a 1st pass language model that deals with Swiss
German word compounding and clitics, an acoustic model that is transfer-
learned from standard German resources and a strong neural language model for
2nd pass rescoring to smoothen translation artifacts. Furthermore, we provided
an ablation study that allows to infer the effect of adding training data,
performing transfer learning and 2nd pass rescoring. Our submission reached
46.04% BLEU on a challenging conversational test set and outperformed all
competing approaches by a large margin.

In terms of future work, we would like to investigate word re-orderings as
part of the translation, which our current model does not actively support.
For instance, Swiss German frequently moves verbs in relative clauses to
different positions with respect to the standard German word order.
Furthermore, sequence discriminative training is a promising route for
exploration as well as using unsupervised data for acoustic model training.

## References

  * Agarwal and Zesch (2020) Aashish Agarwal and Torsten Zesch. 2020.  Ltl-ude at low-resource speech-to-text shared task: Investigating mozilla deepspeech in a low-resource setting.
  * Ali et al. (2021) Ahmed Ali, Shammur Chowdhury, Mohamed Afify, Wassim El-Hajj, Hazem Hajj, Mourad Abbas, Amir Hussein, Nada Ghneim, Mohammad Abushariah, and Assal Alqudah. 2021.  Connecting Arabs: Bridging the gap in dialectal speech recognition.  _Communications of the ACM_ , 64(4):124–129.
  * Ali (2018) Ahmed Mohamed Abdel Maksoud Ali. 2018.  _Multi-Dialect Arabic Broadcast Speech Recognition_.  Ph.D. thesis, University of Edinburgh, Edinburgh, UK.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Bourlard and Dupont (1996) Hervé Bourlard and Stéphane Dupont. 1996.  A new ASR approach based on independent processing and recombination of partial frequency bands.  In _Proc. Int. Conf. on Spoken Language Processing (ICSLP)_ , volume 1, pages 426–429.
  * Büchi et al. (2020) Matthias Büchi, Malgorzata Anna Ulasik, Manuela Hürlimann, Fernando Benites, Pius von Däniken, and Mark Cieliebak. 2020.  Zhaw-init at germeval 2020 task 4: Low-resource speech-to-text.
  * Chen and Huo (2016) Kai Chen and Qiang Huo. 2016.  Training deep bidirectional lstm acoustic model for lvcsr by a context-sensitive-chunk bptt approach.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 24(7):1185–1193.
  * Deoras et al. (2011) Anoop Deoras, Tomáš Mikolov, and Kenneth Church. 2011.  A Fast Re-scoring Strategy to Capture Long-Distance Dependencies.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 1116–1127, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Garner et al. (2014) Philip N. Garner, David Imseng, and Thomas Meyer. 2014.  Automatic Speech Recognition and Translation of a Swiss German Dialect: Walliserdeutsch.  In _Proc. of the Ann. Conf. of the Int. Speech Commun. Assoc. (Interspeech)_ , Singapore.
  * Hannun et al. (2014) Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. 2014\.  Deep speech: Scaling up end-to-end speech recognition.  _arXiv preprint arXiv:1412.5567_.
  * Hollenstein and Aepli (2014) Nora Hollenstein and Noëmi Aepli. 2014.  Compilation of a Swiss German dialect corpus and its application to PoS tagging.  In _Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects_ , pages 85–94.
  * Hussein et al. (2021) Amir Hussein, Shinji Watanabe, and Ahmed Ali. 2021.  Arabic Speech Recognition by End-to-End, Modular Systems and Human.  _arXiv:2101.08454 [cs, eess]_.
  * Kew et al. (2020) Tannon Kew, Iuliia Nigmatulina, Lorenz Nagele, and Tanja Samardzic. 2020.  Uzh tilt: A kaldi recipe for swiss german speech to standard german text.
  * Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995.  Improved backing-off for m-gram language modeling.  In _Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_ , volume 1, pages 181–184.
  * Li et al. (2019) Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. 2019.  Jasper: An end-to-end convolutional neural acoustic model.  _arXiv preprint arXiv:1904.03288_.
  * Nigmatulina et al. (2020) Iuliia Nigmatulina, Tannon Kew, and Tanja Samardzic. 2020.  ASR for non-standardised languages with dialectal variation: the case of Swiss German.  In _Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 15–24, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318. Association for Computational Linguistics.
  * Plüss et al. (2020) Michel Plüss, Lukas Neukom, and Manfred Vogel. 2020.  Swiss parliaments corpus, an automatically aligned swiss german speech to standard german text corpus.  _arXiv preprint arXiv:2010.02810_.
  * Schmidt et al. (2020) Larissa Schmidt, Lucy Linder, Sandra Djambazovska, Alexandros Lazaridis, Tanja Samardžić, and Claudiu Musat. 2020.  A Swiss German Dictionary: Variation in Speech and Writing.  _arXiv:2004.00139 [cs]_.
  * Stadtschnitzer and Schmidt (2018) Michael Stadtschnitzer and Christoph Schmidt. 2018.  Adaptation and training of a swiss german speech recognition system using data-driven pronunciation modelling.  _Proceedings of DAGA-44. Jahrestagung für Akustik, München, Germany_.
  * Sundermeyer et al. (2012) Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012.  LSTM neural networks for language modeling.  In _Proc. of the Ann. Conf. of the Int. Speech Commun. Assoc. (Interspeech)_ , pages 194–197, Portland, OR, USA.
  * Xiong et al. (2016) Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2016.  Achieving human parity in conversational speech recognition.  _arXiv preprint arXiv:1610.05256_.
  * Zhang et al. (2017) Xiaohui Zhang, Vimal Manohar, Daniel Povey, and Sanjeev Khudanpur. 2017.  Acoustic Data-Driven Lexicon Learning Based on a Greedy Pronunciation Selection Framework.  In _Proc. of the Ann. Conf. of the Int. Speech Commun. Assoc. (Interspeech)_ , pages 2541–2545. ISCA.


# Path-GPTOmic: A Balanced Multi-modal Learning Framework
for Survival Outcome Prediction

###### Abstract

For predicting cancer survival outcomes, standard approaches in clinical
research are often based on two main modalities: pathology images for
observing cell morphology features, and genomic (e.g., bulk RNA-seq) for
quantifying gene expressions. However, existing pathology-genomic multi-modal
algorithms face significant challenges: (1) Valuable biological insights
regarding genes and gene-gene interactions are frequently overlooked; (2) one
modality often dominates the optimization process, causing inadequate training
for the other modality. In this paper, we introduce a new multi-modal “Path-
GPTOmic” framework for cancer survival outcome prediction. First, to extract
valuable biological insights, we regulate the embedding space of a foundation
model, scGPT, initially trained on single-cell RNA-seq data, making it
adaptable for bulk RNA-seq data. Second, to address the imbalance-between-
modalities problem, we propose a gradient modulation mechanism tailored to the
Cox partial likelihood loss for survival prediction. The contributions of the
modalities are dynamically monitored and adjusted during the training process,
encouraging that both modalities are sufficiently trained. Evaluated on two
TCGA(The Cancer Genome Atlas) datasets, our model achieves substantially
improved survival prediction accuracy.

Index Terms—  Multi-modal, Survival outcome prediction, Pathology images,
Genomics.

##  1 Introduction

Pathology images and genomic assays are two main data sources for predicting
the survival outcome of cancer patients. Pathology images contain information
on cell mitosis, cell morphology, and the micro-environment. Genomic data from
RNA-seq measure the abundance of RNA transcripts, providing critical
biological insights into cell identity, cellular activity, stage of
development and differentiation, as well as cell functionality [1]. In
clinical practice, bulk RNA-seq (average global gene expression among cells)
is more cost-effective, and is widely employed for analyzing cancer initiation
and progression, as well as predicting survival outcomes [2].

To jointly utilize these two complementary sources for automatic cancer
survival prediction, researchers have developed a series of multi-modality
deep learning algorithms. For example, in [3, 4], genomic data were processed
with self-normalizing networks (SNN) and fused with pathology image embeddings
with Kronecker product. Transformer models were employed to capture genotype-
phenotype interactions through an attention mechanism [5]. Different
modalities were projected into the same latent space to enclose distances
between multi-modal embeddings of the same patients [6, 7].

However, these methods still face two major challenges. First, the biological
insight of genomic data cannot be fully explored by SNN or Transformer models
with limited training data, and external knowledge on human cells is not
utilized to compute more accurate genomic embeddings. Second, researchers
found that the dominant modality (the one with better performance) may
suppress the training process of the other modality [8, 9]. Consequently, the
other modality may not generalize well to test data.

In this paper, we propose a new multi-modal Path-GPTOmic framework, which
combines pathology images and genomic data from patient specimen for
predicting cancer survival outcomes. To address the first challenge, we seek
to learn a smooth latent space for bulk RNA-seq embeddings by incorporating a
foundation model, scGPT [1]. This model was originally trained on single-cell
RNA-seq data from large human cell atlases [10], and has demonstrated superior
performance in cell functionality analysis (e.g., cell type annotation and
genetic perturbation prediction). However, in our prior experiments, we
observed that directly applying scGPT to bulk RNA-seq improved model
performance only marginally for downstream tasks. One possible reason for this
is that the direct application maps bulk RNA to an unfair latent space, where
distances between embedding vectors do not accurately reflect RNA
similarities. To address this issue, we adopt generative model practices [11,
12, 13] to smooth the latent space with mix-up regulation. Specifically, we
append a smoothing module (i.e., a multi-layer-proceptron (MLP) network) after
scGPT. Then we train it by simulating bulk RNA-seq with individual single-cell
RNA expression [2]. This approach enables us to achieve an interpolatable
latent space for gene expression, enhancing generalizability for bulk RNA
data.

To tackle the second challenge, we combine the genomic and image branches of
the model and closely monitor the contribution of each modality in training
process. We find that genomic branch outperforms and contributes less to the
Cox partial likelihood loss [14]. As a result, the genomic branch dominates
the training process, causing under-optimization of the image branch. To
alleviate this issue in survival outcome prediction, we propose to control the
loss optimization process by dynamically adjusting the gradient. Specifically,
we assess the contributions of the two branches and appropriately modulate the
gradient of the under-optimized image modality.

Our contributions can be summarized in three key aspects. (1) In clinical
research, bulk RNA-seq data are more cost-effective and easier to acquire. To
our best knowledge, we are the first to extend the single-cell foundation
model scGPT for processing bulk RNA-seq data of patient samples. (2) We take
pioneering steps to address the training imbalance problem in multi-modal
pathology-genomics fusion tasks. (3) We evaluate our new method using two TCGA
datasets, and obtain performance improvement compared to the baselines.

##  2 Method

Fig. 1 gives an overview of our method. In Section 2.1, we show how to smooth
the scGPT-derived single-cell RNA-seq embedding space, enabling scGPT to be
adapted for bulk RNA-seq embeddings. In Section 2.2, we describe how to
balance the contributions of the genomics branch and image branch in training
Cox partial likelihood loss, encouraging better optimization for both
modalities.

Fig. 1: Illustrating our Path-GPTOmic pipeline. First, we train MLP-A for
regulating the latent space for bulk RNA-seq embeddings, freezing the scGPT
parameters. Second, we train SNN and MLP-B for genomics embedding and Image
Encoder (T2T-ViT [15]) for embedding pathology image features, freezing both
the scGPT parameters and MLP-A parameters.

###  2.1 Regulating the Genomics Embedding Space

Cancer survival outcome predictions often require profiling transcript
abundance through RNA-seq, including both single-cell RNA-seq (scRNAseq) and
bulk RNA-seq (average global gene expression). On one hand, scRNAseq can
provide fine-grained individual cell-level evidence for cancer. By training a
Transformer model on large datasets, scGPT has uncovered gene-gene
interactions and achieved superior performance on various downstream tasks. On
the other hand, bulk RNA is more cost-effective and widely used in clinical
research. Hence, it is desirable to extend scGPT to the analysis of bulk RNA-
seq data for survival outcome prediction.

Our preliminary results show that directly applying scGPT to bulk RNA data is
not significantly beneficial for downstream tasks (see Section 3). One
possible reason is that without sufficient average single-cell RNA-seq input
for training, the intermediate interpolation area of the latent space is
abrupt and does not generalize well for global average data (i.e., bulk RNA-
seq). A similar phenomenon has been observed in generation tasks. For example,
as shown in [16, 17], a model $f(\cdot)$ performing well on inputs
$x_{1},x_{2}$ may collapse and yield artifact results for interpolated input
$(\lambda x_{1}+(1-\lambda)x_{2})$, with $\lambda\in[0,1]$.

By applying scGPT to bulk RNA-seq, our goal is to regulate the latent space
using Mixup-based regulation [11, 18], encouraging “average” RNA-seq input to
yield reasonable embeddings that are beneficial for downstream tasks. We
choose cell type annotation [1] prediction as the probing task, as it is
closely related to survival outcomes. Our pipeline is outlined in the top part
of Fig. 1. Two randomly selected scRNAseq samples are interpolated by a mix
layer, processed by a pre-trained scGPT module, a multi-layer perceptron (MLP)
network (called as MLP-A), and a linear layer as a classifier. By encouraging
the model output to converge to average cell types, we expect the embedding
space (i.e., the output of MLP-A) to be regulated for representing average
cell information.

Our model is pre-trained as follows. First, in each optimization step, we
randomly select two scRNAseq samples from the scGPT assembled human cell
scRNAseq dataset. Formally, we denote the input scRNAseq as $x_{i},x_{j}$,
represented as one-hot cell type label encodings $\bf{t_{i}},\bf{t_{j}}$,
respectively. Second, we simulate the bulk RNAseq by interpolating
$x_{i},x_{j}$ with $(\lambda x_{i}+(1-\lambda)x_{j})$, where $\lambda$ is a
scalar value uniformly and randomly chosen from $[0,1]$. Third, to avoid
excessively perturbing the well-trained scGPT model, we fix the scGPT
parameters and append a three-layer MLP-A to it to regulate the scGPT output.
Then, we optimize the output with a regression target value
$(\lambda\bf{t_{i}}+(1-\lambda)\bf{t_{j}})$. In this way, the distance between
the latent spaces can represent the weighted contributions of the RNAseq from
the two input samples.

###  2.2 Balanced Multi-modal Training

In the optimization process of multi-modal deep learning models, researchers
have observed that the dominant model, which displays better performance,
tends to suppress the optimization of the others [8, 9]. While the issue of
imbalanced training has been addressed in optimizing cross-entropy loss, it
has not been thoroughly investigated in the context of pathology-genomics
multi-modal fusion for optimizing Cox partial likelihood loss in survival
prediction.

Framework. To better discuss the multi-modal imbalance training problem, we
use a common prototype model, as shown in Fig. 1. In this model, the Image
Encoder submodule can be flexibly replaced with alternatives such as ResNet
[3], Transformer [6], or T2T-ViT [15]. We define the training set as
$\mathcal{D}=\\{(g_{k},p_{k})\\}_{k=1}^{K}$, where $K$ is the number of paired
genomic data $g_{k}$ and pathology images $p_{k}$. The genomic data consist of
copy number variations (CNV), mutations, and RNA-seq. We process the CNV and
mutations with SNN in the same structure as in [3], yielding a vector
$\bf{G^{(1)}_{k}}$. The RNA-seq is processed by scGPT and MLP-A sequentially,
with the network parameters pre-trained and fixed as described in Section 2.1,
yielding a vector $\bf{G^{(2)}_{k}}$. These are then concatenated, and we
train a new MLP-B to produce a genomic feature
$\bf{G_{k}}=MLP_{B}([\bf{G^{(1)}_{k}}||\bf{G^{(2)}_{k}}])$. In parallel, the
pathology image $p_{k}$ is processed by the Image Encoder to obtain an image
feature $\bf{P_{k}}$. We then concatenate $\bf{G_{k}}$ and $\bf{P_{k}}$ as
$[\bf{G_{k}}||\bf{P_{k}}]$. The log hazard ratio $\theta_{k}$ for patient $k$
is derived by using a linear classifier with trainable weights $W$ and bias
$b$, as $\theta_{k}=W([\bf{G_{k}}||\bf{P_{k}}])+b$.

Following [8], we represent $W$ as a combination of two blocks $\it{W^{G}}$
and $\it{W^{P}}$, and rewrite the equation as:

$\theta_{k}=\it{W^{G}}\cdot\bf{G_{k}}+\it{W^{P}}\cdot\bf{P_{k}}+b.$ (1)

To predict survival outcome, Cox partial log-likelihood [14, 3] is used as
cost function:

$L_{Cox}=\sum_{C(k)=1}(\theta_{k}-\log\sum_{R(t_{k})}\exp({\theta_{k}})),$ (2)

where $C(k)=1$ represents the uncensored events, and $R(t_{k})$ is the risk
set at time $t_{k}$. The gradient is:

$\displaystyle\frac{\partial L_{Cox}}{\partial\theta_{k}}$
$\displaystyle=\sum_{C(k)=1}(1-\frac{\exp({\theta_{k}})}{\sum_{R(t_{k})}\exp({\theta_{k}})})$
(3)
$\displaystyle=\sum_{C(k)=1}(1-\frac{e^{({W^{G}G_{k}+W^{P}P_{k}+b})}}{\sum_{R(t_{k})}e^{({W^{G}G_{k}+W^{P}P_{k}+b})}}).$
(4)

One can observe that in the above equation, when one modality (say, the
genomics modality) performs well, it will dominate the gradient loss in
Equation (4) via $\it{W^{G}}\cdot\bf{G_{k}}$. The global loss will be close to
zero. Thus, limited training optimization will be applied to the image
modality. Even when the model converges, the image modality possibly remains
inadequately trained.

Balanced Training. To alleviate the imbalance training problem, we evaluate
the contribution discrepancy ratio $\rho^{G},\rho^{P}$ for Cox partial log-
likelihood loss, as:

$\displaystyle\rho^{G}$
$\displaystyle=(\frac{({W^{G}\bf{G_{k}}+\frac{b}{2}})}{\sum_{R(t_{k})}e^{({W^{G}\bf{G_{k}}+\frac{b}{2}})}})/(\frac{({W^{P}\bf{P_{k}}+\frac{b}{2}})}{\sum_{R(t_{k})}e^{({W^{P}\bf{P_{k}}+\frac{b}{2}})}}),$
(5) $\displaystyle\rho^{P}$ $\displaystyle=1/\rho^{G}.$ (6)

Formally, inspired by [8], we modulate the gradient by estimating the
contributions of the two modalities. We update the model parameters in each
iteration $t$, as:

$\displaystyle\phi^{G}_{t+1}$
$\displaystyle=\phi^{G}_{t}-\eta\cdot\min(1-\text{tanh}(\rho^{G}-1),1)\cdot
g(\phi^{G}_{t}),$ (7) $\displaystyle\phi^{P}_{t+1}$
$\displaystyle=\phi^{P}_{t}-\eta\cdot\min(1-\text{tanh}(\rho^{P}-1),1)\cdot
g(\phi^{P}_{t}),$ (8)

where $\phi^{G}_{t+1}$ and $g(\phi^{G}_{t})$ represent the parameters and
gradient for the MLP in the genomics network, $\phi^{P}_{t+1}$ and
$g(\phi^{P}_{t})$ represent the parameters and gradient for the pathology
image network, and $\eta$ is the learning rate. In this way, the learning rate
of the modality with higher $\rho$ is suppressed.

##  3 Experiments

###  3.1 Datasets and Implementation Details

Our experiments utilize two datasets [3] that consist of hematoxylin and eosin
(H&E)-stained pathology images, corresponding genomic features (mutations,
copy number variations (CNV), RNA-seq), and patient survival outcomes.
Specifically, the TCGA-GBMLGG dataset contains 1505 gliomas (brain and spinal
cord tumors) samples, and the TCGA-KIRC dataset contains 1251 clear cell renal
cell carcinoma samples. We apply the same experimental protocol as [3] by
evaluating model performance with 15-fold cross-validation.

In our pipeline, the CNV and mutation information is handled by SNN with the
same architecture as [3]. The bulk RNA-seq data are processed with scGPT [1].
Both MLP-A and MLP-B are three-layer perceptron networks with hidden layer
dimension 128. The classifier is a linear layer mapping the MLP-A generated
feature vector to 17 cell type categories. The batch size is 32. Our model is
implemented with PyTorch and is trained on an NVIDIA A10 GPU.

###  3.2 Results

Main Results. Table 1 and Table 2 show our main results. Our model is compared
with the baselines of SCNN, SGCNN [19], Pathomic Fusion [3], and the
supervised multi-modal setting in [6]. For fair comparison, we also replace
the CNN in Pathomic Fusion [3] with the T2T-ViT [15] backbone to extract
features from pathology images. We use C-Index to measure performance. First,
our model outperforms all the baselines on both datasets. Second, similar to
the setting “Pathomic Fusion [3] (T2T-ViT [15] \+ SNN)”, we also use the
backbone T2T-ViT and SNN to process the image features and CNV features. Our
model gains around $2\%$ improvement on both datasets. This indicates the
effectiveness of our design for regulating scGPT embeddings and gradient
modulation.

Table 1: Comparison of C-Index performance on the TCGA-GBMLGG dataset ($p<0.05$).  Method | C-Index
---|---
SCNN (Histology Only) [19] | 0.754
Histology CNN [3] | 0.792 ± 0.014
Histology T2T-ViT [15] | 0.803 ± 0.016
Genomic SNN [3] | 0.808 ± 0.014
GSCNN (Histology + Genomic) [19] | 0.781
Pathomic Fusion [3] (CNN + SNN) | 0.820 ± 0.009
Pathomic Fusion [3] (T2T-ViT [15] \+ SNN) | 0.826 ± 0.010
PathOmics [4] | 0.833 ± 0.012
Ours | 0.848 ± 0.014
Table 2: Comparison of C-Index performance on the TCGA-KIRC dataset ($p<0.05$). Method | C-Index
---|---
Histology CNN [3] | 0.671 ± 0.023
Histology T2T-ViT [15] | 0.683 ± 0.023
Genomic SNN [3] | 0.684 ± 0.025
Pathomic Fusion [3] (CNN + SNN) | 0.719 ± 0.031
Pathomic Fusion [3] (T2T-ViT [15] \+ SNN) | 0.727 ± 0.033
PathOmics [4] | 0.736 ± 0.024
Ours | 0.754 ± 0.030

Ablation Study. We evaluate the impact of each of our key model components in
Table 3. First, we find that directly applying scGPT is a sub-optimal solution
as the performance is improved only by $0.05\%$ (see Exp. 1 and Exp. 2).
Second, by incorporating our mix-up regulation module to smooth the latent
space, the scGPT’s ability to process bulk RNA genomics data is realized (see
Exp. 1 and Exp. 4). Third, our gradient modulation (GradMod.) can effectively
help improve the performance, in comparison with direct concatenation and the
performance with the Kronecker product-based fusion in [3] (see the comparison
between Exp. 2 and Exp. 3, and the comparison among Exp. 4, Exp. 5, and Exp.
6).

Table 3: Ablation study on the TCGA-GBMLGG dataset. Exp. | Encoding RNA-seq | Fusion | C-Index
---|---|---|---
1 | SNN | Concat | 0.826 ± 0.010
2 | scGPT w/o smooth | Concat | 0.831 ± 0.012
3 | scGPT w/o smooth | GradMod. | 0.840 ± 0.013
4 | scGPT w/ smooth | Concat | 0.838 ± 0.013
5 | scGPT w/ smooth | Kro. Prod. [3] | 0.840 ± 0.013
6 | scGPT w/ smooth | GradMod. | 0.848 ± 0.014

##  4 Conclusions

In this paper, we addressed two main challenges in pathology image and
genomics data fusion. First, we showed how to effectively use the advanced
foundation model scGPT, originally designed for single cell RNA-seq, for
processing bulk RNA-seq data. Second, we tackled the imbalance training
problem between image modality and genomics modality by proposing gradient
modulation for the Cox partial likelihood loss. Evaluated on two TCGA
datasets, our model effectively improved the performance compared to the
baseline models.

## References

  * [1] Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, and Bo Wang,  “scGPT: Towards building a foundation model for single-cell multi-omics using generative AI,”  bioRxiv, pp. 2023–04, 2023.
  * [2] Xinmin Li and Cun-Yu Wang,  “From bulk, single-cell to spatial RNA sequencing,”  International Journal of Oral Science, vol. 13, no. 1, pp. 36, 2021.
  * [3] Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, and Faisal Mahmood,  “Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,”  IEEE Transactions on Medical Imaging, vol. 41, no. 4, pp. 757–770, 2020.
  * [4] Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al.,  “Pan-cancer integrative histology-genomic analysis via multimodal deep learning,”  Cancer Cell, vol. 40, no. 8, pp. 865–878, 2022.
  * [5] Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood,  “Multimodal co-attention Transformer for survival prediction in gigapixel whole slide images,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 4015–4025.
  * [6] Kexin Ding, Mu Zhou, Dimitris N Metaxas, and Shaoting Zhang,  “Pathology-and-genomics multimodal Transformer for survival outcome prediction,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 622–631.
  * [7] Anika Cheerla and Olivier Gevaert,  “Deep learning with multimodal representation for pancancer prognosis prediction,”  Bioinformatics, vol. 35, no. 14, pp. i446–i454, 2019.
  * [8] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu,  “Balanced multimodal learning via on-the-fly gradient modulation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8238–8247.
  * [9] Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu,  “MMCosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [10] Aviv Regev, Sarah A Teichmann, Eric S Lander, Ido Amit, Christophe Benoist, Ewan Birney, Bernd Bodenmiller, Peter J Campbell, Piero Carninci, Menna Clatworthy, et al.,  “Science forum: The human cell atlas,”  eLife, vol. 6, pp. e27041, 2017.
  * [11] Yahui Liu, Enver Sangineto, Yajing Chen, Linchao Bao, Haoxian Zhang, Nicu Sebe, Bruno Lepri, and Marco De Nadai,  “Smooth image-to-image translations with latent space interpolations,”  arXiv preprint arXiv:2210.00841, 2022.
  * [12] Hongxiao Wang, Hao Zheng, Jianxu Chen, Lin Yang, Yizhe Zhang, and Danny Z Chen,  “Unlabeled data guided semi-supervised histopathology image segmentation,”  in 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2020, pp. 815–820.
  * [13] Christopher Beckham, Sina Honari, Vikas Verma, Alex M Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Chris Pal,  “On adversarial mixup resynthesis,”  Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [14] Travers Ching, Xun Zhu, and Lana X Garmire,  “Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data,”  PLoS Computational Biology, vol. 14, no. 4, pp. e1006076, 2018.
  * [15] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan,  “Tokens-to-Token ViT: Training Vision Transformers from scratch on ImageNet,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 558–567.
  * [16] Yahui Liu, Enver Sangineto, Yajing Chen, Linchao Bao, Haoxian Zhang, Nicu Sebe, Bruno Lepri, Wei Wang, and Marco De Nadai,  “Smoothing the disentangled latent style space for unsupervised image-to-image translation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10785–10794.
  * [17] Momchil Peychev, Anian Ruoss, Mislav Balunović, Maximilian Baader, and Martin Vechev,  “Latent space smoothing for individually fair representations,”  in European Conference on Computer Vision. Springer, 2022, pp. 535–554.
  * [18] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel,  “MixMatch: A holistic approach to semi-supervised learning,”  Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [19] Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, José E Velázquez Vega, Daniel J Brat, and Lee AD Cooper,  “Predicting cancer outcomes from histology and genomics using convolutional networks,”  Proceedings of the National Academy of Sciences, vol. 115, no. 13, pp. E2970–E2979, 2018.


# Interactive Distance Field Mapping and Planning to Enable Human-Robot
Collaboration

Usama Ali1∗, Lan Wu2∗, Adrian Müller1, Fouad Sukkar2, Tobias Kaupp1, Teresa
Vidal-Calleja2 This work was supported by the Industrial Transformation
Training Centre (ITTC) for Collaborative Robotics in Advanced Manufacturing
(also known as the Australian Cobotics Centre) funded by ARC (Project ID:
IC200100001) and the Bavarian Research Foundation (grant AZ-1512-21).1Authors
are with the Center for Robotics (CERI) at the Technical University of Applied
Sciences Würzburg-Schweinfurt (THWS), Germany.2Authors are with the Robotics
Institute, Faculty of Engineering and IT, University of Technology Sydney
(UTS), Australia.∗These authors are co-first authors and contributed equally
to this work. Corresponding author<EMAIL_ADDRESS>
###### Abstract

Human-robot collaborative applications require scene representations that are
kept up-to-date and facilitate safe motions in dynamic scenes. In this letter,
we present an interactive distance field mapping and planning (IDMP) framework
that handles dynamic objects and collision avoidance through an efficient
representation. We define interactive mapping and planning as the process of
creating and updating the representation of the scene online while
simultaneously planning and adapting the robot’s actions based on that
representation. Given depth sensor data, our framework builds a continuous
field that allows to query the distance and gradient to the closest obstacle
at any required position in 3D space. The key aspect of this work is an
efficient Gaussian Process field that performs incremental updates and
implicitly handles dynamic objects with a simple and elegant formulation based
on a temporary latent model. In terms of mapping, IDMP is able to fuse point
cloud data from single and multiple sensors, query the free space at any
spatial resolution, and deal with moving objects without semantics. In terms
of planning, IDMP allows seamless integration with gradient-based motion
planners facilitating fast re-planning for collision-free navigation. The
framework is evaluated on both real and synthetic datasets. A comparison with
similar state-of-the-art frameworks shows superior performance when handling
dynamic objects and comparable or better performance in the accuracy of the
computed distance and gradient field. Finally, we show how the framework can
be used for fast motion planning in the presence of moving objects. An
accompanying video, code, and datasets are made publicly
available333https://uts-ri.github.io/IDMP.

###### Index Terms:

Interactive Mapping and Planning, Euclidean Distance Fields, Gaussian Process,
Mapping, Motion Planning, Human-Robot Collaboration.

##  I Introduction

Human-robot collaboration (HRC) and other applications of robots in the field
call for interactive representations to deal with dynamic and evolving scenes.
For true collaboration in industrial settings, humans and robots physically
share the same space, e.g. working jointly and simultaneously on the assembly
of a product. The environment consisting of the human, robot, workspace, and
objects such as tools and assembly components needs to be monitored by sensors
and kept up-to-date in a timely manner. One approach is to create and
continuously update an interactive representation of the changing scene which
can be used for safe and effective robot motion planning without colliding
with obstacles and human operators.

Figure 1: Interactive generation of a distance and gradient field in an HRC
setting. Top right: The depth image from an Intel Realsense camera. Bottom
left: Coloured point cloud map of the scene and one horizontal slice of the
field (red/blue means close/far to an object, arrows point away from the
closest object).

Recent representations that aim to fulfil some of the above-mentioned
requirements have been proposed in the robotics literature [1, 2, 3].
Euclidean Distance Field (EDF), for instance, has recently become a promising
representation suitable for direct collision-checking. As an example, the
methods in [4, 5, 3] compute the projective or non-projective Truncated Signed
Distance Field (TSDF) first, then propagate from TSDF to the free space for
updating the EDF. Han et al. [1] propose to propagate from an occupancy map
instead of the TSDF to maintain efficiency while preserving accuracy for the
EDF computation. EDF representations also have the ability to dynamically
update the changes via the so-called free space carving method. This method
performs expensive ray-casting and progressive integration of the TSDF to
update the map, resulting in free space that is only gradually cleared when
objects move in the scene.

This paper presents an interactive distance field mapping and planning (IDMP)
framework aimed at dynamic scenes common in human-robot collaboration
scenarios (see Fig. 1 as an example). We propose to build and maintain an up-
to-date distance and gradient field using a Gaussian-Process-based method
capable of integrating depth sensor measurements following our previous work
in [6, 7]. Here, we present a simple and elegant method for dynamic updates
and fusion of our Gaussian Process (GP) distance field by querying distances
and gradients from a temporary latent GP distance field generated with only
the points of the current frame. We name this latent representation the
“Frustum Field”. The Frustum Field is able to deal with the discrepancy in the
distance between surface points that move in the current frame without the
need for ray-tracing. It also allows seamless fusion with previously mapped
distance and gradient field.

To overcome the computational complexity of GP-based approaches, our framework
maintains the fusion and updates in an Octree structure that runs online.
Furthermore, our method implicitly handles dynamic objects via the information
provided by the Frustum Field. The fused representation is continuous allowing
querying of the Euclidean distance to the nearest surface and its gradient at
an arbitrary spatial resolution. This property enables seamless integration
with a gradient-based motion planning framework. We experimentally evaluate
IDMP’s performance in both static and dynamic scenes and benchmark it against
state-of-the-art algorithms.

To summarise, the key contributions of the paper are:

  * •

A novel dynamic update and fusion method for GP-based distance and gradient
fields. This method is based on a latent representation named Frustum Field
that enables dealing with static, moving, and new points.

  * •

An interactive distance field mapping and planning (IDMP) framework that
generates a continuous Euclidean distance and gradient field online, and is
integrated with a gradient-based planner for 3D collision-free navigation.

  * •

An efficient open-sourced ROS-based implementation of the full IDMP framework.
111Code will be made publicly available at https://github.com/UTS-RI/IDMP upon
paper acceptance.

To the best of our knowledge, this work is the first framework that addresses
the close interactivity between perception and motion planning aimed at HRC
applications.

##  II Related Work

EDFs are useful representations for a wide range of tasks which require
distance-to-obstacle information at different points in space. For example,
motion planners such as gradient-based trajectory optimisers [8, 9] utilise
EDFs and their gradients to perturb trajectories smoothly in order to avoid
obstacles. EDFs can be computed by solving the Eikonal equation. A major
challenge in exploiting the Eikonal equation to recover the ESDF is that it is
non-linear and hyperbolic. Therefore, it is difficult to solve directly, and
specialised numerical approaches are often required, including discrete
methods [10] such as fast marching or label-correcting to propagate the
distance field through a grid.

Inspired by [10], Voxblox [4] proposes an approach to compute the ESDF based
on the projective TSDF through a wavefront propagation algorithm. Voxfield [5]
and VDBblox [3] adopt the use of a non-projective TSDF to improve the
accuracy. FIESTA [1] proposes to use an occupancy map instead of TSDF and a
novel data structure to maintain efficiency. VED-EDT uses a distance transform
function and VDB data structure [11, 12] to represent the EDF hierarchically
[13]. These EDF representations both have the capability to dynamically update
the changes via the so-called free space carving method, which updates every
voxel along the grouped rays. This method requires performing expensive ray-
casting and progressive integration and weighting of the TSDF to update the
map, resulting in free space that is only gradually cleared when objects move
in the scene. There are also frameworks that explicitly deal with dynamic
objects [14]. However, the dynamic object handling is decoupled from mapping.
Our framework deals implicitly with dynamic objects and their surfaces are
considered for collision avoidance, thus tightly coupled with the estimated
representation.

Over the last few years, there have been numerous works exploring the
potential of learning approaches such as neural networks for implicit signed
distance representations. In particular, inspired by [15], the authors in [16]
propose to use an implicit neural representation to constrain the learning
through the Eikonal regularisation, which approximates the SDF for points far
away from the surfaces. Recently proposed in [17], the authors learn an
approximation of EDF from Neural Radiance Fields (NeRFs) with occupancy
inference. Similarly, inspired from [16], iSDF [18] is proposed to use a
neural signed distance field for mapping. To enable the capability for dynamic
scenes, D-NeRF considers time as an additional input to the system and splits
the learning process into two main stages: one that encodes the scene into a
canonical space and another that maps this canonical representation into the
deformed scene at a particular time. While these approaches have benefits such
as a continuous representation and lower memory cost, they are expensive to
compute and require powerful high-end hardware with a GPU.

GP-based distance fields are appealing as they can represent complex
environments with non-parametric models [19, 6]. In our previous work, we
produce the continuous distance field by formulating the Gaussian Process via
a latent distance function. The regularised Eikonal equation can be simply
solved by applying the logarithmic transformation to a GP formulation to
approximate the accurate Euclidean distance fields [6, 20]. In [20] a
trajectory optimisation approach for motion planning is implemented for the 2D
case. In this work, we generalise and apply this approach to 3D.

Following the theoretical formulation in [6, 20], we have introduced the so-
called reverting GP distance field [7] that increased the accuracy of the
distance field addressing multiple key problems in robotics with noisy
measurements. A GP distance field has been applied to the active dynamic
mapping framework in [2] to enable accurate distance inferences [21]. However,
[2, 21] requires the inverse depth to formulate the occupancy field for
dynamic updates. In contrast, we present a simple and elegant method for
dynamic updates and fusion of our GP distance field by querying distance and
gradients from a latent GP distance field generated with only the points of
the current frame. Our proposed framework and its implementation is faster
than all our previous works, making it suitable for close-proximity HRC tasks.

In the context of HRC, it is important to have an efficient representation
that informs the planner of a possible collision, in particular, when objects
are moving in the scene. Two recent review papers discuss the state of the art
in safe HRC [22, 23]. Both distinguish between methods that are aimed at
contact detection/mitigation, and collision avoidance. Our approach follows
the latter which is enabled by the interactivity between mapping and motion
planning.

Many obstacle avoidance methods for HRC explicitly track the human operator
using RGB-D cameras [24, 25, 26]. Human limbs are represented using skeletons
[25, 26], bounding spheres [26], cylinders [27] or an occupancy box [28].
Based on the model of the human and the known state of the robot, the control
strategy is often based on discrete modes, e.g. ”autonomous behaviour” [24].
In some HRC control frameworks, the human position is assumed to be known,
e.g. [29]. In contrast to those approaches, our framework does not rely on
semantics enabling workspace monitoring of all types of objects (static and
dynamic) using a continuous field representation. Our efficient map update
allows an online use of standard trajectory optimisation algorithms rather
than discrete control modes. Another non-semantic approach is given in [30,
31]. However, they only compute distances to discrete control points while our
approach is able to recover a continuous field including non-observed areas.

##  III Background

###  III-A Gaussian Process Distance Field

Gaussian Process (GP) [32] is a flexible and probabilistic approach to
modelling distributions over functions, particularly suitable in scenarios
with noisy and sparse measurements. Let us consider a set of noisy input
observations (training points)
$\mathbf{y}=\left\\{y_{j}=f(\mathbf{x}_{j})+\epsilon_{j}\right\\}_{j=1}^{J}\in\mathbb{R}$
taken at locations
$\mathbf{X}=\\{\mathbf{x}_{j}\\}_{j=1}^{J}\in\mathbb{R}^{D}$ and corrupted by
additive Gaussian noise $\epsilon_{j}\sim\mathcal{N}(0,\sigma^{2})$. We model
the unknown distribution $f$ as a GP:

$f\sim\mathcal{GP}(0,k(\mathbf{x},\mathbf{x}^{\prime})),$ (1)

where $k(\mathbf{x},\mathbf{x}^{\prime})$ is the covariance function that
controls the properties of $f$. Suppose we have a set of testing points
$\mathbf{X_{*}}=\\{\mathbf{x_{*}}_{q}\\}_{q=1}^{Q}\in\mathbb{R}^{D}$. At the
testing locations, we can express the joint distribution of the function
values and the observed target values using the kernel function,

$\left[\begin{array}[]{l}\mathbf{y}\\\
\mathbf{f}_{*}\end{array}\right]=\mathcal{N}\left(\mathbf{0},\left[\begin{array}[]{ll}\mathbf{K}(\mathbf{X},\mathbf{X})+\sigma_{f}^{2}\mathbf{I}&\mathbf{K}\left(\mathbf{X},\mathbf{X_{*}}\right)\\\
\mathbf{K}\left(\mathbf{X_{*}},\mathbf{X}\right)&\mathbf{K}\left(\mathbf{X_{*}},\mathbf{X_{*}}\right)\end{array}\right]\right).$
(2)

$\mathbf{f}_{*}$ is the vector of predicted function values at
$\mathbf{X_{*}}$, $\mathbf{I}$ is the identity matrix, $\sigma_{f}^{2}$ is the
variance of observation noise and
$\mathbf{K}\left(\mathbf{X},\mathbf{X}\right)$ is the $J\times J$ covariance
matrix between input points. Thus, by conditioning Eq. (2), the posterior
distribution of $f$ at an arbitrary testing point $\mathbf{x}_{*}$ is given by
$f(\mathbf{x}_{*})\sim\mathcal{N}(\hat{f}\left(\mathbf{x_{*}}\right),\mathbb{V}\hat{f}\left(\mathbf{x_{*}}\right))$,
where the predictive mean and variance are:

$\hat{f}\left(\mathbf{x_{*}}\right)=\mathbf{k}_{\mathbf{x_{*}}\mathbf{X}}\left(\mathbf{K}_{\mathbf{X}\mathbf{X}}+\sigma_{f}^{2}\mathbf{I}\right)^{-1}\mathbf{y},$
(3)
$\displaystyle\mathbb{V}\hat{f}\left(\mathbf{x_{*}}\right)=k_{\mathbf{x}_{*}\mathbf{x}_{*}}-\mathbf{k}_{\mathbf{x_{*}}\mathbf{X}}$
$\displaystyle\left(\mathbf{K}_{\mathbf{X}\mathbf{X}}+\sigma_{f}^{2}\mathbf{I}\right)^{-1}\mathbf{k}_{\mathbf{x_{*}}\mathbf{X}}^{\top}.$
(4)

$\mathbf{k}(\mathbf{x_{*}},\mathbf{X})$ is the vector of covariances between
the $J$ input points and the testing point.
$k\left(\mathbf{x}_{*},\mathbf{x}_{*}\right)$ is the covariance function value
of the testing point.

Let us now briefly introduce the so-called reverting GP distance field
originally presented in [7]. Consider a surface $\mathcal{S}$ in a Euclidean
space $\mathbb{R}^{D}$, and
$\mathbf{X}=\\{\mathbf{x}_{j}\\}_{j=1}^{J}\in\mathbb{R}^{D}$ a set of discrete
observations of $S$ as training points. By modelling the occupancy
$o(\mathbf{x}):\mathbb{R}^{D}\mapsto\mathbb{R}$ of the space with a GP as
$o\sim\mathcal{GP}\left(0,k(\mathbf{x},\mathbf{x}^{\prime})\right)$, it is
possible to infer the surface occupancy $\hat{o}(\mathbf{x}_{*})$ at any
location in the space using Eq. (3). In other words, the covariance kernel
${{k}\left(\mathbf{x},\mathbf{x}^{\prime}\right)}$ is a monotonic function of
the distance $\lVert\mathbf{x}-\mathbf{x}^{\prime}\rVert$ between $\mathbf{x}$
and $\mathbf{x}^{\prime}$.

We arbitrarily define the occupied area to be equal to 1. Therefore,
$\mathbf{y}$ is equal to $\mathbf{1}$ in Eq. (3) and

$\displaystyle\hat{o}(\mathbf{x}_{*})$
$\displaystyle={\mathbf{k}_{\mathbf{x}_{*}\mathbf{X}}}\left({\mathbf{K}_{\mathbf{X}\mathbf{X}}}+\sigma_{o}^{2}\mathbf{I}\right)^{-1}\mathbf{1}.$
(5)

The distance field ${\hat{d}}(\mathbf{x}_{*})$ given any location
$\mathbf{x}_{*}$ in the space is obtained by applying a _reverting_ function
$r$ to the occupancy field as

$\displaystyle{\hat{d}}(\mathbf{x}_{*})=r\left(\hat{o}\left(\mathbf{x}_{*}\right)\right).$
(6)

The reverting function depends on the chosen kernel and corresponds to the
inverse operation, computing the distance as a function of the covariance
between $\mathbf{x}$ and $\mathbf{x}^{\prime}$:

$\displaystyle
r\left(\tilde{{{k}}}(\lVert\mathbf{x}-\mathbf{x}^{\prime}\rVert)\right)$
$\displaystyle=\lVert\mathbf{x}-\mathbf{x}^{\prime}\rVert$ (7)
$\displaystyle\text{with }\
\tilde{{{k}}}(\lVert\mathbf{x}-\mathbf{x}^{\prime}\rVert)$
$\displaystyle={{k}\left(\mathbf{x},\mathbf{x}^{\prime}\right)}.$

Considering the square exponential kernel
${{k}\left(\mathbf{x},\mathbf{x}^{\prime}\right)}=\sigma^{2}\exp\left(-\frac{\lVert\mathbf{x}-\mathbf{x}^{\prime}\rVert{}^{2}}{2l^{2}}\right)$,
substituting the reverting function of square exponential kernel into Eq. 6
gives us:

$\displaystyle{\hat{d}}(\mathbf{x}_{*})=\sqrt{-2l^{2}\log\left(\frac{\hat{o}\left(\mathbf{x}_{*}\right)}{\sigma^{2}}\right)}.$
(8)

This method assumes that the GP-based occupancy field in $\mathbb{R}^{D}$
behaves similarly to the one-noiseless-point scenario where the reverting
function yields the exact distance to the data point.

###  III-B Linear Operator for Gradient Field

The derivative of a GP is another GP. One common solution to infer the
gradient field is to have surface normals as input for GP modelling. However,
this requires inverting a computational-heavy joint covariance function of
$k(\mathbf{x},\mathbf{x}^{\prime})$ with the partial derivatives of
$k(\mathbf{x},\mathbf{x}^{\prime})$ at $\mathbf{x}$ and $\mathbf{x}^{\prime}$
[33]. Instead, we propose to use the linear operator to infer the gradient
along with the accurate distance field without the normal as input. Applying
the linear operator to Eq. 5, we have

$\displaystyle\nabla\hat{o}\left(\mathbf{x}_{*}\right)=\nabla\mathbf{k}_{\mathbf{x_{*}}\mathbf{X}}\left(\mathbf{K}_{\mathbf{X}\mathbf{X}}+\sigma_{o}^{2}\mathbf{I}\right)^{-1}\mathbf{1},$
(9)

where $\nabla\mathbf{k}(\mathbf{x_{*}}\mathbf{X})$ is the partial derivative
of $\mathbf{x}$ (gradient matrix) and $\nabla
k_{\mathbf{x}_{*}\mathbf{x}_{*}}\nabla$ is the second-order partial
derivatives of $\mathbf{x}$ and $\mathbf{x}^{\prime}$ (Hessian matrix). Taking
the gradient of both sides of Eq. 8 with respect to the distance, the gradient
of ${\hat{d}}(\mathbf{x}_{*})$ is in the same direction as the gradient of
$\hat{o}(\mathbf{x}_{*})$, subject to a scaling factor, so that
$\nabla{\hat{d}}(\mathbf{x}_{*})\approx\nabla\hat{o}(\mathbf{x}_{*})$.

Figure 2:  System diagram of IDMP. We first model the temporary latent Frustum
Field (blue) using only $\mathcal{P}_{\\{i\\}}$ as training points. All prior
training points $\mathcal{P}_{\\{0,...,i-1\\}}$ in the Fused Field (yellow)
are then passed to the Frustum Field. Given the sensor pose, the Frustum Field
selects from $\mathcal{P}_{\\{0,...,i-1\\}}$ the points that are within the
frustum area $\mathcal{P}_{f\\{0,...,i-1\\}}$ and returns the inferred values
of $\hat{d}_{f}$, $\nabla\hat{d}_{f}$ to the Fused Field. These distances and
gradients are used to perform fusion and dynamic update by updating the
training points that model the Fused Field. The path planner then queries for
$\hat{d}_{o}$, $\nabla\hat{d}_{o}$ in order to compute and adapt its motion
plans in response to a changing map.

##  IV Proposed Framework

An overview of the proposed IDMP framework is shown in Fig. 2. Given data from
one or multiple depth sensors as input, IDMP incrementally builds the Fused
Field, a continuous GP distance and gradient field for the motion planner to
query and adapt its paths in response to a changing map. The measurements in
the sensor frame ${C}_{{i}}$ at current time $i$ are denoted as point cloud
${}^{{C}_{{i}}}\mathcal{P}_{\\{i\\}}$. This raw point cloud is projected from
current sensor frame ${{C}_{{i}}}$ to world frame ${W}$ given a transformation
matrix $\mathbf{T}_{{C}_{{i}}}$ yielding a point cloud $\mathcal{P}_{\\{i\\}}$
in the world reference frame. In the first process (blue block), we model a
temporary latent GP distance and gradient field only using
$\mathcal{P}_{\\{i\\}}$ as training points. This field only covers the region
in the sensor’s field of view, thus we call it Frustum Field. This latent
model allows us to query the distance and gradients with respect to the
current measurements. Note that the Frustum Field is rebuilt at every frame
$i$.

For the next process of IDMP (yellow block), we build and maintain our main
representation, the Fused Field, generated with all prior training points
$\mathcal{P}_{\\{0,...,i-1\\}}$. We pass $\mathcal{P}_{\\{0,...,i-1\\}}$ to
the Frustum Field to select points only within the frustum area, denoted as
$\mathcal{P}_{f\\{0,...,i-1\\}}$. Then we use $\mathcal{P}_{f\\{0,...,i-1\\}}$
to query the Frustum Field for the distance $\hat{d}_{f}$ and gradient
$\nabla\hat{d}_{f}$ as per Eq. (8). Based on the distance and gradient
inference, the fusion method adjusts the training points’ positions, and the
dynamic update method eliminates past training points of the objects moving in
the scene. We call the intersection between $\mathcal{P}_{\\{0,...,i-1\\}}$
and $\mathcal{P}_{\\{i\\}}$ the overlapping area. Note that the new
measurements that are out of the range of the overlapping area, denoted as
$\mathcal{P}_{n\\{i\\}}$, are directly merged into the Fused Field. By
updating the training points, which are maintained via an Octree-based data
structure, we implicitly update the Fused Field.

For planning, let us define workspace points along the robot’s links as
$\mathcal{P}_{o}$. The motion planning algorithm then queries the Fused Field
for $\hat{d}_{o}$, $\nabla\hat{d}_{o}$ at the locations $\mathcal{P}_{o}$.
These distances and gradients are used to compute a safe trajectory and can be
repeatedly queried to adapt to a changing map.

###  IV-A Frustum Field

Every raw point transformed to the world frame $\mathcal{P}_{\\{i\\}}$ is used
to generate the Frustum Field. Note that the Frustum Field is temporary and
therefore trained and regenerated at every frame $i$ using the current
measurements. As explained in Sec. III, we use $\mathcal{P}_{\\{i\\}}$ as a
set of discrete observations $\mathbf{X}=\\{\mathbf{x}_{j}\\}_{j=1}^{J}$ of
surface $S$. We then model the occupancy via Eq. 5 and apply the reverting
function to obtain the distance field. The Frustum Field is then inferred via
Eq. 8. By using the linear operator for gradient inference, we employ Eq. 9 to
infer the gradients along with the distance information. After the Frustum
Field is modelled, we can query the distance and gradient from the points
$\mathcal{P}_{f\\{0,...,i-1\\}}$ to pass them into the next process, the Fused
Field, for the interactive updates.

Figure 3: A simplified illustration of the proposed method. (1) Before the
update, we have the prior training points $\mathcal{P}_{\\{0,...,i-1\\}}$ in
the Fused Field. (2) The coloured area is the Frustum Field (blue is close to
the surface and yellow is far from it). The current points
$\mathcal{P}_{\\{i\\}}$ are shown as red and yellow crosses. The prior
training points within the frustum $\mathcal{P}_{f\\{0,...,i-1\\}}$ are marked
by red and pink. Fusion, dynamic update, and insertion processes are shown in
red, pink, and yellow respectively. (3) After the update, we have the updated
training points $\mathcal{P}_{\\{0,...,i\\}}$ that model the Fused Field used
by the planner.

###  IV-B Fused Field

The training points are used to generate the Fused Field incrementally. Before
the update at frame $i$, the Fused Field is modelled based on the prior
training points $\mathcal{P}_{\\{0,...,i-1\\}}$ from frame $0$ to $i-1$. After
fusion and dynamic update to allow merging $\mathcal{P}_{\\{i\\}}$ with
$\mathcal{P}_{\\{0,...,i-1\\}}$, the Fused Field is modelled using the updated
training points $\mathcal{P}_{\\{0,...,i\\}}$. There are three key processes
performed at every frame: adjusting points for fusion, removing points for
dynamic update, and inserting points for measurements in new areas. Fig. 3
illustrates the way the points are selected given the Frustum Field. The top
figure shows the points $\mathcal{P}_{\\{0,...,i-1\\}}$ before the update
given frame $i$. The middle figure shows the Frustum Field and the points
involved in the three processes as described in the previous section. Red and
yellow crosses are the new measurements $\mathcal{P}_{\\{i\\}}$ captured by
the sensor. The coloured background is the Frustum Field modelled using
$\mathcal{P}_{\\{i\\}}$. The blue area means the distance is close to the
surface, and the yellow area is far away from the surface. Circles coloured in
red and pink are the prior training points of the Fused Field only in the
frustum, denoting as $\mathcal{P}_{f\\{0,...,i-1\\}}$. Note that we use the
sensor pose to select
$\mathcal{P}_{f\\{0,...,i-1\\}}\subset\mathcal{P}_{\\{0,...,i-1\\}}$. We then
query the Frustum Field to infer $\hat{d}_{f}$ and gradient
$\nabla\hat{d}_{f}$ to perform the fusion and dynamic update. An advantage of
this approach is that we only query the Frustum Field once to perform the
three further processes described as follows.

####  IV-B1 Fusion

Let us denote the prior training points within the frustum as
$\mathcal{P}_{a\\{0,...,i-1\\}}\subset\mathcal{P}_{f\\{0,...,i-1\\}}$ (red
circles in Fig. 3) and corresponding distances and gradients as $\hat{d}_{a}$,
$\nabla\hat{d}_{a}$. A threshold on the distance value $\eta$ is then used to
indicate if $\mathcal{P}_{a\\{0,...,i-1\\}}$ are relatively close to the
surface given the current Frustum Field. For each point
$\mathbf{p}_{a\\{0,...,i-1\\}}$ in $\mathcal{P}_{a\\{0,...,i-1\\}}$, the
proposed fusion is performed by updating its position through the distance to
the surface $\hat{d}_{a}$ in the direction of the gradient $\nabla\hat{d}_{a}$
as:

$\hat{\mathbf{p}}_{a}=\mathbf{p}_{a}-\hat{d}_{a}\nabla\hat{d}_{a}.$ (10)

Note that $\hat{d}_{a}$ and $\nabla\hat{d}_{a}$ are the result of simply
querying the Frustum Field. No data association nor iteration is required as
GP distance inference produces directly the distance to the surface and it is
accurate through our reverting GP model. After the fusion,
$\mathcal{P}_{a\\{0,...,i-1\\}}$ is replaced by
$\hat{\mathcal{P}}_{a\\{0,...,i-1\\}}$ as the training point of the Fused
Field. Note that since we have $\hat{\mathcal{P}}_{a\\{0,...,i-1\\}}$, we do
not need to include new points in the overlapping area (red crosses) into the
Fused Field.

####  IV-B2 Dynamic Update

We show the dynamic update in Fig. 3. Dynamic objects are handled based on the
information provided by the Frustum Field as well, as we only update the
moving objects in the current field of view. Let us denote the prior training
points to-be-removed as
$\mathcal{P}_{m\\{0,...,i-1\\}}\subset\mathcal{P}_{f\\{0,...,i-1\\}}$ (pink
circles). Note that in this case and without loss of generality, we can
capture measurements behind $\mathcal{P}_{f\\{0,...,i-1\\}}$ represents that
the points $\mathcal{P}_{f\\{0,...,i-1\\}}$ have been removed and are not in
the scene anymore. The Frustum Field distance values $\hat{d}_{m}$ for these
points are bigger than $\eta$ to indicate they are relatively far away from
the current captured measurements, which means the object has moved. We then
eliminate from the training set the removed points
$\mathcal{P}_{m\\{0,...,i-1\\}}$ of the Fused Field. Note that in some cases,
the points to adjust and remove are very close to each other and may not be
separated explicitly.

####  IV-B3 New Points Insertion

Let us denote the points in $\mathcal{P}_{\\{i\\}}$ that are outside of the
overlapping area as $\mathcal{P}_{n\\{i\\}}$ (yellow crosses). These points
are directly added to the Fused Field training set. Points outside the
overlapping area in the Fused Field remain the same (grey circles). Following
our illustration, at the bottom figure in Fig. 3 after the update, we use the
updated training points $\mathcal{P}_{\\{0,...,i\\}}$ to model the Fused Field
and have it ready for the motion planner query.

We only use one Frustum Field for all interactive updates. The Frustum Field
allows to perform the fusion in just one-step and to clear the removed object
effectively.

###  IV-C Motion Planning

Due to the differentiability of the Fused Field, our framework naturally
accommodates gradient-based motion planners. For example, the motion planner
can query $\hat{d}_{o}$ and $\nabla\hat{d}_{o}$ and utilise these to adapt its
motion plans in response to a changing map. A key advantage of our framework
is that we can use the gradient inference in Eq. 9 to compute the gradients
analytically without surface normals, whereas conventional approaches rely on
discrete grid-based approximation [8, 34]. This results in faster and more
accurate computation of gradients which is crucial for close-proximity human-
robot collaborative tasks where fast replanning is required.

##  V Evaluation

To evaluate the proposed IDMP framework, we a) quantitatively compare the
mapping performance for both static and dynamic scenes with other frameworks,
and b) qualitatively demonstrate online motion planning for a dynamic scene.

###  V-A Distance Field Mapping

We compare IDMP to the two state-of-the-art algorithms: FIESTA [1] and Voxblox
[4]. Both are designed to compute the distance field observed by a 3D sensor
online. One key difference from both approaches to IDMP is the need to specify
a fixed voxel size upfront which directly influences performance. In contrast,
IDMP uses a continuous function to represent the field that can be queried at
any point in space. We also demonstrate IDMP’s ability to handle dynamic
scenes with unknown moving objects and compare the performance to
FIESTA/Voxblox.

The evaluation is conducted on two distinct datasets: a) the widely used Cow &
Lady dataset [4], and b) a custom synthetic dataset featuring a dynamically
moving ball on a table. These datasets are chosen to assess IDMP’s accuracy in
static and dynamic scenarios relative to current state-of-the-art algorithms.
The Cow & Lady dataset features complex surface geometries of static objects
in a room recorded by a moving Kinect-1 RGB-D camera whose pose is tracked by
a vicon system. The ground truth surface reconstruction is obtained by a Leica
TPS-M50 laser scanner. The second dataset is used for evaluating a dynamic
scene. It is generated using Gazebo and features a ball rolling on a table
observed by a simulated RGB-D camera. The ground truth for distances and
gradients are obtained from Gazebo.

The experiments are conducted on a laptop equipped with an AMD Ryzen 7 4800u
CPU, 16GB RAM and no GPU. All three algorithms are allowed to multithread on
up to 16 threads. Each algorithm is allowed to run as a ROS node with only the
playback of the dataset (rosbag) running in the background. Original code
implementations for Fiesta222https://github.com/HKUST-Aerial-Robotics/FIESTA
and Voxblox333https://github.com/ethz-asl/voxblox were used for the
experiments with default parameters.

Root Mean Squared Error (RMSE) is used to evaluate the distance field
performance. For gradients, we use

$\text{cosine\\_similarity}(\mathbf{a},\mathbf{b})=\frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\cdot\|\mathbf{b}\|}$
(11)

to measure the difference between two gradient vectors $\mathbf{a}$ and
$\mathbf{b}$ in terms of their direction.

###  V-B Mapping Results

####  V-B1 Static Scene

Figure 4: Example of an IDMP query on a horizontal plane for the Cow & Lady
dataset. The colour of the arrows represents the distance to the nearest
object. The arrows are normalised gradients that point in the direction away
from the nearest object. The updated training points are coloured using RGB
data from the camera.

Fig. 1 and Fig. 4 present qualitative results where IDMP queries are
visualised. For every point in space, a distance (colour) and direction
(arrow) away from the closest obstacle can be computed online and on-demand.
For both examples, the queries are taken using a regular grid on a plane
parallel to the table and floor, respectively. Note that because of the
continuous nature of IDMP, the query points can be located anywhere in space
and are neither constrained to be on a plane or a regular grid nor bound to a
specific spatial resolution.

(a) Ground truth

(b) IDMP output

(c) IDMP error

Figure 5: Evaluation of the IDMP distance field accuracy using a volume of
size $4m$ x $2.75m$ x $1.25m$ from the Cow & Lady dataset. Queries are made at
a regular grid every $5cm$.

Fig. 5 illustrates the true distance field, the IDMP output, and the absolute
error of IDMP on three perpendicular planes on the Cow & Lady dataset. The
planes are part of a volume that contains a mixture of complex surfaces but
also free space with no obstacles.

(a)

(b)

Figure 6: Evaluation of distances and gradients using the Cow & Lady dataset.
IDMP is compared to FIESTA and Voxblox. The evaluated volume is the same as in
Fig. 5.

To compare IDMP’s distance field accuracy to Voxblox and FIESTA, we compute
the RMSE for each framework while varying the spatial resolution. For
Voxblox/FIESTA, this corresponds to the voxel size. For IDMP we query at a
regular grid that corresponds to the centre of each voxel.

Figure LABEL:fig:cow_rmse shows our experimental results. The accuracy for
Voxblox and FIESTA decreases with increasing spatial resolution. In contrast,
IDMP’s accuracy remains constant and there is no requirement to pre-set a
certain resolution illustrating the advantage of using a continuous
representation.

The evaluation of gradients is presented in Fig. LABEL:fig:cow_cos. This
figure shows the mean of the cosine similarities between the computed
gradients and the ground truth. Both IDMP and FIESTA perform better than
Voxblox for all voxel sizes and resolutions. FIESTA shows slightly better
performance than IDMP in some cases; however, at the cost of higher
variability. In terms of computation time for IDMP, updating the GP takes on
average $250ms$ per frame for this scene while the query step takes $2\mu s$
per point.

####  V-B2 Dynamic Scene

In this experiment, we evaluate how each framework handles dynamic scenes with
moving objects. Fig. 7(a) shows the Gazebo scene with a ball rolling on the
table from left to right. The last three subfigures show the surface points at
the end of the run for the three frameworks. Both FIESTA and Voxblox show
artefacts due to the progressive weighting and integration while IDMP does
not. IDMP results in the best RMSE of $2.6cm$ which is more than a factor of
$2$ better than FIESTA/Voxblox. Updating the GP for a scene of this size takes
on average $50ms$ per frame while the query step takes $2\mu s$ per point.
Only the time for processing the frame is dependent on the size of the scene
whereas the time needed for the query remains constant.

(a) Simulated scene

(b) IDMP

(c) FIESTA

(d) Voxblox

Figure 7: Comparison of IDMP with FIESTA and Voxblox for a dynamic scene of
size $0.39m$ x $1.05m$ x $0.35m$. (a) shows the setup in simulation while (b)
to (d) show the fused surface points at the end of the run. Colours indicate
the height of the points. The spatial resolution is $1cm$. IDMP handles moving
objects seamlessly while FIESTA and Voxblox keep artefacts from older frames.

###  V-C Motion Planning in Dynamic Scenes

In this experiment we demonstrate that our method is able to facilitate safe
motion planning of a robot arm in the presence of dynamic obstacles which is
useful for close-proximity human-robot collaboration tasks. To simulate such a
scenario we task the robot with moving to a goal pose while avoiding a moving
ball object, see Fig. 8.

Trajectory optimisers are well suited to such scenarios due to their fast
computation and smooth trajectory properties. For our experiment we utilise
the Covariant Hamiltonian optimisation-based motion planner (CHOMP) [8], a
gradient-based trajectory optimisation method that iteratively perturbs a
given initial trajectory away from obstacles. To do so, it utilises the
gradient field to minimise its optimisation objective which consists of
trajectory smoothness and obstacle avoidance terms. Due to a) the ability of
our framework to compute updated gradients analytically and b) the need to
only query the points $\mathcal{P}_{o}$ the motion planner requires, we can
continuously replan online. The computation time for the query shown in Fig. 8
is approximately $6.4ms$ ($3190$ points at $2\mu s/$point).

(a) Original planned trajectory (no obstacle encountered).

(b) Modified trajectory due to dynamic obstacle.

Figure 8: Re-planning in the presence of a dynamic obstacle. CHOMP
approximates the manipulator via $29$ spheres. For the trajectory shown, $110$
points per sphere are used, resulting in $29*110=3190$ query points.

##  VI Conclusion

In this letter, we presented an efficient interactive mapping and planning
framework named IDMP. The simple and elegant dynamic updates and fusion are
performed by querying distances and gradients from a temporary latent GP
distance field generated with only the points of the current frame. Our
framework can run online on a modest PC without a GPU and handles dynamic
objects effectively. The fused representation is continuous enabling queries
of the Euclidean distance with gradient at an arbitrary spatial resolution.
This property enables seamless integration with a gradient-based motion
planning framework for HRC applications. Going forward, we aim to further
explore IDMP’s capabilities, such as multi-resolution sampling and mapping
uncertainty, for enhancing motion planning, particularly in HRC applications.

##  VII Acknowledgement

The authors would like to thank Cedric Le Gentil for useful discussions and
code collaboration.

GP

    Gaussian Process
GPIS

    Gaussian Process Implicit Surfaces
CHOMP

    Covariant Hamiltonian Optimization for Motion Planning
EDF

    Euclidean Distance Field
EDFs

    Euclidean Distance Fields
TSDF

    Truncated Signed Distance Field
ESDF

    Euclidean Signed Distance Field

## References

  * [1] L. Han, F. Gao, B. Zhou, and S. Shen, “FIESTA: Fast Incremental Euclidean Distance Fields for Online Motion Planning of Aerial Robots,” 2019.
  * [2] L. Liu, S. Fryc, L. Wu, T. L. Vu, G. Paul, and T. Vidal-Calleja, “Active and Interactive Mapping With Dynamic Gaussian Process Implicit Surfaces for Mobile Manipulators,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 3679–3686, Apr. 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9361306/
  * [3] Y. Bai, Z. Miao, X. Wang, Y. Liu, H. Wang, and Y. Wang, “Vdbblox: Accurate and efficient distance fields for path planning and mesh reconstruction,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 7187–7194.
  * [4] H. Oleynikova, Z. Taylor, M. Fehr, R. Siegwart, and J. Nieto, “Voxblox: Incremental 3D Euclidean Signed Distance Fields for on-board MAV planning,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. Vancouver, BC: IEEE, Sep. 2017, pp. 1366–1373. [Online]. Available: http://ieeexplore.ieee.org/document/8202315/
  * [5] Y. Pan, Y. Kompis, L. Bartolomei, R. Mascaro, C. Stachniss, and M. Chli, “Voxfield: Non-Projective Signed Distance Fields for Online Planning and 3D Reconstruction,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. Kyoto, Japan: IEEE, Oct. 2022, pp. 5331–5338. [Online]. Available: https://ieeexplore.ieee.org/document/9981318/
  * [6] L. Wu, K. M. B. Lee, L. Liu, and T. Vidal-Calleja, “Faithful Euclidean Distance Field from Log-Gaussian Process Implicit Surfaces,” Jan. 2021, arXiv:2010.11487 [cs]. [Online]. Available: http://arxiv.org/abs/2010.11487
  * [7] C. L. Gentil, O.-L. Ouabi, L. Wu, C. Pradalier, and T. Vidal-Calleja, “Accurate Gaussian Process-based Distance Fields with applications to Echolocation and Mapping,” Sep. 2023, arXiv:2302.13005 [cs] version: 2. [Online]. Available: http://arxiv.org/abs/2302.13005
  * [8] M. Zucker, N. Ratliff, A. D. Dragan, M. Pivtoraiko, M. Klingensmith, C. M. Dellin, J. A. Bagnell, and S. S. Srinivasa, “CHOMP: Covariant hamiltonian optimization for motion planning,” _International Journal of Robotics Research_ , vol. 32, no. 9-10, pp. 1164–1193, 2013.
  * [9] M. Mukadam, J. Dong, X. Yan, F. Dellaert, and B. Boots, “Continuous-time gaussian process motion planning via probabilistic inference,” _International Journal of Robotics Research_ , vol. 37, no. 11, pp. 1319–1340, 2018.
  * [10] B. Lau, C. Sprunk, and W. Burgard, “Improved updating of euclidean distance maps and voronoi diagrams,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 281–286.
  * [11] K. Museth, J. Lait, J. Johanson, J. Budsberg, R. Henderson, M. Alden, P. Cucka, D. Hill, and A. Pearce, “Openvdb: an open-source data structure and toolkit for high-resolution volumes,” in _Acm siggraph 2013 courses_ , 2013, pp. 1–1.
  * [12] K. Museth, “Vdb: High-resolution sparse volumes with dynamic topology,” _ACM transactions on graphics (TOG)_ , vol. 32, no. 3, pp. 1–22, 2013.
  * [13] D. Zhu, C. Wang, W. Wang, R. Garg, S. Scherer, and M. Q.-H. Meng, “VDB-EDT: An Efficient Euclidean Distance Transform Algorithm Based on VDB Data Structure,” May 2021. [Online]. Available: https://arxiv.org/abs/2105.04419v1
  * [14] L. Schmid, O. Andersson, A. Sulser, P. Pfreundschuh, and R. Siegwart, “Dynablox: Real-time detection of diverse dynamic objects in complex environments,” _IEEE Robotics and Automation Letters_ , 2023.
  * [15] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 165–174.
  * [16] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman, “Implicit geometric regularization for learning shapes,” _arXiv preprint arXiv:2002.10099_ , 2020.
  * [17] M. Pantic, C. Cadena, R. Siegwart, and L. Ott, “Sampling-free obstacle gradients and reactive planning in neural radiance fields (nerf),” _arXiv preprint arXiv:2205.01389_ , 2022.
  * [18] J. Ortiz, A. Clegg, J. Dong, E. Sucar, D. Novotny, M. Zollhoefer, and M. Mukadam, “isdf: Real-time neural signed distance fields for robot perception,” _arXiv preprint arXiv:2204.02296_ , 2022.
  * [19] B. Lee, C. Zhang, Z. Huang, and D. D. Lee, “Online continuous mapping using gaussian process implicit surfaces,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019.
  * [20] L. Wu, K. M. B. Lee, C. Le Gentil, and T. Vidal-Calleja, “Log-GPIS-MOP: A Unified Representation for Mapping, Odometry, and Planning,” _IEEE Transactions on Robotics_ , pp. 1–17, 2023. [Online]. Available: https://ieeexplore.ieee.org/document/10202666/
  * [21] L. Wu, C. Le Gentil, and T. Vidal-Calleja, “Pseudo Inputs Optimisation for Efficient Gaussian Process Distance Fields,” 2023.
  * [22] S. Robla-Gomez, V. M. Becerra, J. R. Llata, E. Gonzalez-Sarabia, C. Torre-Ferrero, and J. Perez-Oria, “Working Together: A Review on Safe Human-Robot Collaboration in Industrial Environments,” _IEEE Access_ , vol. 5, pp. 26 754–26 773, 2017. [Online]. Available: https://ieeexplore.ieee.org/document/8107677/
  * [23] S. Patil, V. Vasu, and K. V. S. Srinadh, “Advances and perspectives in collaborative robotics: a review of key technologies and emerging trends,” vol. 2, no. 1, p. 13. [Online]. Available: https://doi.org/10.1007/s44245-023-00021-8
  * [24] A. M. Zanchettin, N. M. Ceriani, P. Rocco, H. Ding, and B. Matthias, “Safety in human-robot collaborative manufacturing environments: Metrics and control,” _IEEE Transactions on Automation Science and Engineering_ , vol. 13, no. 2, pp. 882–893, 2016.
  * [25] F. Schirmer, P. Kranz, J. Schmitt, and T. Kaupp, “Anomaly detection for dynamic human-robot assembly: Application of an lstm-based autoencoder to interpret uncertain human behavior in hrc,” in _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 881–883.
  * [26] C. Morato, K. N. Kaipa, B. Zhao, and S. K. Gupta, “Toward Safe Human Robot Collaboration by Using Multiple Kinects Based Real-Time Human Tracking,” _Journal of Computing and Information Science in Engineering_ , vol. 14, no. 1, p. 011006, Mar. 2014. [Online]. Available: https://asmedigitalcollection.asme.org/computingengineering/article/doi/10.1115/1.4025810/370115/Toward-Safe-Human-Robot-Collaboration-by-Using
  * [27] F. Schirmer, P. Kranz, B. Bhat, C. Rose, J. Schmitt, and T. Kaupp, “Towards a path planning and communication framework for seamless human-robot assembly,” in _19th Annual ACM/IEEE International Conference on Human-Robot Interaction_.
  * [28] C. Tonola, M. Faroni, N. Pedrocchi, and M. Beschi, “Anytime informed path re-planning and optimization for human-robot collaboration,” in _2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)_. IEEE, 2021, pp. 997–1002.
  * [29] M. Faroni, M. Beschi, and N. Pedrocchi, “An MPC Framework for Online Motion Planning in Human-Robot Collaborative Tasks,” in _2019 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_. Zaragoza, Spain: IEEE, Sep. 2019, pp. 1555–1558. [Online]. Available: https://ieeexplore.ieee.org/document/8869047/
  * [30] F. Flacco, T. Kroeger, A. De Luca, and O. Khatib, “A Depth Space Approach for Evaluating Distance to Objects,” _Journal of Intelligent & Robotic Systems_, vol. 80, no. 1, pp. 7–22, Dec. 2015. [Online]. Available: https://doi.org/10.1007/s10846-014-0146-2
  * [31] F. Flacco and A. De Luca, “Real-Time Computation of Distance to Dynamic Obstacles With Multiple Depth Sensors,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 1, pp. 56–63, Jan. 2017. [Online]. Available: http://ieeexplore.ieee.org/document/7421962/
  * [32] C. E. Rasmussen and C. K. Williams, _Gaussian Processes for Machine Learning_. Cambridge, Mass.: MIT Press, 2006.
  * [33] W. Martens, Y. Poffet, P. R. Soria, R. Fitch, and S. Sukkarieh, “Geometric priors for gaussian process implicit surfaces,” _IEEE Robotics and Automation Letters (RA-L)_ , pp. 373–380, 2017.
  * [34] H. Oleynikova, Z. Taylor, R. Siegwart, and J. Nieto, “Safe Local Exploration for Replanning in Cluttered Unknown Environments for Microaerial Vehicles,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 1474–1481, Jul. 2018. [Online]. Available: http://ieeexplore.ieee.org/document/8276241/

  *[EDF]: Euclidean Distance Field
  *[TSDF]: Truncated Signed Distance Field