# Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based
Contextual Cues

Tassadaq Hussain  Kia Dashtipour  Yu Tsao  and Amir Hussain Tassadaq Hussain
is with the School of Computing, Edinburgh Napier University, UK. (e-mail:
t.hussain@napier.ac.uk).Kia Dashtipour is with the School of Computing,
Edinburgh Napier University, UK. (e-mail: k.dashtipour@napier.ac.uk).Yu Tsao
is with the Center for Information Technology and Innovation, Academia Sinica,
Taiwan. (e-mail: yu.tsao@citi.sinica.edu.tw).Amir Hussain is with the School
of Computing, Edinburgh Napier University, UK. (e-mail:
a.hussain@napier.ac.uk).This paragraph will include the Associate Editor who
handled your paper.

###### Abstract

In real-world environments, background noise significantly degrades the
intelligibility and clarity of human speech. Audio-visual speech enhancement
(AVSE) attempts to restore speech quality, but existing methods often fall
short, particularly in dynamic noise conditions. This study investigates the
inclusion of emotion as a novel contextual cue within AVSE, hypothesizing that
incorporating emotional understanding can improve speech enhancement
performance. We propose a novel emotion-aware AVSE system that leverages both
auditory and visual information. It extracts emotional features from the
facial landmarks of the speaker and fuses them with corresponding audio and
visual modalities. This enriched data serves as input to a deep UNet-based
encoder-decoder network, specifically designed to orchestrate the fusion of
multimodal information enhanced with emotion. The network iteratively refines
the enhanced speech representation through an encoder-decoder architecture,
guided by perceptually-inspired loss functions for joint learning and
optimization. We train and evaluate the model on the CMU Multimodal Opinion
Sentiment and Emotion Intensity (CMU-MOSEI) dataset, a rich repository of
audio-visual recordings with annotated emotions, specifically chosen for its
diversity and real-world scenarios. Our comprehensive evaluation demonstrates
the effectiveness of emotion as a contextual cue for AVSE. By integrating
emotional features, the proposed system achieves significant improvements in
both objective and subjective assessments of speech quality and
intelligibility, especially in challenging noise environments. Compared to
baseline AVSE and audio-only speech enhancement systems, our approach exhibits
a noticeable increase in PESQ (from 1.43 to 1.67) and STOI (from 0.70 to
0.74), indicating higher perceptual quality and intelligibility. Large-scale
listening tests corroborate these findings, suggesting improved human
understanding of enhanced speech.

Audio-Visual Speech Enhancement (AVSE), Deep Learning, Human-Computer
Interaction (HCI), Emotion Recognition, Speech Quality

##  1 Introduction

Speech enhancement (SE) is a critical area in signal processing, primarily
driven by the imperative to improve the quality and intelligibility of spoken
communication. Despite extensive efforts in this field, conventional SE
algorithms have often fallen short in effectively addressing the challenges
posed by real-world noisy environments. Traditional techniques frequently
struggle to disentangle desired speech from background noise, especially in
the presence of non-stationary or highly unpredictable acoustic interferences.
Their limited capacity to differentiate between speech and noise sources has
led to a quest for more robust solutions [6] [19] [23] [4].

The emergence of deep learning (DL)-based approaches has revolutionized the
landscape of SE [18] [34]. These methodologies leverage the power of neural
networks to learn complex patterns and relationships, enabling them to surpass
their conventional counterparts. DL models, particularly convolutional and
recurrent neural networks, have displayed a remarkable capability to enhance
the quality and intelligibility of speech in noisy conditions [35] [37] [35].
However, despite the significant advancements in DL-based SE, the challenges
posed by specific scenarios, such as the presence of multiple speakers and
challenging acoustic environments, remained unaddressed. This limitation
prompted the exploration of audio-visual speech enhancement (AVSE) systems,
which harness both auditory and visual cues to enhance speech signals.  The
integration of visual information, such as lip movements and facial landmarks,
alongside audio data, has offered promising results in mitigating the impact
of noise and enhancing the quality of spoken communication [1]. AVSE systems,
by jointly processing auditory and visual signals, can provide superior
performance in scenarios where traditional audio-only approaches fall short,
marking a significant step forward in the pursuit of intelligible and noise-
resistant speech [2] [26] [21] [5] [8]. Similarly, researchers have taken
substantial steps to enhance the performance of both Audio-only SE (ASE) and
AVSE systems, especially in the presence of extreme noise conditions. An
innovative approach involves the utilization of distinct loss functions,
particularly distance-based and perceptually-inspired ones, to optimize speech
quality and intelligibility. These loss functions take into account perceptual
aspects of speech quality [15], notably the Perceptual Evaluation of Speech
Quality (PESQ) and Short-Time Objective Intelligibility (STOI), and are
seamlessly integrated into the training process [11] [17] [14] [13]. This
integration ensures that the ASE and AVSE system closely aligns with human
auditory perception, resulting in a marked improvement in speech signal
quality and intelligibility, even in the face of challenging and dynamic noise
environments.  In recent years, a notable paradigm shift has been observed in
the domain of affective computing. Researchers have explored innovative
approaches that effectively leverage both regression and classification
systems to achieve remarkable performance enhancements. A prominent trend in
this context involves the integration of SE systems as crucial pre-processing
components to significantly mitigate noise interference, consequently leading
to a substantial improvement in the accuracy of emotion recognition systems.
One significant development in this direction is the combined use of
regression systems, particularly SE models, as preliminary stages in the
pipeline for classification systems focused on emotion recognition. This novel
approach has demonstrated exceptional potential and garnered substantial
attention in the research community. By employing SE as a pre-processing step,
researchers aim to create highly robust and context-aware solutions for
affective computing, particularly in challenging, real-world scenarios where
environmental noise can significantly affect the accuracy of emotion
detection. For example, this strategy is exemplified in recent studies such as
Triantafyllopoulos et al. [32], who introduced a novel framework that
effectively utilizes SE techniques to preprocess audio data before its
classification for emotion recognition. Their research showcases the potential
of SE as a valuable tool for enhancing emotion detection accuracy in noisy
environments. Additionally, Michael et al. [22] explored a robust approach for
AV emotion recognition under noisy acoustic conditions with a focus on speech
features. Furthermore, the study by Chen et al. delves into the intricacies of
combining SE and emotion recognition. Their work emphasizes the essential role
of SE as a pre-processing mechanism to ensure the robustness of emotion
recognition systems, particularly in contexts where audio quality may be
compromised by environmental noise [3]. In a similar vein, Tiwari et al.
contributed to the research landscape by investigating multi-modal approaches
that integrate SE and emotion recognition, emphasizing the potential for
context-aware affective computing [31].  Motivated by the growing synergy
between SE and emotion recognition systems, as well as the need for more
robust and context-aware SE solutions. We propose a novel SE approach that
harnesses emotion recognition as a contextual cue to improve performance in
extremely noisy environments. Our approach integrates the emotion recognition
mechanism into the SE process to enhance the quality of audio signals and
effectively suppress noise. By integrating emotion recognition into the AVSE
process, we unlock the potential to adapt and fine-tune the enhancement
procedure according to the speaker’s emotional state. This adaptation can lead
to context-aware and emotionally relevant SE, ensuring that not only
background noise is reduced, but the speech itself remains natural and
emotionally resonant. In noisy conditions, where traditional enhancement
methods may falter, this fusion of emotion and SE becomes especially crucial
in achieving enhanced speech quality and intelligibility, providing more
authentic and context-aware communication experiences. This novel approach has
the potential to transform the landscape of SE and affective computing by
addressing real-world challenges where emotional nuances coexist with acoustic
disturbances. Our work is significant because it offers a promising paradigm
shift in AVSE. The experimental results demonstrate invaluable insights and
open the doors to the development of future emotion-aware applications across
diverse domains. Our work not only contributes to advancing the current state
of research but also holds the promise of positively impacting various facets
of human-machine interaction and communication i.e., conversational AI. The
main contributions of our paper are; i) we introduce a joint model for AV
fusion with emotion features integration to exploit the complementary
relationship across modalities for emotional nuances coexisting with acoustic
disturbances. ii) Unlike previous approaches, our model simultaneously
leverages intra- (temporal, spectral, and spatial relationship) and intermodal
(correlation and synchronisation between the audio and video signals)
relationships to capture complementary relationships effectively. iii) By
integrating emotional cues and deploying the joint feature representation, our
model also reduces heterogeneity across audio and visual features, resulting
in robust AV feature representations.iv) Extensive experiments on the
challenging MOSEI dataset demonstrate that our proposed fusion model
outperforms state-of-the-art AVSE models. Spectrogram analysis shows that our
model can efficiently leverage complementary contextual relationships while
retaining intramodal relationships.

Figure 1: Architecture of the proposed Emotion-AVSE system.

##  2 Utilizing Emotion Cues for AVSE System

We develop an encoder-decoder-style UNet architecture to train AVSE models.
Figure 1 provides an overview of our proposed emotion-aware AVSE system
architecture. Algorithm 1 delineates the stepwise process of our novel AVSE
algorithm, designed to enhance speech quality in noisy environments through
the integration of facial information and emotion-based contextual cues. The
proposed algorithm represents a comprehensive approach to AVSE, integrating
facial information and emotional cues. It begins with the processing of a
noisy speech signal through Short-Time Fourier Transform (STFT) for audio
feature extraction. Subsequently, facial detection using Multi-task
Convolutional Neural Network (MTCNN) [36] is applied to identify facial
regions in video frames. Visual feature extraction involves spatiotemporal
convolution and ResNet architectures, while emotion features are extracted
using an EfficientNet model[29]. The three modalities, audio, visual, and
emotion, are then seamlessly fused using a concatenation operation. The
resulting composite tensor undergoes refinement through downsampling
convolutional layers before being processed by a UNet decoder to generate an
enhanced spectrogram. The enhanced speech signal is synthesized by combining
the enhanced spectrogram with the phase of the original noisy speech. This
algorithm showcases a robust framework for enhancing speech quality by
integrating emotion-aware visual information into the audio processing
pipeline.

####  2.0.1 Audio feature extraction

For audio processing, we send the magnitude of $F\times T$ dimensional noisy
STFT to the network (where $F$ and $T$ are frequency and time dimensions) such
that

$X_{i}=STFT(x_{noisy})$ (1)

Here, we denote $X_{i}$ as the magnitude spectrum of the $i$-th noisy speech
signal $x_{noisy}$. The initial audio data is subsequently processed through
two convolutional layers with a filter size of $4$ and a stride of $2$,
resulting in reduced time-frequency dimensions

$X_{conv}=[Conv2D(X_{i})]*2$ (2)

where $X_{conv}$ represents the data after the first and second convolution
and Conv2D represents the 2D convolution operation with the filter size of $4$
and stride of $2$. Following this initial processing, the data then undergoes
further refinement through three consecutive convolutional blocks and
subsequent frequency pooling layers. Each convolutional layers employs a
filter of size $3$ and a stride of $1$. Subsequently, the frequency dimension
is halved due to the subsequent frequency pooling layers. These convolutional
and subsequent frequency poling layers are responsible for down-sampling the
features, ultimately arriving at their outputs, represented as
$X_{blocks}$.This entire process, encapsulated by the convolutional blocks and
frequency pooling operations, is detailed below.

$X_{blocks}=[FrequencyPooling(ConvBlocks(X_{conv}))]*3$ (3)

Note that throughout the convolutional block processing, the spatial dimension
is preserved. The output of the final convolutional block is designated as
$A_{i}$, representing the data at the frequency pooling layer for the $i$-th
frame in the above equation.

$A_{i}=X_{blocks}$ (4)

Algorithm 1 Stepwise AVSE Algorithm in Noisy Environments with Facial
Information and Emotion-Based Contextual Cues

1:Noisy speech signal $x_{noisy}$, Visual data $v_{video}$

2:Enhanced speech signal $x_{enhanced}$

3:Audio Processing:

4: $X_{i}\leftarrow{Magnitudespectrum}(x_{noisy})$

5: $A_{i}\leftarrow{UNetAudioEncoder(X_{i}))}$

6:Visual Processing:

7: $I_{i}\leftarrow{VideoReader}(video)$

8: $D_{i}\leftarrow{MTCNN}(I_{i})$

9: $C_{conv_{i}}\leftarrow ResNet(D_{i})$

10: $T_{i}\leftarrow TCN(C_{conv_{i}})$

11: $V_{i}\leftarrow Upsample(T_{i})$

12:Emotion Processing:

13:Connection: $\triangleright$ Connect Visual Processing to Emotion
Processing

14: $E_{EffNet}\leftarrow{EfficientNet}(D_{i})$ $\triangleright$ Use the
output of MTCNN in Emotion Processing

15: $E_{linear}\leftarrow{LinearLayer}(E_{EffNet})$

16: $E_{i}\leftarrow{Upsample}(E_{linear})$

17:Integration and Decoding:

18: $M_{composite}\leftarrow{Concatenate}(A_{i},V_{i},E_{i})$

19: $M_{composite_{down}}\leftarrow{Downsample}(M_{composite})$

20: $S_{enhanced}\leftarrow{UNetDecoder}(M_{composite_{down}})$

21: $x_{enhanced}\leftarrow{iSTFT}(S_{enhanced},Phase(x_{noisy}))$

####  2.0.2 Face Detection and Facial Emotion Emotion Recognition

In our approach, we employ pre-trained models to facilitate face detection and
Facial Emotion Recognition (FER). The initial step in this pipeline is face
detection, wherein we utilize the MTCNN model for robust and efficient face
localization from video streams. MTCNN excels at identifying faces within
images or video frames, functioning independently for each frame in the latter
scenario. MTCNN identifies and isolates facial regions within the video
frames.  The task of face detection in MOSEI YouTube videos is confronted with
multiple complexities due to differences in video durations and frame sizes.
To address these intricacies, we adopt a standardized approach involving
4-second video segments. Utilizing the VideoReader function, we extract
individual frames ($I_{i}$) from the video such that

$I_{i}=VideoReader(video)$ (5)

Subsequently, the MTCNN model is employed to perform face detection on these
visual frames. MTCNN identifies faces and provides the precise positions of
five key facial landmarks: the left eye, right eye, nose, left mouth corner,
and right mouth corner. The face localization process for the $i$-th frame is
expressed as

$D_{i}=MTCNN(I_{i})$ (6)

with $D_{i}$ symbolizing the resulting detected bounding box. The outcome of
MTCNN is a structured frame format characterized by dimensions (F, C, H, W),
where F pertains to the number of visual frames, C designates the channel, and
H and W correspond to height and width. To ensure consistency, all facial
images are uniformly resized to dimensions of 224x224.  The process of
detecting faces is followed by their subsequent analysis through the visual
feature extraction and Facial Emotion Recognition (FER) blocks. The visual
feature extraction pipeline is initiated by a 3D convolutional layer designed
for spatiotemporal convolution, complemented by an 18-layer ResNet-18
architecture, as described by He et al. (2016) [10]. Within the 3D
convolutional layer, short-term dynamics related to lip articulations are
effectively captured. This is accomplished through the application of a
convolutional layer featuring 64 3D kernels with dimensions of $5\times
7\times 7$ (time$\times$width$\times$height) and a stride of $1\times 2\times
2$. The entire operation can be succinctly denoted as

$C_{conv_{i}}=ResNet(Conv3D(D_{i}),64,(5,7,7))$ (7)

Subsequently, the output of the ResNet-18 architecture feeds into a temporal
convolutional network (TCN), following the methodology outlined in Martinez et
al.’s work on lip-reading [20]. This network is provided with a temporal
sequence of face images, each of size $N\times 224\times 224$, where $N$
represents the number of frames. The output of the visual feature network
yields a 512-D vector for each image frame. This process can be formally
expressed as

$T_{i}=TCN(C_{conv_{i}})$ (8)

To ensure compatibility with the audio features, the visual features are
upsampled to match the audio features’ sampling rate, denoted as

$V_{i}=Upsample(T_{i})$ (9)

In parallel, the Facial Emotion Recognition (FER) stage is executed through
the deployment of the High Speech Emotion Recognition (HSEmotion) model [27].
HSEmotion is a versatile tool capable of processing static images or facial
videos, offering a comprehensive analysis of emotional expressions. It
produces two distinct outputs: high-dimensional visual embeddings, recognized
as ”emotional features,” and posterior probabilities associated with eight
distinct emotions: Anger, Contempt, Disgust, Fear, Happiness, Neutral,
Sadness, and Surprise. These outputs are captured through the EfficientNet
model [29] such as,

$E_{EffNet}=EfficientNet(D_{i})$ (10)

where $E_{EffNet}$ signifies the high-dimensional visual embedding or
emotional features derived from the EfficientNet model.  It is worth noting
that, in this preliminary exploration, no fine-tuning was performed on the
architecture or parameters of the EfficientNet model explicitly for the task
of emotion recognition from facial expressions. Instead, the model was
harnessed to extract emotional feature representations from the faces detected
by the MTCNN model. This unique approach allowed us to investigate the
potential of employing these emotional feature representations as contextual
cues to enhance the overall system’s performance, with a particular emphasis
on emotion recognition. The EfficientNet model excels at capturing subtle
emotional nuances within facial images, thereby furnishing a comprehensive and
valuable representation of the emotional states conveyed within these images.

####  2.0.3 Multimodal fusion and speech resynthesis

The output of the FER model yields emotional features presented in the form of
a tensor with dimensions $(B,1280,F)$, where $B$ designates the batch size,
1280 represents the emotional embedding dimension, and $F$ accounts for the
number of visual frames in a video sequence. To ensure harmonization across
the modalities, the emotional features undergo transformation via a linear
layer, culminating in feature embeddings characterized by dimensions
$(B,512,F)$. This transformation is represented as

$E_{linear}=LinearLayer(E_{EffNet})$ (11)

Following the linear transformation, the output of the linear layer is
upsampled, converting it into a tensor with dimensions $(B,512,8,F)$. This
ensures alignment with the output from the audio UNet encoder and the visual
feature network. The upsampled emotional features are denoted as

$E_{i}=Upsample(E_{linear})$ (12)

Subsequently, the final integration stage takes shape, uniting the outputs
from the audio encoder, visual feature network, and emotion network, resulting
in an embedding tensor characterized by dimensions $(B,1536,8,F)$. This
integration process is concisely represented as

$M_{composite}=Concatenate(A_{i},V_{i},E_{i})$ (13)

The composite tensor, $M_{composite}$, then proceeds through additional
processing via a downsampling convolutional layer, yielding an output tensor
with dimensions $(B,512,8,F)$, and formalized as

$M_{{composite}_{down}}=Downsample(M_{composite})$ (14)

Ultimately, the processed output is fed into the UNet decoder, a pivotal step
in the generation of the final enhanced spectrogram and can be written as

$S_{enhanced}=UNetDecoder(M_{{composite}_{down}})$ (15)

In this work, we employed the ”enet_b0_8_va_mtl” model of EffientNet,
noteworthy for extending its predictions beyond the eight fundamental facial
expressions to encompass valence and arousal. These additional emotional
dimensions, concerned with assessing the level of emotional intensity and the
positive or negative aspects of human behavior, exceed the current scope of
this paper. Nevertheless, we anticipate integrating valence and arousal
dimensions into our future research, thereby enhancing the comprehensiveness
of our emotion-aware SE system.

##  3 Experimental Setup

###  3.1 Description of the CMU-MOSEI Dataset

The CMU Multimodal Opinion Sentiment and Emotion Intensity (MOSEI) dataset
stands out for its richness and suitability for various research tasks.
Comprising over 1,000 YouTube videos, MOSEI captures diverse and realistic
emotional expressions from multiple speakers across a wide range of
sentiments. Notably, the dataset provides detailed emotion and sentiment
labels, making it ideal for training emotion recognition, sentiment analysis,
and opinion mining models. The use of opinionated audio-visual content ensures
the data reflects real-world scenarios and the complex nuances of human
emotions. MOSEI’s significant diversity in both speakers and emotional
expression guarantees its generalizability and effectiveness in various
research applications.

###  3.2 Analyzing Audio-Visual Recordings with Emotions

The MOSEI dataset offers predefined train, validation, and test splits for
robust model development. The training set encompasses a diverse collection of
23,500 segments exceeding 113 hours of data, representing over 1000 unique
male and female speakers. Each audio file is provided in a single-channel
16kHz format, while the accompanying visual data boasts a frame rate of 30
fps. Furthermore, the development set (devset) contains 3,306 video recordings
(approximately 8 hours), meticulously curated to ensure no speaker overlap
exists between the train and dev sets, adhering to speaker independence
criteria. To expand data diversity and augment the training process, we
constructed a more challenging noisy dataset. This custom dataset meticulously
integrates noise samples from three established sources: the Deep Noise
Suppression Challenge (DNS) dataset [24], the Diverse Environments
Multichannel Acoustic Noise Database (DEMAND) [30], and the Clarity
Enhancement Challenge (CEC1) [9]. This enriched dataset further strengthens
the model’s ability to effectively enhance speech in a wider range of complex,
real-world noise scenarios.  To enhance the training process and expose the
model to diverse noise scenarios, we curated a custom noise dataset from three
distinct sources: CEC1, DEMAND, and DNS. This approach enriched our dataset
with a variety of noise types, encompassing both domestic and multi-channel
recordings. CEC1 provided 7 hours of everyday soundscapes, familiarizing the
model with common environmental noises. DEMAND contributed 1 hour of
recordings across 18 distinct soundscapes, further expanding the noise
diversity through its multi-channel approach. Finally, DNS bolstered the
dataset with 25 hours of unique noise selections sourced from Freesound,
ensuring our model encountered a wide range of sounds beyond the other
datasets.  The combined noise dataset was carefully segmented into training
(75%), validation (15%), and testing (10%) sets. To simulate realistic noisy
environments during training and validation, clean utterances were corrupted
with random noise samples at varying Signal-to-Noise Ratio (SNR) levels,
ranging from -9 to 6 dB. Importantly, noise types and SNR levels were kept
distinct within each set to maintain the integrity and quality of our training
data. For comprehensive testing, the clean test set was divided based on
target SNR levels. Each clean utterance was then contaminated with a random
noise sample from the dedicated test noise split, enabling systematic
evaluation across diverse noise conditions. This approach ensured the model’s
ability to perform well in a wide range of realistic scenarios.  To
quantitatively assess the effectiveness of our proposed system in enhancing
speech in noisy environments, we leverage several objective evaluation
metrics: Perceptual Evaluation of Speech Quality (PESQ) [25], Short-Time
Objective Intelligibility (STOI) [28], and Speech Distortion Index (SDI) [12].
These metrics provide valuable insights by offering numerical scores
reflecting the system’s ability to: Improve speech quality: PESQ (evaluates
perceived speech quality, mimicking human judgment), Enhance intelligibility:
STOI (measures the intelligibility of spoken words, quantifying how well the
message is understood), Reduce noise: SDI (assesses the level of noise
suppression achieved by the system).  By analyzing these metrics under various
noise conditions, we gain a comprehensive understanding of the system’s
performance and its potential for real-world applications.

##  4 Experimental Results

We first analyse how the integration of emotion as a contextual queue
described in Section II influences the SE performance of the proposed DL-based
AV SE system. In particular, we look at the sensitivity of the DL-based SE
system for different types of background noises, over a wide range of SNRs to
compare custom loss functions fairly in subsequent investigations. For the
initial comparison between the ASE and AVSE systems, we observe that the
integration of visual cues into the ASE system significantly enhances speech
quality, as evident in the higher PESQ scores. This improvement is attributed
to the ability of visual information to guide the enhancement process and
suppress noise more effectively. The AVSE system outperforms the ASE system,
as evidenced by higher PESQ scores, indicating a noticeable increase in speech
quality. Furthermore, the AVSE system exhibits better intelligibility, as
evidenced by higher STOI scores, suggesting that the combined audio-visual
model successfully leverages both modalities to enhance speech
intelligibility. Consequently, the estimated speech signal experiences less
distortion, as indicated by lower SDI scores.  Building upon these findings,
we incorporate emotional features as contextual cues for the SE process. This
innovative approach further enhances the system’s performance, leading to
excellent results in terms of PESQ, STOI, and SDI. The integration of
emotional cues significantly improves speech quality, evident in higher PESQ
scores. Furthermore, the emotional-aware system achieves superior
intelligibility, as indicated by elevated STOI scores, suggesting that
emotional information aids in effectively enhancing speech intelligibility,
even in extremely noisy conditions. Additionally, the system minimizes
distortion in the estimated speech signal, as observed in reduced SDI scores.
These results highlight the potential of emotional cues to further enhance the
performance of audio-visual speech enhancement systems in challenging acoustic
environments, ultimately contributing to more effective and context-aware
communication solutions.  Table 1 presents a performance comparison between
ASE, AVSE, and our proposed Emotion-AVSE system (termed E-AVSE), against three
baseline models: CochleaNet [7], IO-AVSE [14], and LaiNet [16]. We evaluated
these systems using standard l1-loss functions, except for IO-AVSE, which
employed STOI as a loss function for parameter optimization. The performance
analysis reveals that our AVSE system outperforms the baseline models, as
indicated by better PESQ and STOI scores. This suggests that the integration
of visual information into the audio-only framework significantly enhances
speech quality and intelligibility. Furthermore, the AVSE system demonstrates
substantial advantages, with marked improvements in terms of speech quality
and intelligibility compared to the baseline systems. Although the integration
of emotion information into the AVSE system slightly degrades speech quality,
the resulting score remains competitive and, more importantly, improves speech
intelligibility slightly. This implies that the Emotion-AVSE system
effectively leverages emotion cues to maintain robust performance in terms of
speech intelligibility, even in challenging noisy conditions. In summary, the
results in Table 1 highlight the effectiveness of our system, which, despite a
minor reduction in speech quality, excels in enhancing speech intelligibility,
demonstrating its potential as a context-aware solution for SE in the presence
of acoustic disturbances and emotional nuances.

Table 1: Performance comparison of proposed emotion-aware AVSE system with different Baseline models. Framework | Loss | Objective Evaluation Metrics
---|---|---
Function | PESQ | STOI | SDI
Noisy | – | 1.326 | 0.447 | 3.061
UNetA | MAE | 1.547 | 0.552 | 2.441
CochleaNet | MAE | 1.633 | 0.588 | 2.014
LaiNet [16] | MAE | 1.718 | 0.612 | 2.122
IO-AVSE | STOI | 1.690 | 0.602 | 2.011
UNetE-AVSE (ours) | MAE | 1.685 | 0.703 | 1.994

In Table 2, we present the results of optimizing our proposed model with three
different loss functions: l1-loss, STOI loss, and modulation domain loss
function [33]. These diverse loss functions offer multiple perspectives on the
performance of our AVSE system integrated with emotion cues. Our findings
reveal that, when optimized with STOI and modulation-domain loss functions,
our system demonstrates superior performance compared to the l1-loss function.
This outcome underscores the significance of considering various optimization
strategies in the context of emotion-aware AVSE. The adoption of different
loss functions serves as a pivotal step in our research, as it not only
highlights the adaptability of our model but also underscores the flexibility
of integrating emotion as a contextual cue. This multi-pronged approach
enables us to fine-tune our model parameters and understand the intricacies of
emotional cues for SE. Such findings are invaluable for the development of a
robust emotion-aware AVSE system that can operate optimally across diverse
acoustic conditions and enrich the landscape of affective computing.

Table 2: Performance comparison of proposed emotion-aware AVSE system using different loss functions. Framework | Loss | Objective Evaluation Metrics
---|---|---
Function | PESQ | STOI | SDI
UNetE-AVSE | MAE | 1.685 | 0.703 | 1.994
UNetE-AVSE | STOI | 1.661 | 0.695 | 2.119
UNetE-AVSE | Modulation-loss | 1.692 | 0.710 | 2.005

##  5 Conclusion

By integrating emotion recognition into audio-visual speech enhancement
(AVSE), our work presents a novel approach with demonstrably improved
performance. Leveraging a U-Net architecture, our model significantly
outperforms baselines, highlighting the value of incorporating emotional and
visual cues. Integrating emotion recognition enables the system to adapt its
processing based on the speaker’s emotional state, resulting in context-aware
and emotionally relevant speech enhancement. This adaptability is particularly
impactful in challenging noise environments where traditional methods
struggle.  This emotion-based approach holds vast potential. It can advance
assistive technologies for individuals with hearing impairments, enabling them
to better perceive and understand emotional nuances in speech. Additionally,
it can enhance conversational AI systems, facilitating more natural and
engaging interactions. Furthermore, by preserving emotional cues crucial for
human communication, our model paves the way for richer and more impactful
interactions across various settings. This work signifies a transformative
advancement, and the integration of emotions into speech enhancement promises
significant societal benefits.

## References

  * [1] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 12, pp. 8717–8727, 2018.
  * [2] T. Afouras, J. S. Chung, and A. Zisserman, “The conversation: Deep audio-visual speech enhancement,” _arXiv preprint arXiv:1804.04121_ , 2018.
  * [3] Y.-W. Chen, J. Hirschberg, and Y. Tsao, “Noise robust speech emotion recognition with signal-to-noise ratio adapting speech enhancement,” _arXiv preprint arXiv:2309.01164_ , 2023.
  * [4] I.-C. Chern, K.-H. Hung, Y.-T. Chen, T. Hussain, M. Gogate, A. Hussain, Y. Tsao, and J.-C. Hou, “Audio-visual speech enhancement and separation by utilizing multi-modal self-supervised embeddings,” in _2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)_. IEEE, 2023, pp. 1–5.
  * [5] S.-Y. Chuang, H.-M. Wang, and Y. Tsao, “Improved lite audio-visual speech enhancement,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 1345–1359, 2022.
  * [6] Y. Ephraim and D. Malah, “Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,” _IEEE Transactions on acoustics, speech, and signal processing_ , vol. 32, no. 6, pp. 1109–1121, 1984.
  * [7] M. Gogate, K. Dashtipour, A. Adeel, and A. Hussain, “Cochleanet: A robust language-independent audio-visual model for real-time speech enhancement,” _Information Fusion_ , vol. 63, pp. 273–285, 2020.
  * [8] A. Golmakani, M. Sadeghi, and R. Serizel, “Audio-visual speech enhancement with a deep kalman filter generative model,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [9] S. Graetzer, J. Barker, T. J. Cox, M. Akeroyd, J. F. Culling, G. Naylor, E. Porter, and R. Viveros Munoz, “Clarity-2021 challenges: Machine learning challenges for advancing hearing aid processing,” in _Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH_ , vol. 2, 2021.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [11] T.-A. Hsieh, C. Yu, S.-W. Fu, X. Lu, and Y. Tsao, “Improving perceptual quality by phone-fortified perceptual loss using wasserstein distance for speech enhancement,” _arXiv preprint arXiv:2010.15174_ , 2020.
  * [12] Y. Hu and P. C. Loizou, “Evaluation of objective quality measures for speech enhancement,” _IEEE Transactions on audio, speech, and language processing_ , vol. 16, no. 1, pp. 229–238, 2007.
  * [13] T. Hussain, M. Diyan, M. Gogate, K. Dashtipour, A. Adeel, Y. Tsao, and A. Hussain, “A speech intelligibility enhancement model based on canonical correlation and deep learning for hearing-assistive technologies,” _arXiv preprint arXiv:2202.04172_ , 2022.
  * [14] T. Hussain, M. Gogate, K. Dashtipour, and A. Hussain, “Towards intelligibility-oriented audio-visual speech enhancement,” _arXiv preprint arXiv:2111.09642_ , 2021.
  * [15] M. Kolbæk, Z.-H. Tan, S. H. Jensen, and J. Jensen, “On loss functions for supervised monaural time-domain speech enhancement,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 825–838, 2020.
  * [16] R. L. Lai, J.-C. Hou, M. Gogate, K. Dashtipour, A. Hussain, and Y. Tsao, “Audio-visual speech enhancement using self-supervised learning to improve speech intelligibility in cochlear implant simulations,” _arXiv preprint arXiv:2307.07748_ , 2023.
  * [17] H. Li, S.-W. Fu, Y. Tsao, and J. Yamagishi, “imetricgan: Intelligibility enhancement for speech-in-noise using generative adversarial network-based metric learning,” _arXiv preprint arXiv:2004.00932_ , 2020.
  * [18] X. Lu, Y. Tsao, S. Matsuda, and C. Hori, “Speech enhancement based on deep denoising autoencoder.” in _Interspeech_ , vol. 2013, 2013, pp. 436–440.
  * [19] R. Martin, “Noise power spectral density estimation based on optimal smoothing and minimum statistics,” _IEEE Transactions on speech and audio processing_ , vol. 9, no. 5, pp. 504–512, 2001.
  * [20] B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6319–6323.
  * [21] D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y. Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio-visual speech enhancement and separation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 1368–1396, 2021.
  * [22] M. Neumann and N. T. Vu, “Investigations on audio-visual emotion recognition in noisy conditions,” in _2021 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2021, pp. 358–364.
  * [23] K. Paliwal, B. Schwerin, and K. Wójcicki, “Speech enhancement using a minimum mean-square error short-time spectral modulation magnitude estimator,” _Speech Communication_ , vol. 54, no. 2, pp. 282–305, 2012.
  * [24] C. K. Reddy, H. Dubey, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “Icassp 2021 deep noise suppression challenge,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6623–6627.
  * [25] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in _2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221)_ , vol. 2. IEEE, 2001, pp. 749–752.
  * [26] M. Sadeghi, S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Horaud, “Audio-visual speech enhancement using conditional variational auto-encoders,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 1788–1800, 2020.
  * [27] A. V. Savchenko, “Hsemotion: High-speed emotion recognition library,” _Software Impacts_ , vol. 14, p. 100433, 2022.
  * [28] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short-time objective intelligibility measure for time-frequency weighted noisy speech,” in _2010 IEEE international conference on acoustics, speech and signal processing_. IEEE, 2010, pp. 4214–4217.
  * [29] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [30] J. Thiemann, N. Ito, and E. Vincent, “Diverse environments multichannel acoustic noise database (demand),” 2013.
  * [31] U. Tiwari, M. Soni, R. Chakraborty, A. Panda, and S. K. Kopparapu, “Multi-conditioning and data augmentation using generative noise model for speech emotion recognition in noisy conditions,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7194–7198.
  * [32] A. Triantafyllopoulos, G. Keren, J. Wagner, I. Steiner, and B. Schuller, “Towards robust speech emotion recognition using deep residual networks for speech enhancement,” 2019.
  * [33] T. Vuong, Y. Xia, and R. M. Stern, “A modulation-domain loss for neural-network-based real-time speech enhancement,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6643–6647.
  * [34] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “A regression approach to speech enhancement based on deep neural networks,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 1, pp. 7–19, 2014.
  * [35] G. Zhang, L. Yu, C. Wang, and J. Wei, “Multi-scale temporal frequency convolutional network with axial attention for speech enhancement,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9122–9126.
  * [36] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE signal processing letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [37] S. Zhao, B. Ma, K. N. Watcharasupat, and W.-S. Gan, “Frcrn: Boosting feature representation using frequency recurrence for monaural speech enhancement,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9281–9285.


# Hindsight is Only 50/50: Unsuitability of MDP based Approximate POMDP
Solvers for Multi-resolution Information Gathering

Sankalp Arora1, Sanjiban Choudhury1 and Sebastian Scherer1 1The Robotics
Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA
15213, USA asankalp, sanjibac<EMAIL_ADDRESS>
###### Abstract

Partially Observable Markov Decision Processes (POMDPs) offer an elegant
framework to model sequential decision making in uncertain environments.
Solving POMDPs online is an active area of research and given the size of
real-world problems approximate solvers are used. Recently, a few approaches
have been suggested for solving POMDPs by using MDP solvers in conjunction
with imitation learning. MDP based POMDP solvers work well for some cases,
while catastrophically failing for others. The main failure point of such
solvers is the lack of motivation for MDP solvers to gain information, since
under their assumption the environment is either already known as much as it
can be or the uncertainty will disappear after the next step. However for
solving POMDP problems gaining information can lead to efficient solutions. In
this paper we derive a set of conditions where MDP based POMDP solvers are
provably sub-optimal. We then use the well-known tiger problem to demonstrate
such sub-optimality. We show that multi-resolution, budgeted information
gathering cannot be addressed using MDP based POMDP solvers. The contribution
of the paper helps identify the properties of a POMDP problem for which the
use of MDP based POMDP solvers is inappropriate, enabling better design
choices.

##  I Introduction

MDPs were developed as a part of stochastic control theory [1]. The MDP
formulation of problems makes underlying assumptions that the state of the
environment and the agents is known, while dynamics can be stochastic with
known parameters and the state is Markovian, [2]. MDPs are computationally
relatively efficient to solve [3]. Therefore, in robotics applications,
whenever it is reasonable to make the assumption that the world is completely
known or the uncertainty in the world can be ignored, MDP has been a useful
tool to solve sequential decision making problems [4, 5, 6, 7, 8]. However,
for the problems where uncertainty cannot be ignored POMDPs were developed
[9].

POMDPs are computationally intractable to solve optimally in the worst case
[10]. Approximate POMDP solvers have been developed to solve POMDPs tractably,
we refer readers to [11, 12] for a comprehensive literature review. A popular
approach to solving POMDPs approximately is to leverage MDP solvers and take
the best expected action assuming the uncertainty about the environment will
not change or it will magically disappear in the next step, FF-Re-plan [13],
Hindsight optimization [14] and QMDP [15] are examples of such methods.
Despite strong assumptions these methods found early success as leading online
POMDP solvers and have been effectively applied to problems like shared
autonomy [16], manipulation planning [17, 18] etc. We call such methods MDP
based approximate POMDP solvers, MDP-POMDP solvers in short.

Use of MDP based approximate POMDP solvers requires computing expectation over
either prior or posterior of the distribution over the state space. For large
state spaces computing this expectation online can be challenging, to overcome
this challenge, imitation learning techniques are used in conjunction with MDP
solvers to train a policy offline such that it implicitly takes the
expectation over uncertainty over state space and picks the best action online
[19, 20, 21].

MDP based approximate POMDP solvers, including the imitation learning based
methods that rely on MDPs fail at taking actions that do not belong to optimal
MDP policies. It is well known, actions that lead to reduction in uncertainty
about the state space, while not providing rewards are an important class of
actions that are often not included in optimal MDP policies [14, 15, 22]. The
failure of agents to take such actions can lead to sub-optimal POMDP
solutions, sometimes, catastrophically so. We provide a formal definition for
such actions in this paper and identify with examples the conditions under
which MDP based approximate POMDP solvers fail. The presence of actions
identified in this paper, can be confirmed without solving full POMDP
problems, and for any problem where such actions exist, the readers are
advised to exercise caution against using MDP-POMDP solvers. We also show that
such actions do exist for multi-resolution informative path planning problems.
The primary need to identify problems where MDP-POMDP solvers do not work is
to avoid the implementation effort before realizing the unsuitability and to
make better solver design choices.

Consider the famous tiger problem, [9], there is a tiger hidden behind one of
the two doors in front of the agent, the agent has three possible actions. 1.
Listen to determine the door behind which the tiger is hidden. 2. Open the
door without the tiger for high reward 3. Open the door with the tiger for a
high penalty. When the uncertainty about tiger’s location is removed the
optimal MDP policy is to open the door without the tiger. The action to listen
for the location of tiger is never a part of the optimal MDP policy. Hence, if
there is a $50\%$ chance that the tiger is behind left door and the agent is
using MDP-POMDP solvers, it will never listen for location of the tiger and
have only a $50/50$ chance of opening the door with a tiger in it. Whereas, if
the agent would have listened for the the presence of the tiger, it could have
been guaranteed to avoid the tiger. Knowing the tiger problem is unsuitable
for MDP-POMDP solvers, we can avoid those methods while designing an approach
to solve the problem.

We formalize the conditions under which MDP-POMDP based solvers fail alongside
the actions symptomatic of such conditions in section IV. We then provide a
detailed example of the tiger problem and multi-resolution, budgeted
informative path planning problem where such conditions exist and MDP-POMDP
solvers can lead to unacceptably poor performance in section V. Before going
further we formally re-introduce the MDP and POMDP formulations in section II
for the sake of completeness and briefly describe how MDP solvers are used
solve POMDP problems approximately in section III.

##  II Problem Formulation

###  II-A MDPs

A markov decision process can be represented by the tuple
$MDP=(\mathcal{S},s_{0},\mathcal{A},\Omega,R,Term)$ where

  * •

$\mathcal{S}$ is a set of states.

  * •

$s_{o}$ is the state at time $0$, initial state.

  * •

$\mathcal{A}$ is a set of actions.

  * •

$\Omega$ is a set of state transition probabilities.

  * •

$R:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is the reward function.

  * •

$Term:\mathcal{S}\times\mathcal{A}\to\\{0,1\\}$ is the terminating condition.

At each time step, the agent takes an action $a\in\mathcal{A}$ which causes
the environment to transition from state $s$ to state
$s^{\prime}\in\mathcal{S}$ with probability
$\Omega{}\left(s,a,s^{\prime}\right)=P(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$.
The agent receives a reward $R{}\left(s,a\right)$. On reaching the new state
$s^{\prime}$. The MDP is terminated if $Term{}\left(s,a\right)=1$.

$\pi_{\mathrm{M}}:\mathcal{S}\to\mathcal{A}$ be a policy function for the MDP
that maps from a state to action. Let the state distribution induced by a
policy $\pi_{\mathrm{M}}$ after $t$ time steps, starting with state $s_{0}$ be
$P(s|\pi_{\mathrm{M}},s_{0},t)$. The value of a policy $\pi_{\mathrm{M}}$ is
the expected cumulative reward for executing $\pi_{\mathrm{M}}$ for $T$ time
steps on the induced state and history distribution

$J\left(\pi_{\mathrm{M}}\right)=\sum\limits_{t=1}^{T}\mathbb{E}_{s_{t}\sim
P(s_{t}|\pi_{\mathrm{M}},s_{0},t)}\left[R{}\left(s_{t},\pi_{\mathrm{M}}(s_{t})\right)\right]$
(1)

The optimal policy for the MDP maximizes the expected cumulative reward, i.e
$\pi_{\mathrm{M}}^{*}\in\operatorname*{arg\,max}\limits_{\pi_{\mathrm{M}}\in\Pi_{\mathrm{M}}}J\left(\pi_{\mathrm{M}}\right)$.

The value of a state $s$ for a given policy $\pi_{\mathrm{M}}$ is given by
$V^{\pi_{\mathrm{M}}}(s)=\sum\limits_{i=1}^{T}\mathbb{E}_{s_{i}\sim
P(s_{i}|\pi_{\mathrm{M}},s)}\left[R{}\left(s_{i},\pi_{\mathrm{M}}(s_{i})\right)\right]$

An MDP solver takes an MDP as an input and returns the optimal policy
$\pi_{\mathrm{M}}^{*}$.

###  II-B POMDPs

In most of the real world situations the state of the environment cannot be
fully known. Agents rely on observations to infer the state of the
environment. Since, the state of the environment is inferred, the agent
maintains a probability distribution over possible set of states that the
environment can be in. Let, that distribution be $b:\mathcal{S}\to[0,1]$. At
each time step, the environment is in some state $s\in\mathcal{S}$ which
cannot be directly observed. Let, the initial belief be given by $b_{0}(.)$.
The agent takes an action $a\in\mathcal{A}$ which causes the environment to
transition to state $s^{\prime}\in\mathcal{S}$ with probability
$\Omega{}\left(s,a,s^{\prime}\right)$. The agent receives a reward
$R{}\left(s,a\right)$. On reaching the new state $s^{\prime}$, it receives an
observation $o\in\mathcal{O}$ according to the probability
$Z{}\left(s^{\prime},a,o\right)=P(o_{t+1}=o|s_{t+1}=s^{\prime},a_{t}=a)$.
$b_{t}(.)$ is the state of the belief of the agent at time then $b_{t+1}(.)$,
given an action $a_{t}$ and observation $o_{t+1}$ by

$b_{t+1}(s^{\prime})=\eta\;Z{}\left(s^{\prime},a,o\right)\sum\limits_{s\in\mathcal{S}}\Omega{}\left(s,a,s^{\prime}\right)b(s)$
(2)

where $\eta$ is a normalization constant.

$\pi(b_{t}):B\to\mathcal{A}\in\Pi$ be a policy function for the MDP that maps
from a belief state to action.

The reward of a belief state is given by

$R{}_{B}\left(b_{t},a_{t}\right)=\sum\limits_{s\in\mathcal{S}}R{}\left(s,a_{t}\right)b(s)$
(3)

The value of a policy $\pi$ is the expected cumulative reward for executing
$\pi$ for $T$ timesteps on the induced belief distribution. Given a starting
belief $b$, let $P(b^{\prime}|b,\pi,i)$ be the induced belief distribution
after $i$ timesteps.

$J\left(\pi\right)=\sum\limits_{t=1}^{T}\mathbb{E}_{b_{t}\sim
P(b_{t}|\pi,t)}\left[R{}_{B}\left(b_{t},\pi(b_{t})\right)\right]$ (4)

The optimal policy maximizes the expected cumulative reward, i.e
$\pi^{*}\in\operatorname*{arg\,max}\limits_{\pi\in\Pi}J\left(\pi\right)$.

The value of executing a policy $\pi$ from a belief $b$ is the expected
cumulative reward:

$\tilde{V}^{\pi}(b)=\sum\limits_{i=1}^{T}\mathbb{E}_{b_{i}\sim
P(b_{i}|b,\pi,i)}\left[R{}_{B}\left(b_{i},\pi(b_{i})\right)\right]$ (5)

The Q-value function $\tilde{Q}^{\pi}(b,a)$ is defined as the expected sum of
one-step-reward and value-to-go:

$\displaystyle\tilde{Q}^{\pi}(b,a)=$ $\displaystyle R{}_{B}\left(b,a\right)+$
(6) $\displaystyle\mathbb{E}_{b^{\prime}\sim
P(b^{\prime}|b,a)}\left[\tilde{V}^{\pi}(b^{\prime})\right]$

POMDP provides an elegant framework to formalize and tackle planning problems
for agents operating in partially known environments. There are two major
challenges that POMDP solvers face: (1) Keeping track of evolving uncertainty
about the state space over the planning horizon. As future observations need
to be accounted for, the solver needs to keep track of future beliefs that are
exponential with respect to number of future observation steps. (2) Computing
the expectation over the state space. Since the state space of most of the
problems worth solving is large, computing an expectation over such state
space needs large computation, making it expensive to evaluate online. In the
next section we explore how MDP-POMDP solvers overcome these challenges.

##  III Solving POMDP through MDP Solvers

Approximate POMDP solvers like Hindsight Optimization [14] and QMDP [15]
leverage MDP solvers and simplifying assumptions about the environment
uncertainty to provide approximate solutions to POMDP. Hindsight optimization
finds the optimal action assuming that uncertainty about the state cannot be
changed, whereas QMDP assumes that the uncertainty will magically disappear
after the next action. For both these approaches, action for a given belief
$b$ is given by equation 17. The only difference is that the value function
computed by using QMDP is a tighter upper-bound than Hindsight optimization
for the POMDP value function.

$\displaystyle
a=\operatorname*{arg\,max}\limits_{a\in\mathcal{A}}\mathbb{E}_{s\sim
b(s)}\left[\max_{\pi_{M}\in\Pi_{M}}Q^{\pi_{M}}(s,a)\right]$ (7)

Through the simplifying assumptions about the evolution of belief state, these
approaches overcome the first challenge for POMDP solvers — keeping track of
evolving uncertainty over planning horizon. However, these approaches still
need the expectation over the state space to be computed.

Imitation learning based approaches [19, 20, 21] address this concern through
data driven techniques. MDP solvers are used over sampled MDP problems to
train a policy on the expected distribution of problems. Enabling the online
policy to take the best decision in expectation if magically all the
uncertainty would disappear in the next step.

Family of approximate POMDP solvers that use MDP work quite well in problems
where the required changes in belief can be attained by actions that are
rewarding as well. A well-known problem with MDP solvers is that they do not
take actions that don’t belong to optimal MDP policies. Hence, neither do the
imitation learning methods.

A class of such actions is informative actions. MDP solvers have no motivation
to gain information, since under their assumption, either the environment is
already known as much as it can be or the uncertainty will disappear after the
next step but for POMDP problems gaining information can be useful, [15].

In the next section we show that gaining information through observations can
only lead to a belief state with higher value, and discuss a class of
information gaining actions that will not be taken by an MDP solver leading to
sub-optimal solutions by MDP-POMDP solvers.

##  IV Expected Value of Information

Information is gained through observing the environment. Let a POMDP be given
by $POMDP=(\mathcal{S},b_{0},\mathcal{A},\Omega,R,Z,Term)$. Let
$o\in\mathcal{O}$ be observed and the belief changes from $b$ to $b^{o}$. The
$Q-value$ of the $b^{o}$ is given by equation 8.

$\displaystyle\tilde{V}^{\pi^{*}}(b^{o})=\tilde{Q}^{\pi^{*}}(b^{o},a)=$
$\displaystyle\max_{a\in\mathcal{A}}\bigg{[}R{}_{B}\left(b^{o},a\right)+$ (8)
$\displaystyle\mathbb{E}_{b^{\prime}\sim
P(b^{\prime}|b^{o},a)}\left[\tilde{V}^{\pi^{*}}(b^{\prime})\right]\bigg{]}$

Let’s say the probability of observing observation $o$ given a belief $b$ is
$P(o|b)$.

######  Definition 1.

Expected value of information is given by the difference between the expected
of value of belief state reached after making an observation and the optimal
value of the belief state before the observation, equation 9.

$\displaystyle\mathcal{EVI}_{o}(b)=\mathbb{E}_{o\sim
P(o|b)}\left[\max_{a\in\mathcal{A}}\tilde{Q}^{\pi^{*}}(b^{o},a)\right]-\tilde{V}^{\pi^{*}}(b)$
(9)

######  Theorem 1.

Expected value of information is greater than or equal to 0, for all
observation and belief.

###### Proof.

If we ignore the observation and take the same action that maximizes the
Q-value of $b$, instead of $b^{o}$, then the observation does not affect the
actions, and the expected value of the belief state. Lets say, the optimal
action for $b$ is given by ,
$a^{*}_{b}=\operatorname*{arg\,max}\limits_{a\in\mathcal{A}}\tilde{Q}^{\pi^{*}}(b,a)$.
We replace $\max_{a\in\mathcal{A}}$ with $a^{*}_{b}$ in equation 9, refer to
equation 10.

$\displaystyle\mathbb{E}_{o\sim
P(o|b)}\left[\tilde{Q}^{\pi^{*}}(b^{o},a^{*}_{b})\right]-\tilde{V}^{\pi^{*}}(b)$
(10)

Expanding equation 10.

$\displaystyle\sum\limits_{o\in\mathcal{O}}\sum\limits_{s\in\mathcal{S}}R{}\left(s,a^{*}_{b}\right)P(s|b,o)P(o|b)+$
(11) $\displaystyle\sum\limits_{o\in\mathcal{O}}\sum\limits_{b^{\prime}\in
B}\tilde{V}^{\pi^{*}}(b^{\prime})P(b^{\prime}|b,o,a^{*}_{b})P(o|b)-{\tilde{V}^{\pi^{*}}(b)}=0$

Marginalizing over $o$.

$\displaystyle\sum\limits_{s\in\mathcal{S}}R{}\left(s,a^{*}_{b}\right)P(s|b)+$
(12) $\displaystyle\sum\limits_{b^{\prime}\in
B}\tilde{V}^{\pi^{*}}(b^{\prime})P(b^{\prime}|b,a^{*}_{b})-{\tilde{V}^{\pi^{*}}(b)}$
$\displaystyle{\tilde{V}^{\pi^{*}}(b)}-{\tilde{V}^{\pi^{*}}(b)}=0$

Since we replaced a max function with a fixed action, we can infer that the
minimum value of $\mathcal{EVI}_{o}(b)$ is $0$. ∎

Unfortunately, optimal MDP solvers do not take informative actions if the
actions do not provide with reward or access to more rewarding states.

######  Definition 2.

Informative actions $(a_{I})$ are actions that are not a part of the optimal
MDP policy for any state or time, but lead to observations. Formally, they are
defined by following set of conditions:

  * •

$a_{I}\neq\pi^{*}(s)\forall s\in\mathcal{S}$.

  * •

$P(b^{\prime}|b,a,o)=P(b^{\prime}|b,o)P(o|b)$, where $P(b^{o}|b,o)=1$ and
$P(b^{\prime}|b,o)=0\forall b^{\prime}\neq b^{o}$.

If informative actions lead to more valuable information that they cost, then
MDP-POMDP solvers are provably sub-optimal.

######  Theorem 2.

If there exists an informative action $a_{I}\in\mathcal{A}$ in POMDP, such
that the expected value of information attained by $a_{I}$ is $\geq-
R_{b}(b,a_{I})$, then family of MDP-POMDP solvers will be sub-optimal by
atleast $\mathcal{EVI}_{o}(b)+R{}_{B}\left(b,a_{I}\right)$.

###### Proof.

The difference between the Q-value of $a_{I}$ and Q-value of $a^{*}_{b}$ is
given by equation 13.

$\displaystyle\tilde{Q}^{\pi^{*}}(b,a_{I})-\tilde{Q}^{\pi^{*}}(b,a^{*}_{b})$
(13) $\displaystyle=R{}_{B}\left(b,a_{I}\right)+\mathbb{E}_{b^{\prime}\sim
P(b^{\prime}|b,a)}\left[\tilde{V}^{\pi^{*}}(b^{\prime})\right]-\tilde{V}^{\pi^{*}}(b)$
(14)

Since an observation is observed immediately after taking action $a_{I}$ and
definition of $a_{I}$, equation 14 reduces to equation 16.

$\displaystyle R{}_{B}\left(b,a_{I}\right)+\mathbb{E}_{o\sim
P(o|b)}\left[\max_{a\in\mathcal{A}}\tilde{Q}^{\pi^{*}}(b^{o},a)\right]-\tilde{V}^{\pi^{*}}(b)$
(15) $\displaystyle R{}_{B}\left(b,a_{I}\right)+\mathcal{EVI}_{o}(b)$ (16)

∎

In the next section we show two examples, where MDP-POMDP solvers
catastrophically fail on account of their inability to take informative
actions, Multi-resolution informative path planning being one of them. We
first start with a detailed tiger problem.

##  V Examples

###  V-A Tiger Problem

In the canonical tiger problem there are two doors and a tiger is hidden
behind one of them. There are two possible states, either the tiger is hidden
behind the left door $\mathcal{T}_{L}$ or the right door $\mathcal{T}_{R}$,
$\mathcal{S}=\left\\{\mathcal{T}_{L},\mathcal{T}_{R}\right\\}$. The action set
is given by, $\mathcal{A}=\left\\{d_{L},d_{R},l\right\\}$, where $d_{L}$
denotes opening the door at left, $d_{R}$ denotes opening the door at right
and $l$ denotes the action of listening to determine the location of the
tiger. Action $l$ can return observations,
$\mathcal{O}=\left\\{o_{L},o_{R}\right\\}$, where $o_{R}$ denotes a tiger is
heard behind the right door and $o_{L}$ denotes a tiger is heard behind the
left door. Let the probability of hearing a tiger behind a door $X\in{L,R}$ if
there is a tiger behind door $X$ is given by $P(o_{X}|\mathcal{T}_{X})=1$.

The reward of the action state pairs is given by table LABEL:tab:tiger_reward.

| $s=\mathcal{T}_{L}$ | $s=\mathcal{T}_{R}$
---|---|---
$a=d_{L}$ | $R(\mathcal{T}_{L},d_{L})=0$ | $R(\mathcal{T}_{R},d_{L})=100$
$a=d_{R}$ | $R(\mathcal{T}_{L},d_{R})=100$ | $R(\mathcal{T}_{R},d_{R})=0$
$a=l$ | $R(\mathcal{T}_{L},l)=-1$ | $R(\mathcal{T}_{R},l)=-1$

Let the initial belief be
$b_{0}(\mathcal{T}_{L})=P(\mathcal{T}_{L})=1-P(\mathcal{T}_{R})=0.5$. If the
uncertainty is removed from this problem, the optimal MDP solver will suggest
the action to open the door with no tiger behind it, $a=d_{L}$ or $a=d_{R}$.
And since the action $l$ is neither rewarding or does not lead to a rewarding
state, it will never be taken by the MDP solver. Approaches like hindsight
optimization and Q-MDP use MDP solvers to compute the next action. Both these
approaches find the optimal action assuming that uncertainty about the state
cannot be changed. The action is given by equation 17.

$\displaystyle
a=\operatorname*{arg\,max}\limits_{a\in\mathcal{A}}\mathbb{E}_{s\sim
b(s)}\left[\max_{\pi_{M}\in\Pi_{M}}Q^{\pi_{M}}(s,a)\right]$ (17)

For the tiger example with $b_{0}=0.5$, equation returns actions
$\mathcal{T}_{L}$ or $\mathcal{T}_{R}$ with the hindsight upper-bound value
$\tilde{V_{hs}}^{\pi^{*}_{M}}(b_{0})=100$, whereas the actual value of action
$\mathcal{T}_{L}$ or $\mathcal{T}_{R}$ is $50$. However, if the action $l$ is
taken, the state of the environment is revealed leading to a Q-value of action
$l$, $\tilde{Q}^{\pi*}(b,l)=99$. Therefore, for this problem, Hindsight is
only $50/50$ as it takes best expected action in hindsight and has no implicit
motivation to gain information. In this problem all the MDP-POMDP solvers
discussed in section III, will face the same problem. Making them unfit to
solve the famous tiger problem. In the next section we demonstrate that MDP-
POMDP solvers are unfit to solve multi-resolution, budgeted information
gathering through a scaled down version of the problem.

###  V-B Multi-resolution, Budgeted Information Gathering

Unmanned Aerial Vehicles (UAVs) have the capability to gain height and gather
low resolution information at a large scale and the agility to zoom down on
relevant regions of the information to gain high resolution information. This
ability makes them suitable for locating objects in the environment and gain
information about those objects. The UAV only receives a reward if the high
resolution information about the object of interest is collected. Since, the
action of gaining height and gaining low resolution information only results
in reduction of uncertainty and no actual reward, we show it is an informative
action and MDP-POMDP solvers are unable to exploit the ability of UAVs to
gather information at multiple resolution.

Figure 1: (a) The UAV gets a high reward if it visits the same cell as the car
to gather high resolution information of the car (lets say its numberplate.).
Since the UAV knows where the car is, it does not need to take action $up$ and
can directly go to the cell with the car. (b) In the POMDP version of the
problem the location of the car is unknown, hence there is only a small chance
that the next cell the UAV goes to has a car in it, leading to a small
expected reward. (c) But if UAV gains height, the uncertainty about the
location of the car is removed. Although there is no reward from this action
but the removal uncertainty leads to guarantee of gaining high reward by
visiting the cell in which the car exists. Since, MDP-POMDP solvers cannot
take informative action $up$, they are sub-optimal for such data gathering
problems.

Let us assume the UAV operates in a grid with cells
$C=\\{c_{1},c_{2},c_{3},c_{4},c_{5},c_{6},c_{7},c_{8},c_{9}\\}$, with an
object of interest, a car in cell $c_{r}$, where $r\in[1,9]$, see figure 1.
The UAV starts at time $t=0$ from cell $c_{5}$. At time $t$, the state of the
environment is given by the history of cells visited by the UAV and the
location of the car $c_{r}$,
$s_{t}=<c_{r},c_{i_{1}},c_{i_{2}},c_{i_{3}},...,c_{i_{t}}>$, where
$c_{i_{1:t}}\in C$. There are five possible actions that the UAV can execute
in this grid world, $\mathcal{A}=\\{north,east,south,west,up\\}$. Action
$a=north$ moves the UAV to the north of the cell it currently occupies if
possible, otherwise it keeps the UAV in the same cell. Similarly,
$east,south,west$ actions move the UAV to the east, south or west of cell it
currently occupies respectively. Action $up$ results in the UAV gaining height
to inspect the presence of cars in all the cells and returning to the cell it
executed $up$ in. Let, $T:\mathcal{S}X\mathcal{A}\to C$ be the deterministic
transition function that takes the state of the UAV and an action as input and
returns the the cell in which the UAV will be after the action is executed.

In the MDP definition of the problem, location of the car is known. The UAV
gets a high reward for going to the same cell as the car, all actions cost
$-1$ and action $up$ costs relatively higher, $-2$. The reward function is
given by equation 18.

$R(s,a)=\left\\{\,\begin{IEEEeqnarraybox}[]{[}][c]{l?s}\IEEEstrut
100&if$T(s,a) = c_r \land\nexists c_{i_j} \in\mathcal{S}: c_{i_j}=c_r$,\\\
-2&if$a = up$,\\\ -1&otherwise.\IEEEstrut\end{IEEEeqnarraybox}\right.$ (18)

Terminal state is reached when either cell $c_{5}$ is visited twice or total
cost of taking actions (excluding reward of visiting the same cell as the car)
exceeds five.

Since the location of the car is known, an optimal MDP solver will directly
approach the car taking actions either $east,west,north$ or $south$ and return
to the terminal cell $c_{5}$, the value of the starting state,
$s=<c_{r},c_{5}>$ is given by equation 19.

$V^{\pi_{\mathrm{M}}^{*}}(s=c_{5})=\left\\{\,\begin{IEEEeqnarraybox}[]{[}][c]{l?s}\IEEEstrut
99&if$c_r \in\\{c_2,c_4,c_6,c_8\\}$,\\\ 97&if$c_r
\in\\{c_1,c_3,c_7,c_9\\}$.\IEEEstrut\end{IEEEeqnarraybox}\right.$ (19)

Action $up$ does not offer any reward and increases the cost of all paths that
leads to any rewarding state. Therefore, an MDP solver will never take the
action $up$. Action $up$ here is an informative action according definition 2.
Presence of an informative action cautions us against using MDP based
Approximate POMDP solvers for this problem.

In the POMDP version of the problem, the UAV is uncertain about the location
of the car. Assuming ties between actions are broken randomly, and the belief
about the location of the car is uniformly distributed, the value of initial
belief state $b^{uniform}_{0}$ given hindsight optimization is used at every
step can be computed using equation 5 and is given by equation 20.

$\tilde{V}^{\pi_{h}}(b^{uniform}_{0})=34.125$ (20)

where, $\pi_{h}$ is the policy attained by iteratively using hindsight
optimization.

The value of information about the car is given by the difference between the
expected value if the location of the car is known, equation 19, the expected
value if it is unknown, equation 20, $63.875$

The optimal POMDP policy will use the action $up$ to reduce the uncertainty
and guarantee that it will find the car in the grid, leading to a value of
$96$. As per theorem 2 the MDP based solvers are sub-optimal by the sum of the
expected value of information and the reward achieved by the informative
action, $61.875$.

As is evident, MDP-POMDP solvers perform sub-optimally at the budgeted, multi-
resolution,information gathering task and fail to leverage the agility of
UAVs, making them unsuitable for such applications. In the next section we
discuss a few potential methods to alleviate this problem.

##  VI Discussion and Future Work

Existence of informative actions, section IV, are symptomatic of unsuitability
of MDP-POMDP solvers for solving a POMDP. Identification of such actions
without the need to solve the full POMDP, can help avoid implementation effort
to discover the unsuitability of MDP-POMDP solvers and make better design
choices. Both the famous tiger problem and the multi-resolution information
gathering problems are unsuitable to solve via MDP-POMDP solvers.

However, if MDP solvers itself operate in information spaces and the reward
function is shaped to encourage gathering of information [23, 24, 25, 26, 27,
28, 29, 30], MDP-POMDP solvers can still effectively address problems with
informative actions [22]. We propose to implement such a pipeline and
demonstrate its salient and failure points as compared to vanilla MDP-POMDP
solvers in the future.

## References

  * [1] F. J. Beutler, “Dynamic programming: Deterministic and stochastic models (dimitri p. bertsekas),” _SIAM Review_ , vol. 31, no. 1, p. 132, 1989.
  * [2] S. M. LaValle, _Planning algorithms_. Cambridge university press, 2006.
  * [3] M. L. Littman, T. L. Dean, and L. P. Kaelbling, “On the complexity of solving markov decision problems,” in _Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence_ , ser. UAI’95. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1995, pp. 394–402. [Online]. Available: http://dl.acm.org/citation.cfm?id=2074158.2074203
  * [4] S. Choudhury, S. Arora, and S. Scherer, “The planner ensemble: Motion planning by executing diverse algorithms,” in _Robotics and Automation (ICRA), 2015 IEEE International Conference on_. IEEE, 2015, pp. 2389–2395.
  * [5] B. Bakker, Z. Zivkovic, and B. Krose, “Hierarchical dynamic programming for robot path planning,” in _Intelligent Robots and Systems, 2005.(IROS 2005). 2005 IEEE/RSJ International Conference on_. IEEE, 2005, pp. 2756–2761.
  * [6] P. Trautman and A. Krause, “Unfreezing the robot: Navigation in dense, interacting crowds,” in _Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on_. IEEE, 2010, pp. 797–803.
  * [7] M. Pivtoraiko and A. Kelly, “Kinodynamic motion planning with state lattice motion primitives,” in _Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on_. IEEE, 2011, pp. 2172–2179.
  * [8] P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng, “An application of reinforcement learning to aerobatic helicopter flight,” in _Advances in neural information processing systems_ , 2007, pp. 1–8.
  * [9] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,” _Artificial intelligence_ , vol. 101, no. 1-2, pp. 99–134, 1998.
  * [10] C. Papadimitriou and J. N. Tsitsiklis, “The complexity of markov decision processes,” _Math. Oper. Res._ , vol. 12, no. 3, pp. 441–450, Aug. 1987\. [Online]. Available: http://dx.doi.org/10.1287/moor.12.3.441
  * [11] G. Shani, J. Pineau, and R. Kaplow, “A survey of point-based pomdp solvers,” _Autonomous Agents and Multi-Agent Systems_ , vol. 27, no. 1, pp. 1–51, 2013\.
  * [12] S. Ross, J. Pineau, S. Paquet, and B. Chaib-Draa, “Online planning algorithms for pomdps,” _Journal of Artificial Intelligence Research_ , vol. 32, pp. 663–704, 2008.
  * [13] I. Little, S. Thiebaux, _et al._ , “Probabilistic planning vs. replanning,” in _ICAPS Workshop on IPC: Past, Present and Future_ , 2007\.
  * [14] S. W. Yoon, A. Fern, R. Givan, and S. Kambhampati, “Probabilistic planning via determinization in hindsight.” in _AAAI_ , 2008, pp. 1010–1016.
  * [15] M. L. Littman, A. R. Cassandra, and L. P. Kaelbling, “Learning policies for partially observable environments: Scaling up,” in _Machine Learning Proceedings 1995_. Elsevier, 1995, pp. 362–370.
  * [16] S. Javdani, S. S. Srinivasa, and J. A. Bagnell, “Shared autonomy via hindsight optimization,” _arXiv preprint arXiv:1503.07619_ , 2015.
  * [17] M. C. Koval, N. S. Pollard, and S. S. Srinivasa, “Pre-and post-contact policy decomposition for planar contact manipulation under uncertainty,” _The International Journal of Robotics Research_ , vol. 35, no. 1-3, pp. 244–264, 2016\.
  * [18] M. C. Koval, D. Hsu, N. S. Pollard, and S. S. Srinivasa, “Configuration lattices for planar contact manipulation under uncertainty,” _arXiv preprint arXiv:1605.00169_ , 2016.
  * [19] A. Tamar, G. Thomas, T. Zhang, S. Levine, and P. Abbeel, “Learning from the hindsight plan – episodic mpc improvement,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2017, pp. 336–343.
  * [20] S. Choudhury, A. Kapoor, G. Ranade, and D. Dey, “Learning to gather information via imitation,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2017, pp. 908–915.
  * [21] T. Zhang, G. Kahn, S. Levine, and P. Abbeel, “Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search,” in _Robotics and Automation (ICRA), 2016 IEEE International Conference on_. IEEE, 2016, pp. 528–535.
  * [22] S. Choudhury, M. Bhardwaj, S. Arora, A. Kapoor, G. Ranade, S. Scherer, and D. Dey, “Data-driven planning via imitation learning,” _arXiv preprint arXiv:1711.06391_ , 2017.
  * [23] S. Arora and S. Scherer, “Randomized algorithm for informative path planning with budget constraints,” in _Robotics and Automation (ICRA), 2017 IEEE International Conference on_. IEEE, 2017, pp. 4997–5004.
  * [24] ——, “Pasp: Policy based approach for sensor planning,” in _Robotics and Automation (ICRA), 2015 IEEE International Conference on_. IEEE, 2015, pp. 3479–3486.
  * [25] S. Arora, S. Choudhury, D. Althoff, and S. Scherer, “Emergency maneuver library-ensuring safe navigation in partially known environments,” in _Robotics and Automation (ICRA), 2015 IEEE International Conference on_. IEEE, 2015, pp. 6431–6438.
  * [26] M. Chen, E. Frazzoli, D. Hsu, and W. S. Lee, “Pomdp-lite for robust robot planning under uncertainty,” in _Robotics and Automation (ICRA), 2016 IEEE International Conference on_. IEEE, 2016, pp. 5427–5433.
  * [27] C. Papachristos, S. Khattak, and K. Alexis, “Autonomous exploration of visually-degraded environments using aerial robots,” in _2017 International Conference on Unmanned Aircraft Systems (ICUAS)_ , June 2017, pp. 775–780.
  * [28] A. Bircher, K. Alexis, U. Schwesinger, S. Omari, M. Burri, and R. Siegwart, “An incremental sampling-based approach to inspection planning: the rapidly exploring random tree of trees,” _Robotica_ , vol. 35, no. 6, p. 1327–1340, 2017.
  * [29] T. Cieslewski, E. Kaufmann, and D. Scaramuzza, “Rapid exploration with multi-rotors: A frontier selection method for high speed flight,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Sept 2017, pp. 2135–2142.
  * [30] G. Costante, C. Forster, J. Delmerico, P. Valigi, and D. Scaramuzza, “Perception-aware path planning,” _arXiv preprint arXiv:1605.04151_ , 2016\.