11institutetext: University of Alberta, Canada22institutetext: Alberta Machine
Intelligence Institute (Amii), Canada33institutetext: Kyushu University,
Japan44institutetext: Fujitsu Limited, Japan

# An Exploratory Study of AI System Risk Assessment from the Lens of Data
Distribution and Uncertainty

Zhijie Wang 11 Yuheng Huang 11 Lei Ma Corresponding author112233
Haruki Yokoyama 44 Susumu Tokumoto 44 Kazuki Munakata 44

###### Abstract

In the past few years, deep learning (DL) has become a driving force and has
been widely adopted in many domains and applications with competitive
performance. In practice, to solve the nontrivial and complicated tasks in
real-world applications, DL is often not used standalone, but instead
contributes as a piece of gadget of a larger complex system. Although there
comes a fast increasing trend to study the quality issues of deep neural
networks (DNNs) at the model (or unit) level, few studies have been performed
to investigate the quality and reliability of DNNs at both the unit level and
the potential impacts on the system level. More importantly, there is also a
lack of systematic investigation on how to perform the risk assessment for
data-driven AI systems from unit level to system level.

To bridge this gap, this paper initiates an early exploratory study of AI
system risk assessment from both the data distribution and uncertainty angles
to address these issues. We propose a general framework and an exploratory
study for analyzing AI systems. We first collect four representative AI
systems from different industrial domains (_i.e_., autonomous driving, object
tracking, speech recognition, dialogue) and utilize different common
corruption patterns accordingly to enable the analysis of each AI system’s
robustness. Then, we explore the potential usefulness of two out-of-
distribution (OOD) detection methods and two general uncertainty estimation
metrics for AI systems. Based on this, we continue to leverage the OOD and
uncertainty information collected from each individual module to investigate
the error propagation effects inside the AI systems. Finally, we make an early
attempt to explore simple combination strategies of OOD and uncertainty
information to improve the reliability of AI systems. After large-scale
(_i.e_., 700+ experimental configurations and 5000+ GPU hours) experiments and
in-depth investigations, we have reached a few key interesting findings: (1)
common corruption patterns can still pose threats to AI system’s robustness,
(2) OOD detection and uncertainty estimation techniques are still effective in
analyzing different modules inside AI systems, (3) certain modules of an AI
system could have a strong correlation to the system output error, and (4)
even simple strategies of combining OOD- and uncertainty-analysis can
significantly improve the quality and reliability of AI systems (e.g., system
performance against corrupted data improved by up to 38%). Our results
highlight the practical need for more in-depth investigations into AI systems
and the industrial demands for new quality assurance and risk assessment
techniques for AI systems.

###### Keywords:

AI system, deep Learning, uncertainty estimation, out-of-distribution
detection, risk assessment

##  1 Introduction

Over the past decade, deep learning (DL) has been a driving force behind the
success of many intelligent systems, across various industrial applications
and domains, _e.g_., face recognition [83, 19], dialogue system [87, 45],
speech recognition [76, 37], autonomous driving [15, 21], object tracking [44,
84]. Some of these systems have already been widely deployed into real-world
environments that impact our society and daily life [68, 65].

With the excitement of celebrating the achievement to approach the human
competitive level of intelligence by these data-driven AI systems, recent
concerns are raised regarding the quality and reliability of these AI systems
[1, 2], especially in the context of safety-critical domains and applications,
_e.g_., autonomous driving [65]. Different from traditional rule-based
software, the data-driven nature of the DL fundamentally changes the software
development paradigm, in which the decision logic is automatically learned
from the data instead of being directly coded by a human developer. However,
the probabilistic nature and lacking interpretability [49] make the quality
and risk assessment of Deep Neural Networks (DNNs) rather difficult, even at
the unit (model) level.

Recently, various techniques [64, 52, 38, 90, 53, 51, 100, 78, 96, 82, 73, 95,
93, 89] have been proposed to address DNNs’ quality assurance and risk
assessments at the unit level. For example, quite a few studies(_e.g_., [64,
52, 38, 90, 53, 51, 100, 78, 96]) have been working on the DNN testing, which
aims to identify more inputs that a DNN can not handle before deploying it.
Meanwhile, some progress (_e.g_., [82, 73, 95, 93, 89]) has also been made on
the DNN debugging and repairing, which tries to analyze why a DNN fails to
make a correct prediction and then repair this DNN model accordingly. In
parallel, there is another line of work on DNNs’ risk assessments, where two
representative types of techniques exist: (1) out-of-distribution (OOD)
detection and (2) uncertainty estimation. OOD detection assesses the model’s
prediction risk from the data distribution angle. This is in line with the
fundamental assumption of machine learning in that only on future unseen data
that share a similar distribution with the training data, the learned machine
learning model could exhibit a certain level of prediction accuracy. However,
in practice, the test data might now always fit well with the training data in
terms of distribution, due to various reasons, e.g., data collection,
operational environment changes. OOD detection could be helpful to identify
and filter out those potentially risky inputs that might fall out of the
training data distribution, on which the model could be incapable. In
particular, OOD detection techniques [33, 47, 36, 50, 41, 47, 38] are proposed
to capture the relation of data instances against the distribution of training
data. Based on this, developers can interpret how different the test data is
compared with training data to assess the potential risk of the model’s
prediction on test data. Different from OOD detection, uncertainty estimation
techniques [14, 23, 56, 80, 98] are also proposed to assess prediction risks
from various sources of uncertainty, e.g., a simple and effective way to
estimate the uncertainty for models’ prediction via Bayesian inference. By
interpreting the uncertainty score, developers can be aware of the confidence
in a model’s prediction in terms of uncertainty, before making the decision to
accept or reject the model’s prediction.

Although these DNN quality assurance and risk assessment techniques have been
shown to be useful at the unit (model) level. To the best of our knowledge,
limited studies have been on the risk assessment of DNN-enabled AI systems at
the system level. So far, to the best of our knowledge, it is even not clear
what could be a promising direction for quality assurance and risk assessment
of AI systems, and whether existing techniques at the unit level could still
work at a higher system level, and to what extent. In practice, real-world AI
applications are usually not only based on a single standalone DNN model.
Instead, one or multiple DNN models are often used as the components, forming
the parts of a larger system [102, 3, 99, 68] that often also contain the
traditional software components. Some simplified common patterns that an AI
system constitutes could be:

  * •

Multiple ML/DL models are designed to process input in parallels, the results
of which are aggregated as the system to solve one specific task (_e.g_., Fig.
2 (A)).

  * •

Multiple ML/DL models are used to form a task pipeline, the input of which
ML/DL model could depend on a former one, together solving a complex task
(_e.g_., Fig. 2 (B)).

  * •

ML/DL models are integrated with traditional software components with complex
internal interactions (_e.g_., Fig. 2 (C) and (D)).

A recent study [65] has shown that a real-world AI system, _e.g_., Baidu
Apollo autonomous driving system [4] that includes 30+ ML/DL models and 10+
traditional software components, can be as complex as covering all these
circumstances. The complexity of AI systems hence brings more challenges to
their analysis, compared to the model level. How to understand and analyze the
behavior of individual modules as well as different modules’ interactions
inside a complex AI system, and how to perform the risk assessment could be
some of the key pillars towards building reliable and trustworthy AI systems.
However, to the best of our knowledge, there is no extensive study on
analyzing the AI systems at the system level. In particular,

  * •

AI systems often have limited availability. Although DL techniques and AI
systems built with such techniques have been used in various domains for a
long period, most AI systems used by industry could be intellectual properties
and not open-source. As the key artifacts of an AI system, training data
played an essential role in the development and decision logic learning of
models, however, is often not publically available due to commercial and
privacy restrictions. Therefore, collecting open-sourced AI systems for the
study is not easy, especially for those with training data.

  * •

No general techniques have been established for the analysis of complex AI
systems at the system level. Existing work of quality assurance mainly focuses
on analyzing a single DNN (unit-level). However, there still lack study to
investigate their effectiveness in analyzing a complex AI system at the system
level.

  * •

Evaluating the AI system’s capability in the wild could be challenging. AI
system’s performance could significantly drop after being deployed to real-
world environments due to the data distribution shift between training data
and test data [39]. Therefore it is important to understand the potential
impacts from the operational environment to the system prediction capability
and risks. However, collecting and labeling new data for an existing system
with pre-trained ML/DL models could be both hard and time-consuming.

Therefore, towards addressing these issues and enabling further research on AI
systems, in this paper, we propose an exploratory study of AI system risk
assessment. As an early attempt, this paper performs analysis and risk
assessment of an AI system through characterizing data distribution and
uncertainty inside the system. To be specific, we first investigate a wide
range of AI-enabled systems (mainly based on the criteria of artifact
availability) and come to four representative ones from different tasks and
data domains: ConvLab-2 [102], SpeechBrain [68], Udacity [3], and ByteTrack
[99]. Since collecting and labeling new data could be challenging, we make use
of common corruption patterns in different domains’ data to synthesize real-
world corrupted datasets for evaluating the AI system’s performance (_i.e_.,
robustness) in the wild. Then, to utilize the existing OOD detection
techniques and uncertainty estimation more generally, we propose two general
OOD detection and two general uncertainty estimation metrics techniques.
Overall, we mainly investigate the following research questions and identify
several challenges and potential opportunities for AI system’s risk
assessment: ➊ What are the potential impacts of common corruption patterns on
AI systems in the wilds? We first evaluate AI systems’ performance degradation
against different real-world corruption patterns. Our experimental results
show that these corruption patterns could significantly affect AI systems’
performance, indicating an urgent need for addressing AI systems’ robustness
issues, which could be of great importance, especially for those AI systems
that are designed to be deployed in the wild with various noisy factors in the
operational environment. ➋ What is the OOD awareness of different modules in
AI systems? We continue to study the effectiveness of two proposed general OOD
detection methods based on the corrupted datasets by comparing the OOD scores
on clean and corrupted data. Our experimental results reveal that the proposed
OOD detection methods could detect data distribution changes, showing the
potential for assessing the AI system’s prediction risks. Our investigation
results also reveal that different modules in an AI system could have
different sensitivities to OOD data. ➌ How does the uncertainty estimation
perform on different modules in AI systems? In addition to OOD detection, we
further study the effectiveness of two proposed general uncertainty estimation
metrics on different AI systems by comparing clean data’s uncertainty scores
and corrupted counterparts. We found that the proposed metrics could well
reflect the AI system’s prediction uncertainty on the corrupted datasets
through large-scale experiments. Furthermore, we also find that one module’s
uncertainty score change does not always correlate to another module’s
uncertainty score change in an AI system. ➍ How do the errors propagate among
modules in AI systems? In practice, when AI systems fail to make correct
predictions, it is often of great importance to analyze to find out the
potential reasons, _e.g_., whether a single module has triggered the bug or if
there is any erroneous information propagation inside the AI system. Based on
the OOD detection and uncertainty estimation results from RQ2 & 3, we further
perform an in-depth analysis of how each module’s OOD and uncertainty
information in an AI system has correlated to the output error. Overall, we
find that not every module’s OOD/uncertainty information strongly correlates
to the system output error. ➎ How do simple combinations of OOD analysis- and
uncertainty-based method improve the reliability of AI systems? Finally, we
make an early exploratory investigation on whether using OOD score or
uncertainty information could assess an AI system’s prediction risks, and
further reduce the system prediction risks by rejecting those uncertain and
unreliable cases. Surprisingly, our results indicate that even a simple
combination strategy could enhance the system performance by up to 38%.

In summary, this paper makes the following contributions:

  * •

We design a research process for the analysis and risk assessment of AI
systems at the system level. Then we conduct an exploratory study on several
AI systems based on this process. In order to analyze system quality and
reliability, we leverage common corruption patterns from different domains to
generate and prepare multiple large-scale corrupted datasets to facilitate our
study.

  * •

We study the internal mechanism (_i.e_., interactions among different modules
inside a system) and the external mechanism (_i.e_., the system’s performance
against corrupted data) of AI-enabled systems from the lens of data
distribution and uncertainty through large-scale experiments (total
experimental time is more than 5,000 GPU hours).

  * •

As an early attempt, we utilize five simple strategies of combining OOD- and
uncertainty-analysis to make risk assessments for AI systems.

  * •

Based on our study results and in-depth analysis, we further propose
discussions on the implications of our exploratory study as well as several
future research opportunities towards building reliable and trustworthy AI
systems, _e.g_., system-level quality assurance, fault localization in AI
system.

To the best of our knowledge, this is an early paper that initiates an
exploratory study on analyzing complex AI systems at the system level. Our
empirical study results reveal the current status of analyzing AI systems,
demonstrating multiple demands and potential research opportunities around AI
systems. The results of this study call the attention of researchers and
industrial practitioners, that similar to the traditional software system,
there could be many challenges and potential issues that need to be addressed
at a higher level of the AI-enabled systems, instead of only focusing on the
unit (model) level, where most existing work centers around. We hope our work
could enable better understanding, establish the basis, and pave the path
towards further quality assurance research to build reliable and trustworthy
AI systems.

##  2 Related Work

In this section, we discuss previous work that could be most relevant to this
paper, _i.e_., unit-level AI module analysis, along with three directions: (1)
out-of-distribution detection, (2) uncertainty estimation, and (3) testing,
debugging and repairing of DNNs. Making risk assessments for a standalone DNN
has been studied for many years. Existing work that is relevant to our study
mostly focuses on a single neural network with one specific task like image
classification. However, few works have been made for AI systems involving
multiple AI-enabled modules as well as traditional software. In the following,
we survey related work for unit-level AI module analysis and try to find those
applicable for analyzing complex AI systems.

###  2.1 Out-of-distribution detection.

One fundamental assumption of machine learning is that training and test data
share a similar distribution. In other words, machine learning often does not
make any guarantee for those data that fall out of the distribution of
training data. Therefore, DNNs often fail to offer correct predictions for
out-of-distribution (OOD) data and can even provide high-confidence
predictions even if they’re incorrect [27, 8]. OOD detection aims to alleviate
this problem by detecting test data that are distributionally different from
the training data. Generally speaking, OOD detection seeks to compute an OOD
score for a given input instance [61], and if, it is below a defined
threshold, it is in-distribution (ID) data. Otherwise, it is out-of-
distribution (OOD) one [58]. The score computation is thus important for OOD
detection, and it can be roughly categorized into two types: (1) methods
requiring auxiliary data, and (2) methods that can be applied for pre-trained
models without extra data.

If the model is accessible at the training stage and the training data are
given, regularization specifically designed for OOD detection can be employed
so that the OOD score obtained from the model can better handle OOD cases [26,
56, 41, 61, 34]. However, in this study, as we aim to analyze existing AI
systems before deployment, redesigning the models or retraining the models is
beyond our scope, making such OOD approaches inappropriate for our study.

There is also some work focusing on analyzing pre-trained DNN models.
Hendrycks  _et al_. [33] first establishes a common OOD detection baseline by
directly analyzing the softmax probability at the last layer. Along the line,
there are multiple methods trying to improve OOD detection [47, 36, 50]. One
major limitation of these methods is that they specifically focus on the image
classification problem only, while the systems we study involve multiple input
domains and have various tasks. There is also some work trying to extract
hidden features from intermediate layers of DNNs. Lee  _et al_. [42] propose
to use Mahalanobis distance to measure intermediate hidden features of ID data
and OOD data. Kim  _et al_. [38] propose Surprise Adequacy in which kernel
density estimation is used to estimate the probability density function of ID
data and OOD data. Once the estimation is done, we can know how surprised a
model is for a given input and thus discriminate ID and OOD cases. Although
they originally targeted classification tasks, these two metrics can be
generalizable to either regression or other tasks as well since they pay
attention to the intermediate output instead of classification confidence at
the last layer. In our work, we apply Surprise Adequacy and kernel density
estimation to identify possible OOD cases at test time. However, we would like
to emphasize that OOD cases are just one aspect of risk assessments in our
study. There still exist possible ID (in-distribution) test instances while
the systems make erroneous predictions.

###  2.2 Uncertainty estimation techniques.

Estimating uncertainty inherent in DNNs can help developers understand how
much confidence can be put in the models. Uncertainty analysis can help
identify not only OOD cases [79] but also probably erroneous predictions for
ID cases [14]. In the past few years, uncertainty quantification methodologies
have been studied extensively for DNNs [24, 6]. Gawlikowsk  _et al_. [24]
categorized these methods into four types: single deterministic methods,
Bayesian methods, ensemble methods and test-time augmentation methods.

Single deterministic and ensemble methods require extra attention for
uncertainty measurement early at the model architecture design stage. The
former involves training a model explicitly to quantify uncertainties [71, 56]
or using additional components to estimate the uncertainty [66, 67]. The
latter needs the inference result of several different networks trained for
one task to quantify the uncertainty [40, 59, 30]. As our goal is to measure
the uncertainty of existing complex AI systems, we do not assume there is any
explicit design for uncertainty or ensemble in these systems, and thus these
methods are unsuitable.

In contrast, the test-time augmentation and Bayesian methods can both be non-
parametric and extensible. Test-time augmentation usually generates $T$
augmented examples per input and feeds them to DNN so that a predictive
distribution can be obtained to measure the uncertainty. The augmentation can
be achieved through traditional data augmentation techniques (e.g., image
rotation, translation). While this augmentation is easy to implement, it is
usually used in the image domain [81, 10] and the quality of the generated
images can be hard to control if the changes are made inappropriately [72].
For example, rotating or translating an image might result in the main part of
an image being outside of the image’s bounds. Adding too much noise might
block the main part of an image, and the augmented image is thus invalid since
even humans cannot easily interpret it. As we are studying AI systems with
different input domains, this kind of method is not generalizable as well.
Bayesian methods estimate uncertainty inherent in inference by predicting the
output distribution rather than offering a deterministic result. Although
Bayesian modelling often requires Bayesian Neural Networks (BNNs) for the
uncertainty measurement, Gal  _et al_. [22] propose an easy yet effective
technique called Monte Carlo Dropout (MC Dropout) to approximate Bayesian
inference for any neural networks with dropout layers. This measurement
calculates uncertainty estimates by enabling dropout layers at test time and
approximating variational inference. As it is applicable for a wide range of
DNNs and has been proved successful in identifying adversarial and
misclassified examples [97, 98, 54], we believe it is more suitable for our
study and thus take it as the primary approach to measure uncertainty.
Although MC dropout is applicable for any DNNs with dropout layers, no
previous work uses it to access the uncertainty for classical logical
components inside AI systems and how this impacts the system behavior.
However, this kind of component plays a crucial role in some AI systems like
tracking and dialogue generation. Our study performs the first in-depth
analysis of the uncertainty measurement of these components and their
potential impacts on the whole system.

###  2.3 Testing, debugging and repairing of DNNs

In recent years there has been an ongoing effort to improve the quality and
robustness of DNNs from the perspective of software engineering. DNN testing
aims to reveal the vulnerabilities of DNNs before deployment. Instead of
solely relying on the accuracy of test samples to determine whether the models
are ready to deploy, various techniques have been proposed to probe the
boundaries of DNNs by looking inside the black boxes. For example, a series of
coverage criteria specifically designed for DNNs have been proposed to guide
the testing and help understand the behavior of DNNs [64, 52, 38]. Based on
this, some testing techniques are developed to synthesize test inputs to
thoroughly test the DNNs [90, 53, 51]. Following this line of work, multiple
test case generation techniques for specific application scenarios are
proposed [100, 78, 96]. One major difference between our work and theirs is
that we study DNNs across different domains at the system level, while they
mainly focus on unit-level AI modules under certain scenarios. DNN debugging
and repairing move one step further, trying to repair DNN models after finding
why they fail to make correct predictions. Some previous work proposes to
change model architecture directly. For example, Wang _et al_. [82] repair the
model by integrating an additional layer to pre-process error inputs, Sotoudeh
_et al_. [73] and Zhang _et al_. [95] correct DNNs by changing network
weights. From another perspective, error-guided data augmentation can also
help repair through re-training/fine-tuning. For example, DeepRepair [93] and
RNNRepair [89] synthesize new training data by analyzing errors made by the
trained model. Our work is orthogonal to them in two aspects—first, our work
focus on the risk assessment at the system level instead of the module level.
Instead of proposing a method to improve the robustness of DNNs, we aim to
reveal the effectiveness and limitations of current approaches and identify
the possible challenges to guide future research at the AI system level.

##  3 Overview of Study Design

In this section, we introduce the study design in this paper. Figure 1 shows
the overview of workflow our proposed framework and its three major
components. The first step is the collection of AI systems (the left-side of
Fig. 1). As discussed in Sec. 1, collecting suitable AI systems for our study
could be challenging. Therefore, we mainly focus on selecting AI systems
covering various tasks and applications in our study. The middle part of Fig.
1 illustrates this study’s analysis techniques. According to domains of
selected AI systems, we first use common corruption patterns to synthesize
large-scale corrupted datasets, then we use risk assessment techniques from
two different aspects: (1) out-of-distribution (OOD) detection and (2)
uncertainty estimation. The last component (the right-side of Fig. 1)
summarizes the research questions to be investigated in our study. With the
corrupted datasets, we first examine how AI systems can be risky in the wilds
(RQ0). Since the corrupted datasets are usually with shifted distribution, we
then investigate the effectiveness of OOD detection in the AI systems (RQ1),
followed by the effectiveness of uncertainty estimation (RQ2). These two
techniques form the basis for our next step’s investigation of error
propagation in the AI systems (RQ3) and make it possible to make an early
attempt on the AI system’s risk assessments (RQ4).

Now we summarize the design of each part in our study in Sec. 3.1, 3.2, and
3.3, respectively. We also introduce the details of our experimental
configurations in Sec. 3.4.

Figure 1: Overview of our study workflow.

###  3.1 Subject AI Systems

In real-world applications, multiple AI models are usually combined with
traditional software to together solve complicated tasks as a complex system.
These AI-enabled systems are explicitly designed for different tasks and
application scenarios. To conduct a relatively comprehensive study of these
AI-enabled systems, we try to cover a wide range of tasks including (1)
autonomous driving, (2) object tracking, (3) speech recognition, and (4)
dialogue system. Fig. 2 shows the simplified workflows of each subject system.
These four tasks also cover the three different data domains: image, text, and
audio. As most AI systems are treated as private intellectual properties and
are kept confidential (especially for the artifact of training data), it could
be rather challenging to collect many subject AI systems for our study. In
this paper, we mainly collect systems that (1) have been used in previous work
and (2) have promising performance on related tasks, (3) have their key
artifacts publicly available. Specifically, we select subject AI systems from
different domains based on the following criteria:

  * •

Multi-module. An AI system should have multiple modules instead of a single AI
model, where at least one module should be an AI-enabled one.

  * •

Open-source. An AI system must be open-sourced so that we can access AI models
(_e.g_., training program, model architecture) inside it to quantitatively
evaluate the output of every single module in the system.

  * •

Documentation. An AI system must have comprehensive documentation so that we
can understand its system structure and pipelines.

  * •

Data availability. An AI system must have corresponding publicly available
data (_e.g_., training data and test data).

  * •

Pre-trained model availability. An AI system should have pre-trained models
available. With the pre-trained models released by authors, we can ensure the
performance of each AI system and reduce the reproduction efforts.

  * •

Feasible system complexity. As for an early exploratory study, the system
should not be too complex to make the evaluation feasible and reasonably
achievable, from the perspectives of our available computational resources and
time.

Next, we briefly introduce each of our subject AI systems.

Figure 2: Simplified workflows of four subject AI systems used in this study.

####  3.1.1 Autonomous Driving System

In recent years autonomous driving has attracted much attention in both
academia and industry. One of the major tasks of autonomous driving is
steering angle control. NVIDIA first proposed a simple CNN to solve the
problem in 2016 [16], representing one of the early works to bring the power
of DNNs to autonomous driving. Afterwards, they publish a report demonstrating
a more sophisticated system called PilotNet [15]. Along the line, there are
many efforts to tackle the autonomous steering problem [20, 17, 92, 35, 91].

The system we choose: Udacity. Udacity system [3] is derived from a well-known
self-driving competition Udacity self-driving challenge 2. The goal of this
task is to predict autonomous driving steering angles. Specifically, it takes
two consecutive images as input and outputs the steering angle. However, the
control decisions made by the systems built on this challenge are independent
across different frames, and as a result, the dependencies between adjacent
images are ignored. Because of this, we can take them as systems in image
processing. In this work, we select the “Rambo” system (which ranked 2nd in
this competition) as the candidate system for analysis. This system is a
parallel-executed system where three separately trained CNNs (see Fig. 2 (A))
process two consecutive images individually and predict the steering angle
value independently. After then, their prediction results are merged as the
final output by taking average. The three CNNs are trained separately and
therefore we consider this as a minimal and simple example of an AI system as
it would be interesting to see what the results might be for simple and
complex cases of an AI system in this study. The performance of the Udacity
system is measured by root mean square error (RMSE, Eq. 1) between the
predicted steering angle value and ground truth value.

$RMSE=\sqrt{\frac{\sum_{i=1}^{N}(\hat{y}_{i}-y_{i})^{2}}{N}}$ (1)

where $\hat{y}_{i}$ and $y_{i}$ denote the predicted and ground truth steering
angle value, respectively. $N$ denotes the total number of images.

####  3.1.2 Object Tracking System

Multi-object tracking (MOT) is one of the popular domains nowadays in the
object tracking field. It involves automatically identifying multiple objects
in a video and often requires one module to detect the objects and another
module to track the objects. The detector is often a powerful object detection
model [48, 101, 69, 25], while the tracker can be either DNN-based [88, 46] or
solely powered by traditional software [99, 13].

The system we choose: ByteTrack. ByteTrack system [99] is designed for object
tracking, which has two serial-executed modules: (1) object detector, and (2)
object tracker. The object detector takes two consecutive images as input and
outputs bounding boxes for object instances detected in each image. The object
tracker then takes the bounding boxes detected in two images as input and
outputs the association between the two images’ bounding boxes. Unlike the
Udacity challenge systems, ByteTrack explicitly records the states of multiple
objects across all frames in order to precisely track each of them in the
video. As the cross-frame dependencies are considered, the target system is
thus more complex, representing systems in the video processing. Note that
when using DeepSORT as the tracker in the ByteTrack system, the tracker also
takes images as input (dashed line in Fig. 2 (B)). The performance of the
ByteTrack system is measured by multiple object tracking accuracies (MOTA, Eq.
2) [12].

$MOTA=1-\frac{\sum_{t}(m_{t}+fp_{t}+mmme_{t})}{\sum_{t}g_{t}}$ (2)

where $m_{t}$, $fp_{t}$ and $mme_{t}$ are the number of misses (the tracker
misses an object), of false positives (the tracker reports non-existing
objects) and of mismatches (the tracking result between two bounding boxes is
incorrect) respectively for time $t$. $g_{t}$ denotes the number of
groundtruth bounding boxes tracked at time $t$. For two consecutive frames,
the equation first computes the error rate by taking three different possible
errors (i.e., miss, false positive, and mismatch) into account over the number
of objects, then performs “1-error rate” to calculate the MOTA.

####  3.1.3 Speech Recognition System

Speech recognition is to identify an acoustic input sequence and convert it
into a text transcript. Early in 2011, Dahl _et al_. [18] proposed (CD)-DNN-
HMM as an end-to-end system that involves DNN to do the Automatic Speech
Recognition (ASR) task. Since then, using deep learning as a key component in
ASR systems has been widely studied. Deepspeech series [31, 7], as an end-to-
end method, has achieved great success and is still widely adopted now. As an
alternative, employing the encoder-decoder structure with a classifier can be
more modularized [29, 28, 85, 11]. We prefer this solution as we can study the
relationship between modules more easily.

The system we choose: SpeechBrain. SpeechBrain system [68] is a toolkit for
processing multiple speech-oriented tasks. It offers various kinds of ASR
systems with two-stage architectures (as shown in Fig. 2 (C)) where the first
module process the speech spectrum, and the second module transforms the
processed spectrum into text. The input to SpeechBrain is audio which involves
more complex pre-processing and post-processing compared to image input. The
competitive performance of SpeechBrain makes it an ideal choice to study
systems in audio processing. The performance of the SpeechBrain system is
measured by word error rate (WER, Eq.3) between the predicted text and ground
truth transcript.

$WER=\frac{S+D+I}{N}$ (3)

where $S,D,I$ are the number of substitutions, the number of deletions, and
the number of insertions, respectively. $N$ is the number of words in the
reference. Note that WER can also be seen as an edit distance.

####  3.1.4 Dialogue System

Dialogue generation aims to generate a response given a post by the user
automatically in order to finish specific tasks, _e.g_., recommendation. A
dialogue system involves a complex Natural Language Processing (NLP) task. Wen
_et al_. [86] made an early attempt to use LSTM to solve it. Recently it has
experienced rapid growth. Nowadays, an AI-enabled dialogue system usually
includes: (1) a natural language understanding component [55, 43] which is
used to parse the users’ intent, (2) a dialogue policy [77], which decides the
response of the system, and (3) a natural language generation component [87]
which is used to transform dialogue acts into a natural language sentence.

The system we choose: ConvLab2. ConvLab2 system [102] is a dialogue system
that has four connected modules (Fig. 2 (D)): NLU (natural language
understanding), DST (dialogue state tracking), POL (dialogue policy), and NLG
(natural language generation). NLU is used to parse the user’s intent, takes
an utterance as input and outputs the corresponding dialogue acts. DST updates
the belief state, which contains the constraints and requirements of the user.
POL receives the belief state and outputs system dialogue acts. NLG transforms
dialogue acts into a natural language sentence. Note that the ConvLab2 system
is a dialogue recommendation system, where each dialogue is oriented with a
specific task, e.g., hotel booking. The performance of the ConvLab2 system is
measured by whether the system agent (who was receiving the call) successfully
performs a specific task in a dialogue (success rate, Eq. 4). Suppose the task
for a dialogue is hotel booking. If the agent successfully record the
information about hotel booking (e.g., preferred day and hotel name), then the
task is seen as successful.

$success~{}rate=\frac{|D|_{success}}{|D|}$ (4)

where $|D|_{success}$ and $|D|$ are the number of successful dialogues and
total dialogues, respectively.

###  3.2 Analysis Methodology

Our analysis methodology for AI systems consists of two parts (1) Synthesizing
real-world corrupted datasets by common corruption patterns of the specific
input domain. (2)Based on the pair-wise data of clean and corrupted data, we
then leverage two different aspects of risk assessment techniques: out-of-
distribution detection and uncertainty estimation as the second part of our
analysis.

Table 1: Summary of corruptions used in the proposed study and its corresponding parameters. Domain |  Corruption |  Effects |  Parameters
---|---|---|---
Image |  Gaussian noise (GN) |  Add Gaussian noise pattern on an image. |  $c\in\left\\{0.08,0.18,0.38\right\\}$, where $c$ is the scale of noise (variance of a zero-mean Gaussian distribution).
|  Motion blur (MB) |  Add motion blur effect on an image. |  $(c_{0},c_{1})\in\left\\{(10,3),(15,8),(20,15)\right\\}$, where $c_{0}$ and $c_{1}$ denote the radius and the scale of motion blur.
|  Snow (SN) |  Add snow effect on an image. |  $(c_{0},c_{1},c_{2},c_{3},c_{4})\in~{}${$(0.1,0.3,3,0.5,10,4,0.8)$, $(0.55,0.3,4,0.9,12,8,0.7)$, $(0.55,0.3,2.5,0.85,12,12,0.55)$}, where $c_{i}$ is the parameter for snow effects.
|  Contrast (CO) |  Change the contrast of an image. |  $c\in\left\\{0.4,0.2,0.05\right\\}$, where $c$ is the faxwctor when changing image’s contrast.
|  Pixelate (PI) |  Add pixelate effect on an image. |  $c\in\left\\{0.6,0.4,0.25\right\\}$, where $c$ is the factor of pixelization.
Audio |  Reverb (RE) |  Add reverb effect to an audio clip. |  $c\in\left\\{0.33,0.67,1.0\right\\}$, where $c$ is the wet-level of reverb effect.
|  Speed (SP) |  Change the speed of an audio clip. |  $c\in\left\\{0.8,1.5,2.0\right\\}$, where $c$ is the factor of speed.
|  Distortion (DI) |  Add distorion effect to an audio clip. |  $c\in\left\\{15,20,30\right\\}$, where $c$ is the level of distortion.
|  Background noise (BN) |  Add Gaussian white noise to an audio clip. |  $c\in\left\\{10,5,2\right\\}$, where $c$ is the target signal-to-noise ratio.
|  Pitch shift (PI) |  Shift the pitch of an audio clip. |  $c\in\left\\{2,4,6\right\\}$, where $c$ is the number of semitones when shifting an audio.
Text |  Digits2words (DW) |  Change the digits in a text corpus into corresponding words. |  $c\in\left\\{0.2,0.6,1.0\right\\}$, where $c$ is the probability to change.
|  Remove char (RC) |  Randomly ($p=0.1$) remove a character in a word in a text corpus. |  $c\in\left\\{0.15,0.45,0.75\right\\}$, where $c$ is the probability to change a word in a text corpus.
|  Change char (RC) |  Randomly ($p=0.1$) replace a character with similar one in a word in a text corpus. |  $c\in\left\\{0.15,0.45,0.75\right\\}$, where $c$ is the probability to change a word in a text corpus.
|  Misspelling (MS) |  Replace a word with its commonly misspelling one in a text corpus. |  $c\in\left\\{0.2,0.6,1.0\right\\}$, where $c$ is the probability to replace a word in a text corpus.
|  Swap (SW) |  Randomly swap two characters in a word in a text corpus. |  $c\in\left\\{0.15,0.45,0.75\right\\}$, where $c$ is the probability to swap a word in a text corpus.

####  3.2.1 Common corruption patterns.

Common corruption patterns [32] refer to patterns that (1) naturally exist in
the real world and (2) could significantly affect DNN’s performance. AI
models’ performance against such common corruption patterns could reflect
their robustness and help developers assess potential risks before deploying
to complex real-world environments. Therefore, this paper adopts a similar
idea to analyze AI systems by using different corruption patterns for data
from different domains. We give the introductions of corruption for each
domain of data in Table 1.

  * •

Image data corruption. Among 15 common image corruption patterns proposed by
Hendrycks _et al_. [32], we select the following five representative ones:
Gaussian noise (GN), motion blur (MB), snow (SN), contrast (CO), and pixelate
(PI). These five corruption patterns are selected from different categories in
order to make our selection diverse enough. GN and MB are common noise
patterns and blur effect, respectively. SN is related to a natural weather
effect while CO usually occurs due to the changes in light conditions. The
last one PI usually occurs during the image up-sampling process.

  * •

Audio data corruption. We select the following five audio corruption patterns:
reverb (RE), speed (SP), distortion (DI), background noise (BN), and pitch
shift (PS) from [62, 5]. RE adds reverb effect to an audio clip. SP changes
the speed of an audio clip. DI adds distortion effects to the audio clip. BN
adds Gaussian noise to the audio clip. PS shifts the pitch of an audio clip.

  * •

Text data corruption. We select the following five text corruption patterns
among 11 text corruption patterns proposed by Rychalska _et al_. [70]:
digit2words (DW), remove chars (RC), change chars (CC), misspelling (MS), and
swap (SW). DW replaces the numbers in a text corpus as corresponding words. RC
randomly removes a character of some words in a text corpus, while CC randomly
replaces a character with another. MS replaces words with common misspelling
ones. SW swap two characters in a word.

####  3.2.2 OOD detection technique.

As introduced in Sec. 2, many of the existing OOD detection techniques are
designed specifically for classification problems [33, 47, 36, 50]. However,
some of the related work relies on measuring the distribution difference of
the intermediate representations (i.e. hidden states in DNNs) to determine
whether the given input is OOD or not. As intermediate representations are
generic across classification and regression tasks, in this paper, we adopt
Surprise Adequacy [38] as our OOD detection technique, which is well suited
for measuring complex AI systems under our setting.

1

Input: An AI module $\mathbf{M}$, selected layer $L$, training data
$X_{train}$, test data instance $\hat{x}$, metric $g(\cdot)$

Output: OOD score $p$

2 $Z_{train}\leftarrow\emptyset$;

3 for  _$x_{i}~{}\leftarrow~{}X_{train}$_ do

4 $z_{i}\leftarrow\mathbf{M}(x_{i},L)$;

5 $Z_{train}=Z_{train}~{}\bigcup~{}\\{z_{i}\\}$;

6

7 end for

8$p\leftarrow g(Z_{train},\hat{x})$;

9 return _$p$_ ;

Algorithm 1 OOD detection using Surprise Adequacy [38]

We formally describe the OOD detection workflow in Algorithm 1. Specifically,
we first select a target feature layer $L$ of a DL model in the system and
collect all this layer’s output features $Z_{train}$ by feeding training data
into a trained model (Line 2 - 4). Then, for each new test data instance, we
measure the distance between its feature on $L$ and the distribution of
collected training features $Z_{train}$ using a specific metric $g(\cdot)$
(Line 6). In this work, we use two representative metrics to measure such
distribution distance, _i.e_., Mahalanobis distance (Maha-d) and kernel
distribution estimation (KDE) adopted from existing work [42, 38].

####  3.2.3 Uncertainty analysis.

We make use of the Bayesian methods with the following two uncertainty
metrics: Variation Ratio (VR) and Variation Ratio of Original prediction
(VRO). These two metrics are originally proposed for classification problems
[98]. Though metrics like standard deviation can be applied to simple
regression task, however, most AI systems focus on more complex problems,
e.g., object tracking (which could be a combination of classification and
regression). Hence, to use generalizable analysis techniques, we make
adaptions to the calculation of VR and VRO and define them as follows.

###### Definition 1

Extended variation ratio (VR)

$VR=1-\frac{\sum_{i=1}^{T}w*\frac{\sum_{j=1,j\neq
i}^{j=T}(1-dist(p_{i},p_{j}))}{T-1}}{T}$ (5)

where $T$ is the inference time, $dist(\cdot)$ represents the distance
function between two points. $p_{i}$ represents the inference result at $i$
inference time. $w$ is the weight.

###### Definition 2

Extended variation ratio for original prediction (VRO)

$VRO=1-\frac{\sum_{i=1}^{T}(1-dist(p_{i},p_{L_{M}}))}{T}$ (6)

where $T$ is the inference time, $dist(\cdot)$ represents the distance
function between two points. $p_{i}$ represents the inference result at $i$
inference time. $p_{L_{M}}$ is the prediction result from the original model
$M$.

We argue that these adaptions are necessary for our study. Unlike
classification problems, it’s hard to find two identical outputs through
multiple runs in other problems AI systems face, _e.g_., regression problems.
Therefore we use a distance function to measure the similarity between two
outputs instead of checking if the two outputs are numerically equal.

###  3.3 Research Questions

Figure 3: The workflow and relationship of research questions in the study
design in this paper.

We facilitate our study by investigating five different research questions .
Fig. 3 shows the logic flow in our design of research questions. We first
create large-scale corrupted datasets and investigate the potential risks
brought by these common corruption patterns in RQ0, serving as the basis of
this study. Then we discuss the effectiveness of two aspects’ risk assessment
techniques (_i.e_., OOD detection and uncertainty estimation) in RQ1 and RQ2
by leveraging the corrupted data generated in RQ0. After a deeper
understanding of these techniques, we further investigate their potential as
system error indicators in RQ3. Finally, we make an early attempt to combine
these techniques for making risk assessments on AI systems. We detail the
design of each research question in the following.

  * •

RQ0. How common corruption patterns increase risks in the wild for AI systems?
While common corruption patterns’ effects on a single AI model have been
widely studied, yet it’s still unclear whether common corruption patterns
could also affect AI systems’ performance or not. In this research question,
we first create 15 corrupted versions (5 corruption patterns $\times$ 3
severity levels) for each system’s corresponding dataset. Then, we evaluate
the performance of each AI system against different corruption patterns and
severity levels with related performance metrics as introduced in Sec. 3.1.

  * •

RQ1. What is the OOD awareness of different modules in AI systems? Based on
the corrupted dataset created in RQ1, we evaluate our OOD detection methods by
comparing the OOD score on clean data and corrupted data created in RQ1. We
first use the Wilcoxon signed-rank test to measure if the OOD score’s
distribution on clean and corrupted data is significantly different. Then, we
further measure if different AI modules are sensitive to different OOD data.

  * •

RQ2. How does the uncertainty information behave on different modules in AI
systems? Similar to RQ2, in RQ3 we evaluate our uncertainty estimation methods
by comparing the uncertainty score on clean data and corrupted data in this
research question. Specifically, we use Wilcoxon signed-rank test to measure
if the uncertainty score’s distribution on clean data and corrupted data are
significantly different. Then, we use Kendall’s $\tau$ test to measure if
there is any correlation between different modules’ uncertainty inside an AI
system.

  * •

RQ3. How do the errors propagate among modules in AI systems? In this research
question, we investigate the potential propagation of erroneous information
among modules in AI systems. Specifically, we use Kendall’s $\tau$ test to
measure if there is any correlation between different modules’ OOD/uncertainty
information and the system’s output error.

  * •

RQ4. How do simple combinations of OOD analysis- and uncertainty-based method
improve the reliability of AI systems? In this RQ, we make an early attempt to
combine OOD analysis and uncertainty information to make risk assessments of
AI systems by rejecting unreliable predictions through five different
strategies. Then, we evaluate systems’ performance improvement after utilizing
each strategy.

###  3.4 Experimental Configurations

In this section, we introduce the common configurations that are shared by
multiple RQs in our study, _e.g_., model and dataset configurations,
parameters of OOD detection and uncertainty estimation, and hardware and
software dependencies. We will further detail the specific configurations (if
any) of each individual research question in Sec. 4

####  3.4.1 Model & dataset configurations.

We summarize the selection of each AI system’s configurations and datasets for
evaluation in Table 2. A complex AI system could have multiple configurations
for each corresponding dataset, and each configuration could also have
different settings for both AI models and traditional software. In this study,
we choose all available configurations (with pre-trained models available) in
the Udacity, ByteTrack, and SpeechBrain systems. While in the ConvLab-2
system, we select four configurations with the highest success rate on clean
data. Note that we create fifteen corrupted versions (5 corruption patterns
$\times$ 3 severity levels) for each dataset. Therefore, the dataset size in
Table 2 refers to the total size of clean data and corrupted data.

Table 2: Model & corresponding dataset used for evaluating different AI systems System | Configuration | Dataset | Size of Dataset
---|---|---|---
Udacity | CNN & CNN & CNN |  Challenge2 [3] | 48,000 images
ByteTrack | YOLO+Track | MOT17 [57] | 3 hours video clips
YOLO+SORT
YOLO+DeepSORT
SpeechBrain | CRDNN+RNN |  |
CRDNN+Transformer | LibriSpeech [60] | 100 hours audio clips
CRDNN+RNN | CommonVoice-it [9] | 150 hours audio clips
Wav2Vec2+RNN
ConvLab2 | BERT+RULE | MultiWOZ [94] | 16,000 dialogues
MILU+RULE
BERT+PPO
MILU+PPO
Table 3: The choices of $dist(\cdot)$ function for different output formats in different systems. Output Format | $dist(\cdot)$ | Related System(s)
---|---|---
Numerical value | Standard deviation | Udacity
Numerical vector | Cosine distance | SpeechBrain
Detection box | IOU | ByteTrack
Tracklet |  MOTA [12] | ByteTrack
Text |  BLEU [63] | ConvLab-2 & SpeechBrain

####  3.4.2 OOD detection configurations & uncertainty estimation.

For OOD detection, each AI model could have multiple choices on the feature
layer. In our experiments, we select the penultimate layer (the layer before
the final output layer) as the target layer $L$, which is also adopted by
existing work [50]. For uncertainty estimation, we set the inference times of
Monte Carlo dropout as 10 and the dropout rate as 0.1 throughout our
experiments. Since different AI systems and modules might have different tasks
and output formats, we adjust the $dist(\cdot)$ function (Eq. 5 & 6) according
to the output format of a system or an individual module. For example, the
detector of a tracking system outputs bounding boxes, therefore we use IOU as
the $dist(\cdot)$ function. We summarize the $dist(\cdot)$ function used in
our study in Table 3 and briefly describe Cosine distance in eq. 7, IOU in eq.
8, MOTA in eq. 2, BLEU in eq. 9 and eq. 10. We also refer audience to Sec. 3.1
for the details of MOTA.

$\displaystyle Cos(\mathbf{x_{1}},\mathbf{x_{2}})$
$\displaystyle=\frac{x_{1}\cdot x_{2}}{||x_{1}||\times||x_{2}||}$ (7)

where $x_{1}$ and $x_{2}$ stand for the output (hidden representations in the
format of vectors) of the neural networks with respect to two data points.

$\displaystyle IOU$ $\displaystyle=\frac{|A\cap B|}{|A\cup B|}$ (8)

where $A$ and $B$ stand for the bounding box of two different predictions.
$IOU$ computes the intersection over the union, which represents the
overlapping degree of two predictions. This criterion is commonly used to
evaluate object detection systems where a higher value means more overlapping
between predicted results and ground truth.

$\displaystyle BLUE=BP\cdot exp(\sum_{n=1}^{N}w_{n}logp_{n})$ (9)

The BLUE score aims to compute how one sentence resembles the other, which
usually serves as a criterion to evaluate the machine translation system. It
is based on brevity penalty BP and n-gram precision, where $p_{n}$ is n-gram
precision, and $w_{n}$ is positive weights summing to one. $w_{n}$ is usually
set to uniform weights $w_{n}=1/N$. Specifically, BP is computed by:

$BP=\left\\{\begin{array}[]{lr}1&\quad\text{if}\mkern 9.0muc>r\\\
e^{(1-r/c)}&\quad\text{if}\mkern 9.0muc\leq r\\\ \end{array}\right\\}\\\ $
(10)

where $c$ is the length of the candidate translation and $r$ is the effective
reference corpus length. $BP$ is used to ensure that the length of high-
scoring translation matches the reference translations.

####  3.4.3 Hardware & software dependencies.

Given that our experiments are highly computationally intensive, all the
evaluations were run on a small cluster of four specialized Deep Learning
servers. Each server has an AMD 3955WX CPU (3.9GHz), 256GB RAM, and four
NVIDIA A4000 GPUs (16GB VRAM of each). Overall, our evaluation takes more than
5,000 GPU hours to complete.

##  4 Empirical Study Results

In this section, we present the experiment and study results of each research
question. In the paper, we only report our main results and summarized
findings.

###  4.1 RQ0. How common corruption patterns increase risks in the wild for AI
systems?

To answer RQ0, we investigate the common corruption patterns’ effects on four
AI systems by measuring the corresponding performance drop. We show the
system’s performance of Udacity, ByteTrack, ConvLab-2, and SpeechBrain against
corrupted test data in Table 4, Table 5, Table 6, and Table 7, respectively.
In these tables, $\Delta$ indicates the relative performance decrease after
applying common corruption patterns. A larger $\Delta$ indicates a bigger
performance drop and suggests that the corresponding setting (_i.e_., specific
corruption pattern and severity level) could threaten the system’s robustness.

Table 4: Udacity system’s performance against different corruption patterns on
Challenge2 dataset (each cell of results in the table is averaged over 3000
images’ runs).

Corruption | Severity | Module 1 |  | Module 2 |  | Module 3 |  | System
---|---|---|---|---|---|---|---|---
RMSE | $\Delta(\%)^{\dagger}$ |  | RMSE | $\Delta(\%)^{\dagger}$ |  | RMSE | $\Delta(\%)^{\dagger}$ |  | RMSE | $\Delta(\%)^{\dagger}$
Original | N/A | 0.065 | — |  | 0.054 | — |  | 0.048 | — |  | 0.045 | —
| 1 | 0.071 | 9.2 |  | 0.054 | <0.1 |  | 0.049 | 2.0 |  | 0.049 | 8.9
| 2 | 0.087 | 33.8 |  | 0.089 | 64.8 |  | 0.068 | 41.7 |  | 0.072 | 60.0
Gaussian noise (GN) | 3 | 0.164 | 152.3 |  | 0.178 | 229.6 |  | 0.144 | 200.0 |  | 0.159 | 253.3
Motion blur (MB) | 1 | 0.068 | 4.6 |  | 0.062 | 14.8 |  | 0.055 | 14.6 |  | 0.051 | 13.3
2 | 0.081 | 24.6 |  | 0.123 | 127.8 |  | 0.092 | 91.6 |  | 0.081 | 80.0
3 | 0.104 | 60.0 |  | 0.233 | 331.5 |  | 0.183 | 281.3 |  | 0.143 | 217.8
| 1 | 0.064 | -1 |  | 0.052 | 3.7 |  | 0.048 | <0.1 |  | 0.044 | -2.2
| 2 | 0.070 | 7.7 |  | 0.072 | 33.3 |  | 0.056 | 16.7 |  | 0.056 | 24.4
Snow (SN) | 3 | 0.091 | 40.0 |  | 0.099 | 83.3 |  | 0.088 | 83.3 |  | 0.086 | 91.1
Contrast (CO) | 1 | 0.077 | 18.5 |  | 0.081 | 50.0 |  | 0.072 | 50.0 |  | 0.064 | 42.2
2 | 0.112 | 72.3 |  | 0.107 | 98.1 |  | 0.113 | 135.4 |  | 0.093 | 106.7
3 | 0.167 | 156.9 |  | 0.167 | 209.3 |  | 0.182 | 279.2 |  | 0.170 | 277.8
| 1 | 0.068 | 4.6 |  | 0.061 | 13.0 |  | 0.056 | 16.7 |  | 0.052 | 15.6
| 2 | 0.071 | 9.2 |  | 0.071 | 31.5 |  | 0.064 | 33.3 |  | 0.058 | 28.9
Pixelate (PI) | 3 | 0.075 | 15.4 |  | 0.088 | 63.0 |  | 0.074 | 54.2 |  | 0.068 | 51.1

  * $\dagger$

$\Delta=\frac{\text{RMSE}}{\text{RMSE}_{\text{Original}}}-1$; $\Delta$
suggests the performance decrease against different corruption patterns.

Table 5: ByteTrack’s performance against different corruption patterns on
MOT17 dataset (each cell of results in the table is averaged over 21,600
frames of videos’ runs).

Corruption | Severity | Configuration $1^{\star}$ |  | Configuration $2^{\star}$ |  | Configuration $3^{\star}$
---|---|---|---|---|---|---
MOTA | $\Delta(\%)^{\dagger}$ |  | MOTA | $\Delta(\%)^{\dagger}$ |  | MOTA | $\Delta(\%)^{\dagger}$
Original | N/A | 0.765 | — |  | 0.746 | — |  | 0.754 | —
| 1 | 0.728 | 4.8 |  | 0.704 | 5.6 |  | 0.725 | 3.8
| 2 | 0.662 | 13.5 |  | 0.622 | 16.6 |  | 0.651 | 13.6
Gaussian noise (GN) | 3 | 0.362 | 52.7 |  | 0.302 | 59.5 |  | 0.355 | 52.9
Motion blur (MB) | 1 | 0.745 | 2.6 |  | 0.722 | 3.2 |  | 0.736 | 2.4
2 | 0.645 | 5.1 |  | 0.619 | 17.0 |  | 0.636 | 15.6
3 | 0.441 | 42.4 |  | 0.413 | 44.6 |  | 0.441 | 41.5
| 1 | 0.750 | 2.0 |  | 0.736 | 1.3 |  | 0.746 | 1.1
| 2 | 0.726 | 5.1 |  | 0.704 | 5.6 |  | 0.717 | 4.9
Contrast (CO) | 3 | 0.283 | 63.0 |  | 0.240 | 67.8 |  | 0.259 | 65.6
Snow (SN) | 1 | 0.725 | 5.2 |  | 0.686 | 8.0 |  | 0.722 | 4.2
2 | 0.682 | 10.8 |  | 0.604 | 19.0 |  | 0.670 | 11.1
3 | 0.613 | 19.9 |  | 0.538 | 27.9 |  | 0.606 | 19.6
| 1 | 0.757 | 1.0 |  | 0.735 | 1.5 |  | 0.747 | 0.9
| 2 | 0.721 | 5.8 |  | 0.692 | 7.2 |  | 0.710 | 5.8
Pixelate (PI) | 3 | 0.624 | 18.4 |  | 0.602 | 19.3 |  | 0.618 | 18.0

  * $\dagger$

$\Delta=1-\frac{\text{MOTA}}{\text{MOTA}_{\text{Original}}}$; $\Delta$
suggests the performance decrease against different corruption patterns.

  * $\star$

Configuration 1: YOLO + Track; Configuration 2: YOLO + SORT; Configuration 3:
YOLO + DeepSORT.

Table 6: ConvLab-2’s performance against different corruption patterns on
MultiWOZ dataset (each cell of results in the table is averaged over 1000
dialogues’ runs).

Corruption | Severity |  Configuration $1^{\star}$ |  |  Configuration $2^{\star}$ |  |  Configuration $3^{\star}$ |  |  Configuration $4^{\star}$
---|---|---|---|---|---|---|---|---
SR (%) | $\Delta(\%)^{\dagger}$ |  | SR (%) | $\Delta(\%)^{\dagger}$ |  | SR (%) | $\Delta(\%)^{\dagger}$ |  | SR (%) | $\Delta(\%)^{\dagger}$
Original | N/A | 70.5 | — |  | 63.2 | — |  | 49.7 | — |  | 44.4 | —
| 1 | 63.8 | 9.5 |  | 56.5 | 10.6 |  | 43.9 | 11.7 |  | 36.6 | 17.6
| 2 | 60.3 | 14.5 |  | 53.3 | 15.7 |  | 43.0 | 13.5 |  | 36.4 | 18.0
Digits2words (DW) | 3 | 55.6 | 21.1 |  | 50.2 | 20.6 |  | 41.9 | 15.7 |  | 35.9 | 19.1
Change char (CC) | 1 | 57.7 | 18.2 |  | 51.1 | 19.1 |  | 31.1 | 37.4 |  | 33.2 | 25.2
2 | 45.0 | 36.2 |  | 33.0 | 47.8 |  | 30.7 | 38.2 |  | 23.8 | 46.4
3 | 29.7 | 57.9 |  | 19.1 | 69.8 |  | 16.9 | 66.0 |  | 11.2 | 74.8
| 1 | 52.5 | 25.5 |  | 47.8 | 24.4 |  | 40.4 | 18.7 |  | 35.9 | 19.1
| 2 | 31.5 | 55.3 |  | 30.3 | 52.1 |  | 22.6 | 54.5 |  | 20.2 | 54.5
Remove char (RC) | 3 | 13.7 | 80.6 |  | 13.2 | 79.1 |  | 5.5 | 88.9 |  | 4.7 | 89.4
Misspelling (MS) | 1 | 63.3 | 10.2 |  | 54.5 | 13.8 |  | 44.0 | 11.5 |  | 36.4 | 18.0
2 | 56.2 | 20.3 |  | 44.1 | 30.2 |  | 41.0 | 17.5 |  | 34.7 | 21.8
3 | 29.7 | 57.9 |  | 21.4 | 66.1 |  | 31.2 | 37.2 |  | 25.5 | 42.6
| 1 | 47.5 | 32.6 |  | 44.3 | 29.9 |  | 36.0 | 27.6 |  | 33.3 | 25.0
| 2 | 20.6 | 70.8 |  | 14.8 | 76.6 |  | 8.6 | 82.7 |  | 4.8 | 89.2
Swap (SW) | 3 | 3.2 | 95.5 |  | 1.4 | 97.8 |  | 3.0 | 94.0 |  | 2.1 | 95.3

  * $\dagger$

$\Delta=1-\frac{\text{SR}}{\text{SR}_{\text{Original}}}$, SR denotes success
rate; $\Delta$ suggests the performance decrease against different corruption
patterns.

  * $\star$

Configuration 1: BERT NLU + Rule POL; Configuration 2: MILU NLU + Rule POL;
Configuration 3: BERT NLU + PPO POL; Configuration 4: MILU NLU + PPO POL.

Table 7: SpeechBrain’s performance against different corruption patterns on
LibriSpeech (each cell of results in the table is averaged over 2600 audio
samples’ runs) & CommonVoice-it (each cell of results in the table is averaged
over 8940 audio samples’ runs) dataset.

Corruption | Severity |  Configuration $1^{\star}$ |  |  Configuration $2^{\star}$ |  |  Configuration $3^{\star}$ |  |  Configuration $4^{\star}$
---|---|---|---|---|---|---|---|---
WER | $\Delta(\%)^{\dagger}$ |  | WER | $\Delta(\%)^{\dagger}$ |  | WER | $\Delta(\%)^{\dagger}$ |  | WER | $\Delta(\%)^{\dagger}$
Original | N/A | 0.212 | — |  | 0.035 | — |  | 0.249 | — |  | 0.299 | —
| 1 | 0.185 | -12.7 |  | 0.100 | 185.7 |  | 0.379 | 26.8 |  | 0.295 | 18.5
| 2 | 0.334 | 57.5 |  | 0.560 | 1500.0 |  | 0.645 | 115.7 |  | 0.482 | 93.6
Reverb (RE) | 3 | 0.766 | 261.3 |  | 0.852 | 2334.3 |  | 0.883 | 195.3 |  | 0.749 | 200.8
Speed (SP) | 1 | 0.214 | 0.9 |  | 0.047 | 34.3 |  | 0.455 | 52.2 |  | 0.288 | 15.7
2 | 0.608 | 196.8 |  | 0.339 | 868.6 |  | 0.676 | 126.1 |  | 0.360 | 44.6
3 | 1.159 | 446.7 |  | 0.981 | 2702.9 |  | 1.013 | 238.8 |  | 0.991 | 298.0
| 1 | 0.195 | -8.0 |  | 0.041 | 17.1 |  | 0.344 | 15.1 |  | 0.270 | 8.4
| 2 | 0.247 | 16.5 |  | 0.052 | 48.6 |  | 0.390 | 30.4 |  | 0.308 | 23.7
Distortion (DI) | 3 | 0.374 | 76.4 |  | 0.216 | 517.1 |  | 0.553 | 84.9 |  | 0.473 | 90.0
Pitch Shift (PS) | 1 | 0.114 | -46.2 |  | 0.043 | 22.9 |  | 0.336 | 12.4 |  | 0.264 | 6.0
2 | 0.170 | -19.8 |  | 0.065 | 85.7 |  | 0.385 | 28.8 |  | 0.291 | 16.9
3 | 0.245 | 15.6 |  | 0.174 | 397.1 |  | 0.496 | 65.9 |  | 0.347 | 39.4
| 1 | 0.164 | -22.6 |  | 0.330 | 842.9 |  | 0.387 | 29.4 |  | 0.291 | 16.9
| 2 | 0.194 | -8.5 |  | 0.720 | 1957.1 |  | 0.479 | 60.2 |  | 0.350 | 40.6
Background noise (BN) | 3 | 0.243 | 14.6 |  | 0.896 | 2460.0 |  | 0.568 | 90.0 |  | 0.416 | 67.1

  * $\dagger$

$\Delta=\frac{\text{WER}}{\text{WER}_{\text{Original}}}-1$, WER denotes word
error rate; $\Delta$ suggests the performance decrease against different
corruption patterns.

  * $\star$

Configuration 1: CRDNN + RNN (LibriSpeech); Configuration 2: CRDNN +
Transformer (LibriSpeech); Configuration 3: CRDNN + RNN (CommonVoice-it);
Configuration 4: Wav2Vec2 + RNN (CommonVoice-it).

Overall, we confirm that AI systems’ robustness and reliability can be
threatened by the common corruption patterns. In the first image-oriented
system, _i.e_., Udacity system, we observe that the highest severity level of
corruption could drop the system’s performance by 178.2% on average across
different configurations, respectively. Specifically, we find that the system
performs worse when against the highest severity level of Gaussian noise (GN),
motion blur (MB) and contrast change (CO). Meanwhile, we also find that in
some cases, even the lowest severity level of corruption could affect systems
performance significantly (_e.g_., level 1 of CO could decrease the Udacity
system’s performance by 42.2%). In the ByteTrack system, we find that with the
highest level of severity, the system performance is dropped by $59.5\%$
across five different corruption patterns and three system configurations. The
performance of the dialogue system, _i.e_., ConvLab-2, is also significantly
affected by common text corruption patterns. On average, even the lowest
severity of corruption can drop the systems’ performance by 20.1% across
different configurations and corruption patterns, respectively. Specifically,
we also find that the SW corruption poses the severest threat to the ConvLab-2
system’s robustness. On average, the system’s performance dropped by 22.0%,
79.8%, and 95.7% (from lowest severity level to highest one, respectively)
across four configurations when facing SW. Audio corruption patterns also
affect the performance of the speech recognition system, _i.e_., SpeechBrain.
On average, when facing the lowest and highest severity level of corruption,
the system’s performance drop is by 60.8% and 529.8% across different system
configurations and corruption patterns, respectively. Surprisingly, in the
SpeechBrain system, we find that the performance on clean data can’t indicate
the system’s robustness. For instance, on the LibriSpeech dataset,
configuration 1 (CRDNN+RNN) has a much higher WER than configuration 2
(CRDNN+Transformer) on the clean data (0.212 vs. 0.035). However, when facing
different corruption patterns, there is a significant performance drop with
configuration 2. For instance, the WER with configuration 2 increases by
1500.0% when facing the level-2 RE effect. By contrast, configuration 1 only
drops by 57.5% when facing the same corruption. In fact, different from the
system with configuration 2, the performance of the system with configuration
1 does not even drop when being against 4 out of 5 level-1 corruption
patterns.

We also find some possibilities to improve the AI system’s reliability based
on the different performances of each AI system with different configurations
against common corruption patterns. Recall that the Udacity system’s output is
determined by three independently executed modules (by taking the average). We
find that different modules can be robust to different corruption patterns.
For instance, when increasing the severity level of MB corruption, the
performance (_i.e_., RMSE) of Module 2 and Module 3 are significantly affected
(dropped by 331.5% and 281.3%, respectively). However, the performance drop of
Module 1 is only 60.0%. Therefore, we can further improve the system’s
performance if we can reject the bad prediction results from Modules 2 and 3
instead of always taking the average of all three modules. This also motivates
us to investigate whether there is a good indicator to detect such bad
prediction results. Similarly, in the SpeechBrain system, we observe that BN
corruption does not have many effects on configuration 1 (even level 3
corruption only increases WER from 0.212 to 0.243). However, the effects of BN
are significant in configuration 2, where WER increased from 0.035 to 0.896
when facing level 3 corruption. Hence, given that different AI system
configurations can be robust against different corruption patterns, a
comprehensive understanding of each system is needed in order to achieve
better system robustness.

Answer to RQ0: In general, we confirm that common corruption patterns also
significantly affect the AI system’s performance, posing threats to the AI
system’s prediction. A higher severity level of corruption usually has more
substantial effects on the AI system’s performance. In addition, we also find
AI systems with different configurations can be resilient against different
corruption patterns, revealing possibilities of improving system reliability
by the ensemble.

###  4.2 RQ1. What is the OOD awareness of different modules in AI systems?

In this research question, we utilize the corrupted datasets created in RQ1 to
further evaluate the usefulness of two OOD detection techniques in this
study—Mahalanobis distance (Maha-d) and kernel distribution estimation (KDE).
We first evaluate the OOD score’s distribution differences between clean data
and corrupted data in Sec. 4.2.1, then we investigate if different modules
inside an AI system could have different OOD awareness in Sec. 4.2.2.

####  4.2.1 OOD detection against common corruption patterns.

Table 8: Wilcoxon signed-rank test on ByteTrack system’s OOD score
distribution difference between clean data and corrupted data on MOT17
dataset.

Corruption | Settings⋄ | ${\mathcal{D}_{\textbf{Maha-d}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{KDE}}}^{\dagger}$
---|---|---|---|---
YOLO detector | DeepSORT tracker |  | YOLO detector | DeepSORT tracker
| L1-Ori | $0.85$ | $0.73$ |  | $1.00$ | $0.06*$
| L2-L1 | $0.67$ | $0.84$ |  | $0.74$ | $0.08*$
Gaussian noise (GN) | L3-L2 | $0.96$ | $0.62$ |  | $0.98$ | $0.19*$
Motion blur (MB) | L1-Ori | $0.30*$ | $0.67$ |  | $1.00$ | $0.49*$
L2-L1 | $0.93$ | $0.72$ |  | $0.96$ | $0.29*$
L3-L2 | $0.90$ | $0.69$ |  | $0.89$ | $0.27*$
| L1-Ori | $0.35*$ | $0.78$ |  | $1.00$ | $0.17*$
| L2-L1 | $0.96$ | $0.84$ |  | $0.92$ | $0.10*$
Contrast (CO) | L3-L2 | $1.00$ | $0.82$ |  | $0.97$ | $0.09*$
Snow (SN) | L1-Ori | $0.69$ | $0.93$ |  | $1.00$ | $0.22*$
L2-L1 | $0.56$ | $0.88$ |  | $0.60$ | $0.21*$
L3-L2 | $0.72$ | $0.70$ |  | $0.68$ | $0.23*$
| L1-Ori | $0.30*$ | $0.45*$ |  | $1.00$ | $0.31*$
| L2-L1 | $0.91$ | $0.46*$ |  | $0.93$ | $0.31*$
Pixelate (PI) | L3-L2 | $0.97$ | $0.40*$ |  | $0.97$ | $0.18*$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * ∗

This result is not statistically significant ($p>0.0001$).

Table 9: Wilcoxon signed-rank test on ConvLab-2 system’s OOD score
distribution difference between clean data and corrupted data on MultiWOZ
dataset.

Corruption | Settings⋄ | ${\mathcal{D}_{\textbf{Maha-d}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{KDE}}}^{\dagger}$
---|---|---|---|---
BERT NLU | MILU NLU | PPO Pol |  | BERT NLU | MILU NLU | PPO Pol
| L1-Ori | $0.55*$ | $0.75$ | $0.42*$ |  | $0.60$ | $0.72$ | $0.80$
| L2-L1 | $0.53*$ | $0.55*$ | $0.44*$ |  | $0.57$ | $0.54*$ | $0.49*$
Digits2words (DW) | L3-L2 | $0.48*$ | $0.49*$ | $0.44*$ |  | $0.55$ | $0.54*$ | $0.42*$
Change char (CC) | L1-Ori | $0.96$ | $0.96$ | $0.48*$ |  | $0.98$ | $0.84$ | $0.88$
L2-L1 | $0.81$ | $0.97$ | $0.43*$ |  | $0.81$ | $0.87$ | $0.75$
L3-L2 | $0.23*$ | $0.27*$ | $0.33*$ |  | $0.78$ | $0.90$ | $0.45*$
| L1-Ori | $0.98$ | $0.96$ | $0.46*$ |  | $0.98$ | $0.86$ | $0.89$
| L2-L1 | $0.80$ | $0.91$ | $0.32*$ |  | $0.80$ | $0.84$ | $0.73$
Remove char (RC) | L3-L2 | $0.16*$ | $0.21*$ | $0.28*$ |  | $0.80$ | $0.90$ | $0.38*$
Misspelling (MS) | L1-Ori | $0.75$ | $0.82$ | $0.43*$ |  | $0.87$ | $0.75$ | $0.85$
L2-L1 | $0.80$ | $0.68$ | $0.44*$ |  | $0.84$ | $0.61$ | $0.56$
L3-L2 | $0.23*$ | $0.29*$ | $0.29*$ |  | $0.88$ | $0.78$ | $0.55$
| L1-Ori | $0.97$ | $0.93$ | $0.45*$ |  | $0.97$ | $0.89$ | $0.89$
| L2-L1 | $0.80$ | $0.93$ | $0.05*$ |  | $0.80$ | $0.93$ | $0.41*$
Swap (SW) | L3-L2 | $0.28*$ | $0.31*$ | $0.48*$ |  | $0.78$ | $0.98$ | $0.21*$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 10: Wilcoxon signed-rank test on SpeechBrain system’s OOD score
distribution difference between clean data and corrupted data on LibriSpeech
dataset.

Corruption | Settings⋄ | Configuration 1 |  | Configuration 2
---|---|---|---|---
${\mathcal{D}_{\textbf{Maha-d}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{KDE}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{Maha-d}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{KDE}}}^{\dagger}$
CRDNN | RNN |  | CRDNN | RNN |  | CRDNN | Trans- |  | CRDNN | Trans-
| L1-Ori | $1.00$ | $0.67$ |  | $0.74$ | $1.00$ |  | $1.00$ | $0.46^{*}$ |  | $0.99$ | $1.00$
| L2-L1 | $1.00$ | $0.45^{*}$ |  | $0.69$ | $0.68$ |  | $1.00$ | $0.45^{*}$ |  | $1.00$ | $0.74$
Reverb (RE) | L3-L2 | $1.00$ | $0.56$ |  | $0.64$ | $0.66$ |  | $1.00$ | $0.58$ |  | $1.00$ | $0.52^{*}$
Speed (SP) | L1-Ori | $1.00$ | $0.72$ |  | $0.70$ | $1.00$ |  | $1.00$ | $0.51^{*}$ |  | $1.00$ | $1.00$
L2-L1 | $0.46^{*}$ | $0.58$ |  | $0.61$ | $0.56$ |  | $0.19^{*}$ | $0.41^{*}$ |  | $0.52^{*}$ | $0.83$
L3-L2 | $0.84$ | $0.54^{*}$ |  | $0.75$ | $0.57$ |  | $1.00$ | $0.65$ |  | $1.00$ | $0.66$
| L1-Ori | $1.00$ | $0.72$ |  | $0.93$ | $1.00$ |  | $1.00$ | $0.50^{*}$ |  | $1.00$ | $1.00$
| L2-L1 | $0.92$ | $0.49^{*}$ |  | $0.74$ | $0.48^{*}$ |  | $1.00$ | $0.43^{*}$ |  | $1.00$ | $0.73$
Distortion (DI) | L3-L2 | $0.99$ | $0.44^{*}$ |  | $0.83$ | $0.52^{*}$ |  | $1.00$ | $0.38^{*}$ |  | $1.00$ | $0.81$
Pitch shift (PS) | L1-Ori | $1.00$ | $0.73$ |  | $0.55$ | $1.00$ |  | $1.00$ | $0.50^{*}$ |  | $0.73$ | $1.00$
L2-L1 | $0.71$ | $0.48^{*}$ |  | $0.58$ | $0.48^{*}$ |  | $1.00$ | $0.44^{*}$ |  | $0.95$ | $0.65$
L3-L2 | $0.62$ | $0.48^{*}$ |  | $0.60$ | $0.50^{*}$ |  | $0.99$ | $0.45^{*}$ |  | $0.97$ | $0.70$
| L1-Ori | $1.00$ | $0.69$ |  | $0.91$ | $1.00$ |  | $1.00$ | $0.50^{*}$ |  | $1.00$ | $1.00$
| L2-L1 | $0.80$ | $0.46^{*}$ |  | $0.67$ | $0.48^{*}$ |  | $1.00$ | $0.49^{*}$ |  | $1.00$ | $0.59$
Background noise (BN) | L3-L2 | $0.68$ | $0.47^{*}$ |  | $0.63$ | $0.51^{*}$ |  | $1.00$ | $0.51^{*}$ |  | $1.00$ | $0.52^{*}$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 11: Wilcoxon signed-rank test on SpeechBrain system’s OOD score
distribution difference between clean data and corrupted data on CommonVoice-
it dataset.

Corruption | Settings⋄ | Configuration 3 |  | Configuration 4
---|---|---|---|---
${\mathcal{D}_{\textbf{Maha-d}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{KDE}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{Maha-d}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{KDE}}}^{\dagger}$
CRDNN | RNN |  | CRDNN | RNN |  | Wav2Vec2 | RNN |  | Wav2Vec2 | RNN
| L1-Ori | $1.00$ | $0.39^{*}$ |  | $1.00$ | $1.00$ |  | $0.93$ | $0.79$ |  | $0.85$ | $1.00$
| L2-L1 | $1.00$ | $0.18^{*}$ |  | $0.97$ | $0.99$ |  | $0.94$ | $0.87$ |  | $0.92$ | $0.98$
Reverb (RE) | L3-L2 | $1.00$ | $0.29^{*}$ |  | $0.85$ | $0.81$ |  | $0.78$ | $0.86$ |  | $0.93$ | $0.94$
Speed (SP) | L1-Ori | $1.00$ | $0.49^{*}$ |  | $1.00$ | $1.00$ |  | $0.47^{*}$ | $0.70$ |  | $0.49^{*}$ | $1.00$
L2-L1 | $0.92$ | $0.19^{*}$ |  | $0.60$ | $0.91$ |  | $0.90$ | $0.67$ |  | $0.70$ | $0.82$
L3-L2 | $1.00$ | $0.32^{*}$ |  | $0.91$ | $0.76$ |  | $0.99$ | $0.97$ |  | $0.85$ | $0.99$
| L1-Ori | $1.00$ | $0.53$ |  | $1.00$ | $1.00$ |  | $0.99$ | $0.68$ |  | $0.72$ | $1.00$
| L2-L1 | $0.98$ | $0.45^{*}$ |  | $0.63$ | $0.92$ |  | $1.00$ | $0.66$ |  | $0.81$ | $0.86$
Distortion (DI) | L3-L2 | $0.97$ | $0.29^{*}$ |  | $0.70$ | $0.97$ |  | $1.00$ | $0.84$ |  | $0.91$ | $0.96$
Pitch shift (PS) | L1-Ori | $1.00$ | $0.52^{*}$ |  | $1.00$ | $1.00$ |  | $0.60$ | $0.65$ |  | $0.73$ | $1.00$
L2-L1 | $0.96$ | $0.42^{*}$ |  | $0.98$ | $0.91$ |  | $0.84$ | $0.61$ |  | $0.78$ | $0.76$
L3-L2 | $0.92$ | $0.36^{*}$ |  | $0.78$ | $0.95$ |  | $0.78$ | $0.70$ |  | $0.81$ | $0.87$
| L1-Ori | $1.00$ | $0.36^{*}$ |  | $1.00$ | $1.00$ |  | $1.00$ | $0.73$ |  | $1.00$ | $1.00$
| L2-L1 | $1.00$ | $0.28^{*}$ |  | $0.80$ | $0.94$ |  | $1.00$ | $0.70$ |  | $1.00$ | $0.90$
Background noise (BN) | L3-L2 | $1.00$ | $0.28^{*}$ |  | $0.75$ | $0.85$ |  | $0.99$ | $0.70$ |  | $1.00$ | $0.87$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * ∗

This result is not statistically significant ($p>0.0001$).

Table 8, Table 9, and Table 10&11 show the Wilcoxon signed-rank test results
on significance between the OOD score of clean data and corrupted data on
ByteTrack system, ConvLab-2 system, and SpeechBrain system, respectively. A
positive effect size in these tables indicates that this module’s OOD score on
corrupted data is significantly larger than on clean data. The larger the
value, the more significant the result can be. In the ByteTrack system, we
find that no matter whether using Maha-d or KDE, the YOLO detector’s OOD score
measured on corrupted data is larger than it is on clean data. When the
severity level of corruption increases, such difference also becomes more
significant. This suggests that both these two OOD detection techniques can
well capture the data distribution between clean data and corrupted data on
the YOLO detector module. However, it becomes harder for these two methods to
measure the data distribution difference on the second module of the ByteTrack
system, _i.e_., the DeepSORT tracker. Specifically, we find that KDE failed to
capture the data distribution difference between clean and corrupted data. In
the ConvLab-2 system, we find that both these two methods can well measure the
data distribution difference on either BERT or MILU NLU module (which is the
first module in the system). Nevertheless, Maha-d fails to measure it on the
PPO policy module, while KDE succeeds. The observation of these two systems
suggests that for a serial execution system, OOD detection techniques are
better at measuring the data distribution difference on the first module of
the system. We further verify this finding on the SpeechBrain system, where we
find that Maha-d method fails on the second module of Configuration 3 and 4.
One plausible explanation for these observations is: that the first module in
a serial-executed AI system usually directly takes the corrupted data as
input, therefore, it can better capture the data distribution difference.

####  4.2.2 OOD awareness of different modules inside AI systems.

Table 12: Similarity of top-1% OOD data detected according two different
modules in SpeechBrain system.

Corruption | Severity | Configuration 1† |  | Configuration 2† |  | Configuration 3⋄ |  | Configuration 4⋄
---|---|---|---|---|---|---|---|---
Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE
| 1 | $0.0\%$ | $0.0\%$ |  | $7.7\%$ | $3.8\%$ |  | $1.1\%$ | $0.0\%$ |  | $0.0\%$ | $0.0\%$
| 2 | $0.0\%$ | $0.0\%$ |  | $0.0\%$ | $0.0\%$ |  | $1.1\%$ | $0.0\%$ |  | $0.0\%$ | $0.0\%$
Reverb (RE) | 3 | $0.0\%$ | $0.0\%$ |  | $0.0\%$ | $3.8\%$ |  | $1.1\%$ | $0.0\%$ |  | $1.1\%$ | $1.1\%$
Speed (SP) | 1 | $3.8\%$ | $0.0\%$ |  | $3.8\%$ | $0.0\%$ |  | $2.2\%$ | $0.0\%$ |  | $1.1\%$ | $5.6\%$
2 | $0.0\%$ | $11.5\%$ |  | $11.5\%$ | $3.8\%$ |  | $1.1\%$ | $0.0\%$ |  | $0.0\%$ | $0.0\%$
3 | $0.0\%$ | $0.0\%$ |  | $0.0\%$ | $3.8\%$ |  | $4.5\%$ | $0.0\%$ |  | $2.2\%$ | $1.1\%$
| 1 | $0.0\%$ | $0.0\%$ |  | $11.5\%$ | $0.0\%$ |  | $4.5\%$ | $0.0\%$ |  | $4.5\%$ | $4.5\%$
| 2 | $0.0\%$ | $1.1\%$ |  | $3.8\%$ | $0.0\%$ |  | $5.6\%$ | $1.1\%$ |  | $4.5\%$ | $4.5\%$
Distortion (DI) | 3 | $0.0\%$ | $3.4\%$ |  | $0.0\%$ | $0.0\%$ |  | $2.2\%$ | $3.4\%$ |  | $3.4\%$ | $4.5\%$
Pitch shift (PS) | 1 | $0.0\%$ | $0.0\%$ |  | $7.7\%$ | $0.0\%$ |  | $0.0\%$ | $0.0\%$ |  | $0.0\%$ | $3.4\%$
2 | $0.0\%$ | $0.0\%$ |  | $3.8\%$ | $3.8\%$ |  | $4.5\%$ | $0.0\%$ |  | $1.1\%$ | $2.2\%$
3 | $0.0\%$ | $0.0\%$ |  | $7.7\%$ | $3.8\%$ |  | $2.2\%$ | $0.0\%$ |  | $1.1\%$ | $1.1\%$
| 1 | $0.0\%$ | $0.0\%$ |  | $0.0\%$ | $3.8\%$ |  | $0.0\%$ | $0.0\%$ |  | $3.4\%$ | $0.0\%$
| 2 | $0.0\%$ | $0.0\%$ |  | $0.0\%$ | $7.7\%$ |  | $0.0\%$ | $0.0\%$ |  | $2.2\%$ | $0.0\%$
Background noise (BN) | 3 | $0.0\%$ | $0.0\%$ |  | $3.8\%$ | $11.5\%$ |  | $0.0\%$ | $0.0\%$ |  | $1.1\%$ | $0.0\%$

  * $\dagger$

Configuration 1 and Configuration 2 is tested on LibriSpeech dataset, where
$1\%$ data has a size of 26 instances.

  * $\diamond$

Configuration 3 and Configuration 4 is tested on CommonVoice-it dataset, where
$1\%$ data has a size of 89 instances.

Table 13: Similarity of top-5% OOD data detected according two different
modules in SpeechBrain system.

Corruption | Severity | Configuration 1† |  | Configuration 2† |  | Configuration 3⋄ |  | Configuration 4⋄
---|---|---|---|---|---|---|---|---
Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE
| 1 | $0.8\%$ | $6.9\%$ |  | $12.3\%$ | $5.4\%$ |  | $2.0\%$ | $0.0\%$ |  | $0.9\%$ | $1.3\%$
| 2 | $4.6\%$ | $6.2\%$ |  | $6.9\%$ | $4.6\%$ |  | $3.0\%$ | $0.1\%$ |  | $0.9\%$ | $4.5\%$
Reverb (RE) | 3 | $3.8\%$ | $4.6\%$ |  | $1.5\%$ | $6.9\%$ |  | $3.4\%$ | $0.1\%$ |  | $0.9\%$ | $3.6\%$
Speed (SP) | 1 | $4.6\%$ | $5.4\%$ |  | $6.2\%$ | $3.8\%$ |  | $3.1\%$ | $0.0\%$ |  | $3.1\%$ | $8.9\%$
2 | $1.5\%$ | $3.8\%$ |  | $15.4\%$ | $5.4\%$ |  | $2.0\%$ | $0.1\%$ |  | $1.3\%$ | $6.0\%$
3 | $5.4\%$ | $5.4\%$ |  | $1.5\%$ | $4.6\%$ |  | $6.3\%$ | $0.0\%$ |  | $4.0\%$ | $4.3\%$
| 1 | $3.8\%$ | $3.1\%$ |  | $15.4\%$ | $3.8\%$ |  | $10.1\%$ | $2.2\%$ |  | $13.6\%$ | $8.7\%$
| 2 | $1.5\%$ | $5.4\%$ |  | $16.9\%$ | $5.4\%$ |  | $10.2\%$ | $3.1\%$ |  | $11.6\%$ | $9.8\%$
Distortion (DI) | 3 | $0.1\%$ | $6.9\%$ |  | $7.7\%$ | $6.2\%$ |  | $8.9\%$ | $12.1\%$ |  | $9.6\%$ | $18.3\%$
Pitch shift (PS) | 1 | $3.8\%$ | $3.8\%$ |  | $16.9\%$ | $8.5\%$ |  | $3.8\%$ | $0.1\%$ |  | $2.7\%$ | $5.6\%$
2 | $3.8\%$ | $4.6\%$ |  | $26.1\%$ | $12.3\%$ |  | $3.8\%$ | $0.1\%$ |  | $3.4\%$ | $5.1\%$
3 | $4.6\%$ | $8.5\%$ |  | $17.7\%$ | $13.8\%$ |  | $2.5\%$ | $0.1\%$ |  | $3.1\%$ | $5.6\%$
| 1 | $3.1\%$ | $6.9\%$ |  | $10.8\%$ | $18.5\%$ |  | $1.8\%$ | $0.1\%$ |  | $6.5\%$ | $2.5\%$
| 2 | $3.1\%$ | $6.2\%$ |  | $8.5\%$ | $20.0\%$ |  | $2.0\%$ | $1.8\%$ |  | $4.3\%$ | $2.2\%$
Background noise (BN) | 3 | $3.8\%$ | $5.4\%$ |  | $6.2\%$ | $20.8\%$ |  | $1.6\%$ | $3.8\%$ |  | $5.1\%$ | $0.1\%$

  * $\dagger$

Configuration 1 and Configuration 2 is tested on LibriSpeech dataset, where
$5\%$ data has a size of 130 instances.

  * $\diamond$

Configuration 3 and Configuration 4 is tested on CommonVoice-it dataset, where
$5\%$ data has a size of 447 instances.

Table 14: Similarity of top-10% OOD data detected according two different
modules in SpeechBrain system.

Corruption | Severity | Configuration 1† |  | Configuration 2† |  | Configuration 3⋄ |  | Configuration 4⋄
---|---|---|---|---|---|---|---|---
Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE
| 1 | $6.5\%$ | $12.3\%$ |  | $24.1\%$ | $6.5\%$ |  | $7.3\%$ | $1.1\%$ |  | $4.0\%$ | $2.8\%$
| 2 | $6.5\%$ | $11.5\%$ |  | $12.3\%$ | $9.2\%$ |  | $8.0\%$ | $1.2\%$ |  | $2.5\%$ | $7.5\%$
Reverb (RE) | 3 | $5.7\%$ | $11.1\%$ |  | $7.3\%$ | $12.3\%$ |  | $7.2\%$ | $0.9\%$ |  | $3.1\%$ | $6.1\%$
Speed (SP) | 1 | $7.7\%$ | $8.8\%$ |  | $13.4\%$ | $8.4\%$ |  | $5.7\%$ | $1.2\%$ |  | $8.9\%$ | $17.8\%$
2 | $5.4\%$ | $11.1\%$ |  | $27.6\%$ | $20.3\%$ |  | $5.6\%$ | $2.6\%$ |  | $5.5\%$ | $7.9\%$
3 | $10.0\%$ | $11.5\%$ |  | $4.6\%$ | $15.7\%$ |  | $11.1\%$ | $0.3\%$ |  | $6.6\%$ | $7.3\%$
| 1 | $6.1\%$ | $9.2\%$ |  | $22.6\%$ | $10.3\%$ |  | $16.1\%$ | $5.9\%$ |  | $19.6\%$ | $11.4\%$
| 2 | $4.2\%$ | $8.0\%$ |  | $18.4\%$ | $14.6\%$ |  | $14.5\%$ | $10.7\%$ |  | $18.4\%$ | $15.1\%$
Distortion (DI) | 3 | $3.4\%$ | $10.7\%$ |  | $16.1\%$ | $16.1\%$ |  | $14.0\%$ | $20.7\%$ |  | $15.5\%$ | $19.8\%$
Pitch shift (PS) | 1 | $8.4\%$ | $11.9\%$ |  | $26.4\%$ | $16.5\%$ |  | $7.7\%$ | $2.1\%$ |  | $7.0\%$ | $9.1\%$
2 | $10.3\%$ | $12.3\%$ |  | $36.0\%$ | $22.2\%$ |  | $7.2\%$ | $1.7\%$ |  | $6.1\%$ | $7.8\%$
3 | $6.9\%$ | $12.6\%$ |  | $30.7\%$ | $25.7\%$ |  | $6.5\%$ | $2.6\%$ |  | $5.9\%$ | $8.7\%$
| 1 | $5.0\%$ | $10.3\%$ |  | $18.0\%$ | $34.5\%$ |  | $5.4\%$ | $3.8\%$ |  | $7.3\%$ | $3.9\%$
| 2 | $4.2\%$ | $10.7\%$ |  | $19.9\%$ | $34.1\%$ |  | $4.5\%$ | $5.4\%$ |  | $6.9\%$ | $4.2\%$
Background noise (BN) | 3 | $5.4\%$ | $8.8\%$ |  | $16.5\%$ | $27.6\%$ |  | $5.4\%$ | $6.1\%$ |  | $7.8\%$ | $5.9\%$

  * $\dagger$

Configuration 1 and Configuration 2 is tested on LibriSpeech dataset, where
$10\%$ data has a size of 260 instances.

  * $\diamond$

Configuration 3 and Configuration 4 is tested on CommonVoice-it dataset, where
$10\%$ data has a size of 894 instances.

While the OOD techniques used in this study can only measure on AI module,
hence in this subsection, we mainly investigate the results from the
SpeechBrain system and ByteTrack system (with DeepSORT tracker), where two
modules in the system are both AI ones. We select the $1\%$ test data with the
highest OOD score detected on two different modules, then analyze if they are
the same on two modules. We measure such similarity according to Eq. 11.

$sim=\frac{|\mathcal{S}^{n}_{module1}\cap\mathcal{S}^{n}_{module2}|}{|\mathcal{S}^{n}_{module1}|}$
(11)

where $\mathcal{S}^{n}_{module1}$ and $\mathcal{S}^{n}_{module2}$ denotes the
set of top-$n$% OOD data detected by module 1 and module 2, respectively.

Table 15: Similarity of top-1%, top-5%, and top-10% OOD data detected according two different modules in ByteTrack system (YOLO+DeepSORT). Corruption | Severity | Top-1% |  | Top-5% |  | Top-10%
---|---|---|---|---|---|---
Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE
| 1 | $0.0\%$ | $3.8\%$ |  | $2.3\%$ | $0.0\%$ |  | $4.2\%$ | $1.9\%$
| 2 | $3.8\%$ | $0.0\%$ |  | $12.1\%$ | $1.5\%$ |  | $33.2\%$ | $1.5\%$
Gaussian noise (GN) | 3 | $3.8\%$ | $0.0\%$ |  | $22.0\%$ | $5.3\%$ |  | $45.7\%$ | $10.9\%$
Motion blur (MB) | 1 | $0.0\%$ | $0.0\%$ |  | $7.6\%$ | $3.0\%$ |  | $14.7\%$ | $6.8\%$
2 | $0.0\%$ | $3.8\%$ |  | $3.0\%$ | $1.5\%$ |  | $6.8\%$ | $6.8\%$
3 | $0.0\%$ | $0.0\%$ |  | $1.5\%$ | $0.0\%$ |  | $5.7\%$ | $2.3\&$
| 1 | $0.0\%$ | $3.8\%$ |  | $2.3\%$ | $3.0\%$ |  | $4.2\%$ | $6.0\%$
| 2 | $0.0\%$ | $3.8\%$ |  | $6.8\%$ | $0.1$ |  | $11.7\%$ | $4.9\%$
Contrast (CO) | 3 | $7.7\%$ | $0.0\%$ |  | $15.2\%$ | $3.8$ |  | $40.0\%$ | $5.7\%$
Snow (SN) | 1 | $0.0\%$ | $0.0\%$ |  | $1.5\%$ | $0.0\%$ |  | $5.7\%$ | $4.2\%$
2 | $0.0\%$ | $0.0\%$ |  | $7.6\%$ | $0.1\%$ |  | $20.0\%$ | $4.2\%$
3 | $0.0\%$ | $0.0\%$ |  | $6.8\%$ | $2.2\%$ |  | $21.9\%$ | $7.5\%$
| 1 | $11.5\%$ | $0.0\%$ |  | $12.1\%$ | $6.8\%$ |  | $15.5\%$ | $14.7\%$
| 2 | $3.8\%$ | $0.0\%$ |  | $12.9\%$ | $1.5\%$ |  | $14.3\%$ | $9.1\%$
Pixelate (PI) | 3 | $0.0\%$ | $0.0\%$ |  | $5.3\%$ | $1.5\%$ |  | $19.2\%$ | $4.2\%$

Table 12, 13, 14 and Table 15 summarize the similarity of top-1%, top-5%, and
top-10% OOD data detected on two different modules in the SpeechBrain system
and ByteTrack system, respectively. In most cases, we can find that two
modules in the SpeechBrain system detect different OOD data. When only
considering top-1% data, the highest similarity (11.5%, Configuration 2
against Background noise corruption, measured by KDE) only suggests that there
are 3 OOD instances that both two modules detect. When considering top-10%
data, even the highest similarity only suggest less than half of OOD data are
similar (36%, Configuration 2 against Pitch shift corruption, measured by
Maha-d) .We can observe similar symptoms in the ByteTrack system, where the
top-1% OOD data are very different for two modules (_i.e_., detector and
tracker). These results reveal that different modules in an AI system could
treat different data instances as OOD data.

To further investigate the difference of OOD data identified by different
modules, we use a threshold-independent assessment, _i.e_., Spearman’s rank
correlation coefficient to measure such differences. We show the results in
Table 16 and Table 17. We can find there is no strong evidence showing that
there are correlations between two modules’ OOD data ranking in SpeechBrain
system when measured by either Maha-d or KDE. These results suggest that the
OOD data identified by these two modules could be very different. Similarly,
Spearman’s rank correlation coefficient results of ByteTrack system also
suggest that there is no significant correlation between two modules OOD data
ranking. Again, these results reveal that OOD sensitivity could be very
different among modules even in an identical AI system.

Table 16: Spearman’s rank correlation coefficient of OOD score computed
according two different modules in SpeechBrain system.

Corruption | Severity | Configuration 1 |  | Configuration 2 |  | Configuration 3 |  | Configuration 4
---|---|---|---|---|---|---|---|---
Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE |  | Maha-d | KDE
| 1 | $0.01$ | $0.20$ |  | $0.33$ | $-0.02$ |  | $0.03$ | $-0.36$ |  | $-0.09$ | $-0.34$
| 2 | $0.01$ | $0.33$ |  | $0.16$ | $-0.02$ |  | $0.06$ | $-0.46$ |  | $-0.17$ | $-0.39$
Reverb (RE) | 3 | $<0.01$ | $0.34$ |  | $0.03$ | $0.13$ |  | $0.03$ | $-0.42$ |  | $-0.18$ | $-0.47$
Speed (SP) | 1 | $-0.02$ | $-0.05$ |  | $0.13$ | $0.10$ |  | $-0.09$ | $-0.24$ |  | $0.01$ | $0.24$
2 | $-0.02$ | $-0.02$ |  | $0.55$ | $0.44$ |  | $0.03$ | $-0.41$ |  | $-0.09$ | $-0.25$
3 | $<0.01$ | $-0.01$ |  | $0.15$ | $0.15$ |  | $0.05$ | $-0.42$ |  | $-0.13$ | $-0.38$
| 1 | $-0.01$ | $-0.04$ |  | $0.25$ | $0.13$ |  | $-0.05$ | $-0.01$ |  | $0.15$ | $-0.12$
| 2 | $-0.03$ | $0.05$ |  | $0.30$ | $0.19$ |  | $-0.11$ | $0.09$ |  | $0.21$ | $-0.05$
Distortion (DI) | 3 | $-0.04$ | $<0.01$ |  | $0.22$ | $0.21$ |  | $-0.22$ | $0.21$ |  | $0.23$ | $-0.01$
Pitch shift (PS) | 1 | $-0.02$ | $-0.01$ |  | $0.39$ | $0.20$ |  | $-0.02$ | $-0.30$ |  | $0.01$ | $-0.22$
2 | $-0.03$ | $0.01$ |  | $0.49$ | $0.38$ |  | $-0.03$ | $-0.26$ |  | $-0.02$ | $-0.24$
3 | $0.02$ | $-0.03$ |  | $0.56$ | $0.52$ |  | $-0.03$ | $-0.24$ |  | $-0.04$ | $-0.23$
| 1 | $0.01$ | $-0.03$ |  | $0.37$ | $0.57$ |  | $-0.02$ | $-0.20$ |  | $-0.06$ | $-0.25$
| 2 | $0.03$ | $-0.06$ |  | $0.42$ | $0.49$ |  | $-0.02$ | $-0.17$ |  | $-0.09$ | $-0.22$
Background noise (BN) | 3 | $0.01$ | $-0.04$ |  | $0.38$ | $0.43$ |  | $-0.01$ | $-0.20$ |  | $-0.09$ | $-0.20$

Table 17: Spearman’s rank correlation coefficient of OOD score computed according two different modules in ByteTrack system (YOLO+DeepSORT). Corruption | Severity | Maha-d | KDE
---|---|---|---
| 1 | $-0.07$ | $-0.20$
| 2 | $0.39$ | $-0.44$
Gaussian noise (GN) | 3 | $0.39$ | $-0.14$
Motion blur (MB) | 1 | $-0.05$ | $<0.01$
2 | $0.04$ | $-0.10$
3 | $0.13$ | $-0.13$
| 1 | $-0.15$ | $<0.01$
| 2 | $0.28$ | $-0.15$
Contrast (CO) | 3 | $0.31$ | $-0.02$
Snow (SN) | 1 | $-0.02$ | $-0.02$
2 | $0.16$ | $0.02$
3 | $0.19$ | $0.06$
| 1 | $-0.18$ | $0.01$
| 2 | $-0.10$ | $0.14$
Pixelate (PI) | 3 | $0.29$ | $0.08$

Answer to RQ1: Both studied OOD detection methods (_i.e_., Maha-d and KDE) can
capture the data distribution difference between clean data and corrupted
data, while there are no systematic evidences that one method is better than
the other. These two methods are also more effective on the first module in a
serial-executed system than later, indicated by Wilcoxon signed-rank test
results. Furthermore, different modules of an AI system could detect different
OOD data, suggesting that developers could take the OOD sensitivity of
different modules into account when performing risk assessments.

###  4.3 RQ2. How does the uncertainty information behave on different modules
in AI systems?

In this research question, we utilize the corrupted datasets created in RQ1
again to evaluate the effectiveness of two uncertainty estimation metrics used
in this study—variation ratio (VR) and variation ratio of original prediction
(VRO). We first evaluate the uncertainty score’s distribution differences
between clean data and corrupted data in Sec. 4.3.1, then we investigate if
there is any correlation between different modules’ uncertainty information of
an AI system in Sec. 4.3.2.

####  4.3.1 Uncertainty estimation against common corruption patterns.

Table 18: Wilcoxon signed-rank test on ByteTrack system’s uncertainty score
distribution difference between clean data and corrupted data on MOT17
dataset.

Corruption | Settings⋄ | ${\mathcal{D}_{\textbf{VR}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{VRO}}}^{\dagger}$
---|---|---|---|---
YOLO | Track | SORT | DeepSORT |  | YOLO | Track | SORT | DeepSORT
| L1-Ori | $0.83$ | $0.85$ | $0.81$ | $0.68$ |  | $0.82$ | $0.84$ | $0.82$ | $0.72$
| L2-L1 | $0.89$ | $0.89$ | $0.92$ | $0.26^{*}$ |  | $0.95$ | $0.93$ | $0.95$ | $0.71$
Gaussian noise (GN) | L3-L2 | $0.65$ | $0.39^{*}$ | $0.31^{*}$ | $0.02^{*}$ |  | $0.97$ | $0.82$ | $0.77$ | $0.63$
Motion blur (MB) | L1-Ori | $0.41^{*}$ | $0.69$ | $0.69$ | $0.55$ |  | $0.47^{*}$ | $0.67$ | $0.69$ | $0.62$
L2-L1 | $0.67$ | $0.88$ | $0.86$ | $0.62$ |  | $0.70$ | $0.87$ | $0.87$ | $0.71$
L3-L2 | $0.78$ | $0.86$ | $0.85$ | $0.48^{*}$ |  | $0.82$ | $0.87$ | $0.86$ | $0.64$
| L1-Ori | $0.60$ | $0.66$ | $0.67$ | $0.46^{*}$ |  | $0.67$ | $0.69$ | $0.75$ | $0.61$
| L2-L1 | $0.68$ | $0.81$ | $0.84$ | $0.36^{*}$ |  | $0.75$ | $0.86$ | $0.88$ | $0.63$
Contrast (CO) | L3-L2 | $0.49^{*}$ | $0.10^{*}$ | $0.07^{*}$ | $0.01^{*}$ |  | $0.96$ | $0.63$ | $0.59$ | $0.59$
Snow (SN) | L1-Ori | $0.86$ | $0.82$ | $0.83$ | $0.44^{*}$ |  | $0.91$ | $0.84$ | $0.85$ | $0.66$
L2-L1 | $0.82$ | $0.80$ | $0.79$ | $0.39^{*}$ |  | $0.80$ | $0.79$ | $0.79$ | $0.56$
L3-L2 | $0.62$ | $0.66$ | $0.65$ | $0.36^{*}$ |  | $0.67$ | $0.69$ | $0.67$ | $0.52^{*}$
| L1-Ori | $0.62$ | $0.67$ | $0.64$ | $0.60$ |  | $0.68$ | $0.69$ | $0.71$ | $0.62$
| L2-L1 | $0.62$ | $0.72$ | $0.72$ | $0.58$ |  | $0.67$ | $0.73$ | $0.73$ | $0.63$
Pixelate (PI) | L3-L2 | $0.72$ | $0.82$ | $0.79$ | $0.62$ |  | $0.74$ | $0.79$ | $0.78$ | $0.67$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 19: Wilcoxon signed-rank test on ConvLab-2 system’s uncertainty score
distribution difference between clean data and corrupted data on MultiWOZ
dataset.

Corruption | Settings⋄ | ${\mathcal{D}_{\textbf{VR}}}^{\dagger}$ |  | ${\mathcal{D}_{\textbf{VRO}}}^{\dagger}$
---|---|---|---|---
BERT NLU | MILU NLU |  | BERT NLU | MILU NLU
| L1-Ori | $0.06*$ | $0.32$ |  | $0.07*$ | $0.29$
| L2-L1 | $0.08$ | $0.12*$ |  | $0.08$ | $0.12*$
Digits2words (DW) | L3-L2 | $0.12$ | $0.13*$ |  | $0.12$ | $0.12*$
Change char (CC) | L1-Ori | $0.75$ | $0.78$ |  | $0.73$ | $0.72$
L2-L1 | $0.95$ | $0.89$ |  | $0.94$ | $0.83$
L3-L2 | $0.96$ | $0.95$ |  | $0.95$ | $0.91$
| L1-Ori | $0.84$ | $0.84$ |  | $0.81$ | $0.75$
| L2-L1 | $0.98$ | $0.95$ |  | $0.98$ | $0.92$
Remove char (RC) | L3-L2 | $0.99$ | $0.99$ |  | $0.99$ | $0.98$
Misspelling (MS) | L1-Ori | $0.37$ | $0.54$ |  | $0.37$ | $0.49$
L2-L1 | $0.65$ | $0.55$ |  | $0.62$ | $0.53$
L3-L2 | $0.80$ | $0.67$ |  | $0.78$ | $0.65$
| L1-Ori | $0.92$ | $0.94$ |  | $0.90$ | $0.89$
| L2-L1 | $1.00$ | $1.00$ |  | $1.00$ | $1.00$
Swap (SW) | L3-L2 | $0.99$ | $0.99$ |  | $0.94$ | $0.97$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 20: Wilcoxon signed-rank test on SpeechBrain system’s uncertainty score
distribution difference between clean data and corrupted data on LibriSpeech
dataset.

Corruption | Settings⋄ | Configuration 1 |  | Configuration 2
---|---|---|---|---
${\mathcal{D}_{VR}}^{\dagger}$ |  | ${\mathcal{D}_{VRO}}^{\dagger}$ |  | ${\mathcal{D}_{VR}}^{\dagger}$ |  | ${\mathcal{D}_{VRO}}^{\dagger}$
CRDNN | RNN |  | CRDNN | RNN |  | CRDNN | Trans- |  | CRDNN | Trans-
| L1-Ori | $0.65$ | $0.58$ |  | $0.65$ | $0.56$ |  | $0.72$ | $0.97$ |  | $0.76$ | $0.98$
| L2-L1 | $0.83$ | $0.94$ |  | $0.83$ | $0.91$ |  | $0.93$ | $0.99$ |  | $0.95$ | $0.99$
Reverb (RE) | L3-L2 | $0.66$ | $0.97$ |  | $0.65$ | $0.95$ |  | $0.70$ | $0.91$ |  | $0.72$ | $0.89$
Speed (SP) | L1-Ori | $0.65$ | $0.47$ |  | $0.64$ | $0.44$ |  | $0.58$ | $0.64$ |  | $0.60$ | $0.68$
L2-L1 | $0.94$ | $0.93$ |  | $0.94$ | $0.91$ |  | $0.95$ | $0.98$ |  | $0.96$ | $0.98$
L3-L2 | $0.90$ | $0.98$ |  | $0.89$ | $0.96$ |  | $0.99$ | $0.97$ |  | $0.99$ | $0.96$
| L1-Ori | $0.62$ | $0.45$ |  | $0.62$ | $0.43^{*}$ |  | $0.50^{*}$ | $0.60$ |  | $0.51^{*}$ | $0.63$
| L2-L1 | $0.60$ | $0.54$ |  | $0.61$ | $0.53$ |  | $0.51^{*}$ | $0.70$ |  | $0.52^{*}$ | $0.75$
Distortion (DI) | L3-L2 | $0.79$ | $0.88$ |  | $0.78$ | $0.84$ |  | $0.72$ | $0.96$ |  | $0.75$ | $0.97$
Pitch shift (PS) | L1-Ori | $0.79$ | $0.42^{*}$ |  | $0.81$ | $0.40^{*}$ |  | $0.54$ | $0.68$ |  | $0.56$ | $0.72$
L2-L1 | $0.66$ | $0.55$ |  | $0.66$ | $0.52$ |  | $0.63$ | $0.75$ |  | $0.65$ | $0.78$
L3-L2 | $0.77$ | $0.74$ |  | $0.77$ | $0.72$ |  | $0.70$ | $0.91$ |  | $0.73$ | $0.92$
| L1-Ori | $0.63$ | $0.60$ |  | $0.63$ | $0.56$ |  | $0.93$ | $0.99$ |  | $0.95$ | $0.99$
| L2-L1 | $0.61$ | $0.71$ |  | $0.61$ | $0.67$ |  | $0.84$ | $0.95$ |  | $0.85$ | $0.96$
Background noise (BN) | L3-L2 | $0.54$ | $0.75$ |  | $0.54$ | $0.71$ |  | $0.60$ | $0.81$ |  | $0.60$ | $0.77$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 21: Wilcoxon signed-rank test on SpeechBrain system’s uncertainty score
distribution difference between clean data and corrupted data on CommonVoice-
it dataset.

Corruption | Settings⋄ | Configuration 3 |  | Configuration 4
---|---|---|---|---
${\mathcal{D}_{VR}}^{\dagger}$ |  | ${\mathcal{D}_{VRO}}^{\dagger}$ |  | ${\mathcal{D}_{VR}}^{\dagger}$ |  | ${\mathcal{D}_{VRO}}^{\dagger}$
CRDNN | RNN |  | CRDNN | RNN |  | Wav2Vec2 | RNN |  | Wav2Vec2 | RNN
| L1-Ori | $0.69$ | $0.76$ |  | $0.72$ | $0.73$ |  | $1.00$ | $0.64$ |  | $1.00$ | $0.63$
| L2-L1 | $0.95$ | $0.96$ |  | $0.96$ | $0.95$ |  | $1.00$ | $0.96$ |  | $1.00$ | $0.95$
Reverb (RE) | L3-L2 | $0.85$ | $0.91$ |  | $0.86$ | $0.88$ |  | $0.97$ | $0.97$ |  | $0.93$ | $0.96$
Speed (SP) | L1-Ori | $0.92$ | $0.82$ |  | $0.93$ | $0.80$ |  | $0.95$ | $0.45$ |  | $0.96$ | $0.44$
L2-L1 | $0.73$ | $0.88$ |  | $0.74$ | $0.85$ |  | $0.95$ | $0.77$ |  | $0.93$ | $0.75$
L3-L2 | $0.98$ | $0.95$ |  | $0.98$ | $0.93$ |  | $1.00$ | $1.00$ |  | $0.99$ | $1.00$
| L1-Ori | $0.70$ | $0.57$ |  | $0.71$ | $0.55$ |  | $0.88$ | $0.38$ |  | $0.88$ | $0.37$
| L2-L1 | $0.66$ | $0.61$ |  | $0.67$ | $0.58$ |  | $0.93$ | $0.50$ |  | $0.92$ | $0.48$
Distortion (DI) | L3-L2 | $0.79$ | $0.84$ |  | $0.81$ | $0.81$ |  | $0.99$ | $0.82$ |  | $0.99$ | $0.81$
Pitch shift (PS) | L1-Ori | $0.62$ | $0.56$ |  | $0.62$ | $0.54$ |  | $0.97$ | $0.39$ |  | $0.96$ | $0.37$
L2-L1 | $0.74$ | $0.65$ |  | $0.75$ | $0.63$ |  | $0.92$ | $0.47$ |  | $0.92$ | $0.45$
L3-L2 | $0.81$ | $0.81$ |  | $0.82$ | $0.78$ |  | $0.98$ | $0.72$ |  | $0.97$ | $0.70$
| L1-Ori | $0.93$ | $0.75$ |  | $0.93$ | $0.73$ |  | $1.00$ | $0.60$ |  | $1.00$ | $0.59$
| L2-L1 | $0.79$ | $0.78$ |  | $0.80$ | $0.75$ |  | $1.00$ | $0.75$ |  | $0.99$ | $0.74$
Background noise (BN) | L3-L2 | $0.74$ | $0.79$ |  | $0.75$ | $0.75$ |  | $0.99$ | $0.84$ |  | $0.98$ | $0.82$

  * $\diamond$

A setting of “A-B” indicates that the test is conducted between uncertainty
score on A and B, _e.g_., “L1-Ori” means the test is conducted between level 1
corrupted data and clean data.

  * $\dagger$

An effect size $\mathcal{D}$ indicates the probability that uncertainty score
on corrupted data is larger than it on clean data; larger the effect size,
more significant the result is.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 18, Table 19, and Table 20&21 show the Wilcoxon signed-rank test results
on differences between uncertainty score of clean data and corrupted data on
ByteTrack system, ConvLab-2 system, and SpeechBrain system, respectively. In
these tables, an effect size larger than 0.5 indicates that this module’s
uncertainty score on corrupted data is significantly larger than it on clean
data. Larger the value, the more significant the result is. In the ConvLab-2
system (Table 19) and SpeechBrain system (Table 20&21), we can find that the
uncertainty score measured by either VR or VRO on corrupted data is usually
larger than it on clean data, demonstrating that uncertainty estimation
techniques can capture the system’s prediction difference on clean data and
corrupted data. Moreover, we also find that in the ConvLab-2 and SpeechBrain
systems, a larger severity level of corruption leads to a larger effect size,
showing that the system’s predictions on high-severity corrupted data are more
uncertain than with predictions on clean data.

There are also some exceptions in the results. For instance, we find that in
the ByteTrack system, sometimes the uncertainty score measured by VR on
corrupted data is not larger than it on clean data (_e.g_., DeepSORT module
against Gaussian noise pattern). Further investigation results suggest that
this is because the tracking module’s robustness is significantly affected by
these common corruption patterns. Therefore, a lot of prediction results
(_i.e_., tracking results) from the tracking module are empty tracklets
(“nothing tracked”). Recall that the calculation of VR is based on the major
prediction results after applying Monte Carlo dropout. When there are a lot of
empty tracklets, it’s not surprising that VR on corrupted data is even lower,
especially on high severity corrupted data. While in such cases, VRO performs
much better than VR. In Table 18, the effect size suggests that the
uncertainty score measured by VRO on corrupted data is significantly larger
than it on clean data. We also find that in the ConvLab-2 system, both VR and
VRO work only effectively on the NLU module. However, it’s hard for them to
measure the uncertainty on the other modules, especially DST and NLG.

####  4.3.2 Correlations of different modules’ uncertainty change

Table 22: Kendall’s $\tau$ test on correlation between YOLO detector’s
uncertainty score change and different tracker’s uncertainty score change in
ByteTrack system.

Corruption | Severity |  Kendall’s $\tau$-statistic of VR† |  |  Kendall’s $\tau$-statistic of VRO†
---|---|---|---|---
Track | SORT | DeepSORT |  | Track | SORT | DeepSORT
| 1 | $0.139$ | $0.141$ | $-0.01^{*}$ |  | $0.172$ | $0.143$ | $0.025^{*}$
| 2 | $0.236$ | $0.213$ | $-0.08$ |  | $0.243$ | $0.256$ | $0.015^{*}$
Gaussian noise (GN) | 3 | $0.302$ | $0.174$ | $0.07$ |  | $0.454$ | $0.358$ | $0.127$
Motion blur (MB) | 1 | $0.058$ | $0.036$ | $-0.03^{*}$ |  | $0.058$ | $0.042$ | $-0.02^{*}$
2 | $0.113$ | $0.101$ | $0.01^{*}$ |  | $0.144$ | $0.141$ | $0.04^{*}$
3 | $0.070$ | $0.093$ | $-0.186$ |  | $0.150$ | $0.153$ | $-0.04$
| 1 | $0.089$ | $0.103$ | $-0.03^{*}$ |  | $0.112$ | $0.083$ | $0.04^{*}$
| 2 | $0.187$ | $0.167$ | $-0.097$ |  | $0.211$ | $0.204$ | $-0.02$
Contrast (CO) | 3 | $0.478$ | $0.382$ | $0.397$ |  | $0.381$ | $0.323$ | $0.314$
Snow (SN) | 1 | $0.134$ | $0.131$ | $-0.039^{*}$ |  | $0.218$ | $0.231$ | $0.041$
2 | $0.237$ | $0.188$ | $-0.144$ |  | $0.288$ | $0.268$ | $0.025$
3 | $0.168$ | $0.115$ | $-0.291$ |  | $0.276$ | $0.233$ | $-0.084$
| 1 | $0.038$ | $0.055$ | $-0.013^{*}$ |  | $0.064$ | $0.062$ | $0.025^{*}$
| 2 | $0.116$ | $0.086$ | $0.006^{*}$ |  | $0.184$ | $0.160$ | $0.047$
Pixelate (PI) | 3 | $0.102$ | $0.087$ | $-0.090$ |  | $0.162$ | $0.143$ | $0.043$

  * $\dagger$

A positive Kendall’s $\tau$-statistic means when detector’s uncertainty score
increases on corrupted data comparing with it on clean data, then tracker’s
uncertainty score also increases. A larger $\tau$-statistic suggests more
significant correlation.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 23: Kendall’s $\tau$ test on correlation between two module’s (module
1—module 2) uncertainty score change in SpeechBrain system.

Corruption | Severity |  Kendall’s $\tau$-statistic of VR† |  |  Kendall’s $\tau$-statistic of VRO†
---|---|---|---|---
Config-1 | Config-2 | Config-3 | Config-4 |  | Config-1 | Config-2 | Config-3 | Config-4
| 1 | $-0.036^{*}$ | $0.089$ | $0.105$ | $0.411$ |  | $-0.061$ | $0.091$ | $0.099$ | $0.489$
| 2 | $0.012$ | $0.082$ | $0.132$ | $0.355$ |  | $0.002^{*}$ | $0.076$ | $0.131$ | $0.434$
Reverb (RE) | 3 | $0.061$ | $0.014^{*}$ | $0.111$ | $0.218$ |  | $0.075$ | $0.027$ | $0.115$ | $0.319$
Speed (SP) | 1 | $-0.045$ | $0.024^{*}$ | $0.197$ | $0.368$ |  | $-0.060$ | $0.034^{*}$ | $0.194$ | $0.506$
2 | $0.087$ | $0.303$ | $0.212$ | $0.510$ |  | $0.055$ | $0.335$ | $0.207$ | $0.611$
3 | $0.132$ | $0.009^{*}$ | $0.097$ | $0.214$ |  | $0.0.134$ | $0.029^{*}$ | $0.122$ | $0.330$
| 1 | $-0.074$ | $0.004^{*}$ | $0.091$ | $0.374$ |  | $-0.088$ | $0.009^{*}$ | $0.088$ | $0.572$
| 2 | $-0.031^{*}$ | $0.030^{*}$ | $0.173$ | $0.508$ |  | $-0.057$ | $0.044$ | $0.164$ | $0.674$
Distortion (DI) | 3 | $0.139$ | $0.208$ | $0.234$ | $0.598$ |  | $0.109$ | $0.239$ | $0.226$ | $0.704$
Pitch shift (PS) | 1 | $-0.067$ | $0.014^{*}$ | $0.047$ | $0.295$ |  | $-0.083$ | $0.022^{*}$ | $0.036$ | $0.381$
2 | $0.007^{*}$ | $0.086$ | $0.110$ | $0.395$ |  | $-0.024^{*}$ | $0.102$ | $0.106$ | $0.523$
3 | $0.174$ | $0.229$ | $0.152$ | $0.502$ |  | $0.137$ | $0.271$ | $150$ | $0.592$
| 1 | $-0.020^{*}$ | $0.259$ | $0.141$ | $0.463$ |  | $-0.036^{*}$ | $0.290$ | $0.132$ | $0.641$
| 2 | $0.004^{*}$ | $0.136$ | $0.169$ | $0.535$ |  | $-0.003^{*}$ | $0.158$ | $0.163$ | $0.699$
Background noise (BN) | 3 | $0.002^{*}$ | $0.007^{*}$ | $0.198$ | $0.554$ |  | $-0.006^{*}$ | $0.044^{*}$ | $0.190$ | $0.711$

  * $\dagger$

A positive Kendall’s $\tau$-statistic means when module 1’s uncertainty score
increases on corrupted data comparing with it on clean data, then module 2’s
uncertainty score also increases. A larger $\tau$-statistic suggests more
significant correlation.

  * $*$

This result is not statistically significant ($p>0.0001$).

We show Kendall’s $\tau$ test results on the ByteTrack system and SpeechBrain
system in Table 22 and Table 23, while there is no evidence suggesting that
there exists any correlation between different modules’ uncertainty scores in
the ConvLab-2 system. We find that when using Track/SORT as the tracker in the
ByteTrack system, Kendall’s $\tau$-statistic suggests that there is some
correlation between the detector’s and tracker’s uncertainty score change.
However, when using DeepSORT as the tracker, Kendall’s $\tau$-statistic is
usually around 0, suggesting that there is no correlation. One plausible
explanation is that, different from Track/SORT only takes the bounding box
from the detector as input, whereas DeepSORT also takes the image features as
an extra input (See Fig. 2 (B)), resulting in a more complex data flow in the
system. In the SpeechBrain system, we find that Kendall’s $\tau$-statistic
suggests there is some correlation between two modules’ uncertainty score
change. Specifically, some configurations have strong correlations, _e.g_.,
Configuration 4 (Wav2Vec2+RNN on CommonVoice-it dataset).

Answer to RQ2: In most cases, both studied uncertainty estimation methods
(_i.e_., VR and VRO) can capture the difference in the model’s prediction
change on clean data and corrupted data according to Wilcoxon signed-rank test
results. In some cases (_e.g_., ByteTrack system), VRO performs better than
VR. We also find that one module’s uncertainty score change does not always
correlate to another module’s uncertainty score change according to Kendall’s
$\tau$ test in an AI system.

###  4.4 RQ3. How do the errors propagate among modules in AI systems?

In RQ2 and RQ3, we’ve investigated how our OOD detection and uncertainty
estimation techniques work on AI systems. RQ3 aims to investigate whether such
OOD/uncertainty information correlates to the system output error, which could
be helpful to analyze the error propagation effects. We use Kendall’s $\tau$
test to measure if there is any correlation between each module’s uncertainty
score change and system output error change. Specifically, we leverage the
results from the SpeechBrain system as an example in this section. Based on
our findings in RQ2 and RQ3, we use VRO as the uncertainty estimation
technique and KDE as the OOD detection technique.

Table 24: Kendall’s $\tau$ test on the correlation between module’s
uncertainty (VRO) and system output error in SpeechBrain system.

Corruption | Severity | Configuration 1 |  | Configuration 2 |  | Configuration 3 |  | Configuration 4
---|---|---|---|---|---|---|---|---
M-1† |  M-2⋄ |  |  M-1† |  M-2⋄ |  |  M-1† |  M-2⋄ |  |  M-1† |  M-2⋄
| 1 | $-0.06$ | $0.34$ |  | $0.08$ | $0.19$ |  | $0.09$ | $0.27$ |  | $0.23$ | $0.34$
| 2 | $-0.01^{*}$ | $0.48$ |  | $0.09$ | $0.26$ |  | $0.11$ | $0.35$ |  | $0.23$ | $0.47$
Reverb (RE) | 3 | $0.01^{*}$ | $0.40$ |  | $0.03^{*}$ | $0.23$ |  | $0.10$ | $0.36$ |  | $0.12$ | $0.41$
Speed (SP) | 1 | $-0.06$ | $0.32$ |  | $0.01^{*}$ | $0.05$ |  | $0.18$ | $0.38$ |  | $0.19$ | $0.29$
2 | $0.04^{*}$ | $0.62$ |  | $0.34$ | $0.53$ |  | $0.19$ | $0.40$ |  | $0.29$ | $0.46$
3 | $0.03^{*}$ | $0.27$ |  | $-0.01^{*}$ | $0.81$ |  | $0.10$ | $0.24$ |  | $0.05$ | $0.23$
| 1 | $-0.08$ | $0.33$ |  | $-0.01^{*}$ | $0.02$ |  | $0.10$ | $0.24$ |  | $0.22$ | $0.29$
| 2 | $-0.03^{*}$ | $0.37$ |  | $0.01^{*}$ | $0.11$ |  | $0.16$ | $0.35$ |  | $0.35$ | $0.43$
Distortion (DI) | 3 | $0.10$ | $0.63$ |  | $0.24$ | $0.52$ |  | $0.21$ | $0.49$ |  | $0.46$ | $0.61$
Pitch shift (PS) | 1 | $-0.09$ | $0.30$ |  | $-0.01^{*}$ | $0.01$ |  | $0.03$ | $0.17$ |  | $0.12$ | $0.21$
2 | $-0.01^{*}$ | $0.38$ |  | $0.06$ | $0.15$ |  | $0.11$ | $0.28$ |  | $0.19$ | $0.30$
3 | $0.13$ | $0.53$ |  | $0.23$ | $0.42$ |  | $0.14$ | $0.37$ |  | $0.29$ | $0.44$
| 1 | $-0.04^{*}$ | $0.40$ |  | $0.30$ | $0.48$ |  | $0.12$ | $0.33$ |  | $0.28$ | $0.37$
| 2 | $-0.01^{*}$ | $0.46$ |  | $0.17$ | $0.43$ |  | $0.15$ | $0.41$ |  | $0.34$ | $0.47$
Background noise (BN) | 3 | $-0.01^{*}$ | $0.51$ |  | $-0.01^{*}$ | $0.25$ |  | $0.17$ | $0.46$ |  | $0.36$ | $0.52$

  * $\dagger$

$\tau$-statistic of the correlation between Module 1’s (M-1) uncertainty
change (VRO) and system output error (WER) change.

  * $\diamond$

$\tau$-statistic of the correlation between Module 2’s (M-2) uncertainty
change (VRO) and system output error (WER) change.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 25: Kendall’s $\tau$ test on the correlation between module’s OOD score
(KDE) and system output error in SpeechBrain system.

Corruption | Severity | Configuration 1 |  | Configuration 2 |  | Configuration 3 |  | Configuration 4
---|---|---|---|---|---|---|---|---
M-1† |  M-2⋄ |  |  M-1† |  M-2⋄ |  |  M-1† |  M-2⋄ |  |  M-1† |  M-2⋄
| 1 | $0.04^{*}$ | $0.03^{*}$ |  | $0.02^{*}$ | $0.34$ |  | $0.01^{*}$ | $0.23$ |  | $-0.01^{*}$ | $0.27$
| 2 | $-0.03^{*}$ | $-0.01^{*}$ |  | $0.09$ | $0.40$ |  | $-0.01^{*}$ | $0.15$ |  | $-0.02^{*}$ | $0.35$
Reverb (RE) | 3 | $0.06$ | $0.04^{*}$ |  | $0.12$ | $0.22$ |  | $0.02$ | $-0.05^{*}$ |  | $-0.01^{*}$ | $0.13$
Speed (SP) | 1 | $0.01^{*}$ | $0.04^{*}$ |  | $0.05^{*}$ | $0.14$ |  | $0.06$ | $0.36$ |  | $0.02^{*}$ | $0.21$
2 | $-0.01^{*}$ | $0.01^{*}$ |  | $0.31$ | $0.60$ |  | $0.03$ | $0.20$ |  | $-0.01^{*}$ | $0.38$
3 | $0.02$ | $0.09$ |  | $0.04$ | $0.81$ |  | $0.01^{*}$ | $-0.01^{*}$ |  | $0.01^{*}$ | $0.04$
| 1 | $0.03^{*}$ | $0.02^{*}$ |  | $0.07$ | $0.13$ |  | $0.10$ | $0.24$ |  | $0.06$ | $0.24$
| 2 | $0.05^{*}$ | $0.03^{*}$ |  | $0.09$ | $0.24$ |  | $0.13$ | $0.36$ |  | $0.08$ | $0.38$
Distortion (DI) | 3 | $0.04*$ | $-0.04^{*}$ |  | $0.14$ | $0.61$ |  | $0.21$ | $0.43$ |  | $0.11$ | $0.53$
Pitch shift (PS) | 1 | $-0.01^{*}$ | $0.01^{*}$ |  | $0.03^{*}$ | $0.12$ |  | $0.04$ | $0.14$ |  | $0.01^{*}$ | $0.16$
2 | $0.02^{*}$ | $0.01*$ |  | $0.11$ | $0.27$ |  | $0.06$ | $0.26$ |  | $0.01^{*}$ | $0.25$
3 | $0.04^{*}$ | $-0.01^{*}$ |  | $0.26$ | $0.54$ |  | $0.06$ | $0.34$ |  | $-0.01^{*}$ | $0.39$
| 1 | $0.06$ | $0.02^{*}$ |  | $0.38$ | $0.62$ |  | $0.05$ | $0.27$ |  | $0.02^{*}$ | $0.29$
| 2 | $0.07$ | $-0.01^{*}$ |  | $0.34$ | $0.49$ |  | $0.07$ | $0.29$ |  | $0.03$ | $0.38$
Background noise (BN) | 3 | $0.06$ | $0.01^{*}$ |  | $0.23$ | $0.26$ |  | $0.08$ | $0.21$ |  | $0.06$ | $0.41$

  * $\dagger$

$\tau$-statistic of the correlation between Module 1’s (M-1) OOD score change
(KDE) and system output error (WER) change.

  * $\diamond$

$\tau$-statistic of the correlation between Module 2’s (M-2) OOD score change
(KDE) and system output error (WER) change.

  * $*$

This result is not statistically significant ($p>0.0001$).

Table 24 and Table 25 summarize the correlation between the module’s
OOD/uncertainty score change and system output error. Overall, we find that
two indicators (_i.e_., VRO and KDE) behave similarly, where the correlation
between OOD/uncertainty score and system output error is more significant when
measuring on the second module. Surprisingly, in RQ2, we find that OOD
detection methods are more effective in capturing the data distribution shift
when measuring on the first module. However, we find that the first module’s
OOD score change does not have much correlation with the system output error.
In fact, both OOD scores and uncertainty scores measured on the first module
do not show an obvious correlation with the system output error with
configurations 1 and 2. While there is some correlation between the first
module’s uncertainty score change and system output error with configurations
3 and 4, however, such correlation is always less significant than it between
the second module’s uncertainty score change and system output error. We can
also connect these observations with results in Table 23, where Kendall’s
$\tau$ test suggests that there is some correlation between two modules’
uncertainty with configurations 3 and 4. These results indicate that for the
SpeechBrain system, when the second module’s OOD/uncertainty score is high,
it’s relatively possible to trigger an error output.

Besides, we also find that the correlation does not always become stronger
when the severity level increases. For instance, in Table 24, we can observe
that when against level-2 SP corruption, the second module’s uncertainty score
change of configuration 1, 3 & 4 has a stronger correlation with the system
output error compared with them against level-3 corruption. Recall that when
against level-3 SP corruption, the system output error is relatively higher
(see Table 7). This indicates that when against high severity-level corruption
patterns, current uncertainty estimation and OOD detection techniques could be
less effective for risk assessments. Therefore, more advanced OOD detection
techniques and uncertainty estimation are needed for analyzing AI systems,
especially when interpreting system performance against severely corrupted
data.

Answer to RQ3: We find that for a serial executed system, _e.g_., SpeechBrain,
neither the first module’s OOD nor uncertainty information strongly correlates
to the system output error. However, both the second module’s OOD and
uncertainty information can indicate potential output errors of the system.
Besides, when the corruption severity level increases, OOD detection and
uncertainty estimation sometimes become less effective as indicators of the
system’s potential risk. This result also implies that more advanced OOD
detection techniques/uncertainty estimation might be needed at the system
level.

###  4.5 RQ4. How do simple combinations of OOD analysis- and uncertainty-
based method improve the reliability of AI systems?

Figure 4: Five simple risk assessment strategies we used in this study.

In RQ4, we first use the SpeechBrain system as an example to show that a
simple combination of OOD analysis- and uncertainty-based methods can improve
the reliability of AI systems. We compare the following strategies (See Fig.4
for the workflow of each) to reject a certain number of risky predictions for
the systems, suppose we are rejecting $n$ risky predictions:

  * •

Strategy 1 (S1): Reject only uncertain predictions. Through this strategy, we
will reject the predictions with top-$n$ highest uncertainty scores.

  * •

Strategy 2 (S2): Reject only OOD data. Through this strategy, we will reject
the predictions with top-$n$ highest OOD scores.

  * •

Strategy 3 (S3): First reject uncertain predictions, then reject OOD data.
Through this strategy, we will first reject the predictions with
top-$\frac{n}{2}$ highest uncertainty scores, then reject the predictions with
top-$\frac{n}{2}$ highest OOD scores.

  * •

Strategy 4 (S4): First reject OOD data, then reject uncertain predictions.
Through this strategy, we will first reject the predictions with
top-$\frac{n}{2}$ highest OOD scores, then reject the predictions with
top-$\frac{n}{2}$ highest uncertainty scores.

  * •

Strategy 5 (S5): Reject either uncertain predictions or OOD data
simultaneously. Through this strategy, we will first reject rank the
predictions according to uncertainty and OOD scores separately, then we will
reject top-$n$ common predictions between this two ranked lists.

Table 26: SpeechBrain’s performance improvement on LibriSpeech dataset by
rejecting 1% predictions.

Corruption | Severity | Configuration 1 |  | Configuration 2
---|---|---|---|---
S1 (%) | S2 (%) | S3 (%) | S4 (%) | S5 (%) |  | S1 (%) | S2 (%) | S3 (%) | S4 (%) | S5 (%)
| 1 | $27.55$ | $-0.61$ | $-0.61$ | $27.55$ | $27.21$ |  | $2.74$ | $4.53$ | $4.55$ | $2.62$ | $6.69$
| 2 | $0.30$ | $0.05$ | $0.05$ | $0.30$ | $9.66$ |  | $0.39$ | $0.50$ | $0.53$ | $0.44$ | $0.88$
Reverb (RE) | 3 | $0.33$ | $0.28$ | $0.28$ | $0.33$ | $1.99$ |  | $0.01$ | $0.03$ | $0.04$ | $0.04$ | $0.17$
Speed (SP) | 1 | $36.01$ | $1.49$ | $1.49$ | $36.01$ | $37.88$ |  | $4.77$ | $7.76$ | $7.59$ | $4.22$ | $9.82$
2 | $-0.29$ | $0.54$ | $0.54$ | $-0.29$ | $7.12$ |  | $-0.06$ | $0.36$ | $0.34$ | $0.19$ | $2.51$
3 | $-0.05$ | $-0.08$ | $-0.08$ | $-0.05$ | $0.65$ |  | $0.06$ | $0.11$ | $0.08$ | $0.03$ | $0.11$
| 1 | $28.54$ | $-0.47$ | $-0.47$ | $28.54$ | $28.35$ |  | $2.28$ | $4.95$ | $5.19$ | $1.78$ | $6.50$
| 2 | $13.28$ | $-0.60$ | $-0.60$ | $13.28$ | $28.61$ |  | $2.48$ | $5.50$ | $5.49$ | $1.67$ | $6.95$
Distortion (DI) | 3 | $1.21$ | $-0.14$ | $-0.14$ | $1.21$ | $10.10$ |  | $1.29$ | $2.14$ | $2.02$ | $1.07$ | $5.29$
Pitch shift (PS) | 1 | $30.18$ | $4.27$ | $0.22$ | $26.48$ | $30.62$ |  | $3.31$ | $6.41$ | $5.79$ | $2.93$ | $7.61$
2 | $7.11$ | $-0.10$ | $-0.25$ | $7.05$ | $27.58$ |  | $4.14$ | $7.02$ | $6.14$ | $2.62$ | $12.27$
3 | $5.44$ | $0.52$ | $0.29$ | $5.58$ | $20.01$ |  | $2.87$ | $3.08$ | $2.86$ | $2.32$ | $6.12$
| 1 | $30.02$ | $-0.25$ | $-0.25$ | $30.24$ | $30.03$ |  | $1.27$ | $1.56$ | $1.56$ | $1.27$ | $2.71$
| 2 | $2.37$ | $-0.38$ | $-0.38$ | $2.55$ | $20.22$ |  | $0.31$ | $0.36$ | $0.36$ | $0.31$ | $1.08$
Background noise (BN) | 3 | $1.85$ | $-0.42$ | $-0.42$ | $1.99$ | $8.50$ |  | $0.09$ | $0.09$ | $0.09$ | $0.09$ | $0.16$

Table 27: SpeechBrain’s performance improvement on CommonVoice-it dataset by
rejecting 1% predictions.

Corruption | Severity | Configuration 3 |  | Configuration 4
---|---|---|---|---
S1 (%) | S2 (%) | S3 (%) | S4 (%) | S5 (%) |  | S1 (%) | S2 (%) | S3 (%) | S4 (%) | S5 (%)
| 1 | $1.15$ | $0.63$ | $0.67$ | $1.26$ | $1.72$ |  | $2.21$ | $2.13$ | $1.04$ | $1.86$ | $3.13$
| 2 | $0.47$ | $0.37$ | $0.36$ | $0.47$ | $1.13$ |  | $0.83$ | $0.85$ | $0.73$ | $0.80$ | $1.89$
Reverb (RE) | 1 | $0.15$ | $0.07$ | $0.06$ | $0.14$ | $0.69$ |  | $0.25$ | $0.27$ | $0.24$ | $0.27$ | $1.63$
Speed (SP) | 1 | $1.99$ | $1.51$ | $1.32$ | $1.83$ | $3.25$ |  | $3.13$ | $1.72$ | $1.69$ | $2.96$ | $4.17$
2 | $0.11$ | $0.21$ | $0.19$ | $0.09$ | $1.07$ |  | $0.30$ | $0.56$ | $0.64$ | $0.34$ | $2.61$
3 | $0.00$ | $0.04$ | $0.02$ | $-0.00$ | $0.27$ |  | $0.01$ | $0.03$ | $0.08$ | $0.02$ | $1.05$
| 1 | $1.65$ | $1.79$ | $1.71$ | $1.65$ | $3.13$ |  | $1.99$ | $1.69$ | $1.60$ | $1.83$ | $3.21$
| 2 | $1.20$ | $1.73$ | $1.64$ | $1.19$ | $3.00$ |  | $2.02$ | $1.76$ | $1.51$ | $1.94$ | $3.75$
Distortion (DI) | 1 | $1.05$ | $1.03$ | $0.94$ | $1.06$ | $2.50$ |  | $1.38$ | $1.75$ | $1.71$ | $1.33$ | $3.02$
Pitch shift (PS) | 1 | $2.76$ | $1.81$ | $1.57$ | $1.95$ | $3.55$ |  | $1.70$ | $3.78$ | $3.62$ | $1.60$ | $5.06$
2 | $1.84$ | $1.76$ | $1.58$ | $1.47$ | $2.38$ |  | $3.94$ | $2.09$ | $2.03$ | $3.78$ | $5.40$
3 | $0.87$ | $0.76$ | $0.63$ | $0.79$ | $3.01$ |  | $2.25$ | $2.11$ | $1.27$ | $1.41$ | $4.53$
| 1 | $1.63$ | $0.69$ | $0.67$ | $1.62$ | $2.28$ |  | $1.94$ | $1.76$ | $1.28$ | $1.91$ | $3.05$
| 2 | $1.00$ | $0.50$ | $0.47$ | $0.98$ | $1.70$ |  | $1.51$ | $0.99$ | $1.01$ | $1.52$ | $2.48$
Background noise (BN) | 1 | $0.72$ | $0.39$ | $0.39$ | $0.72$ | $1.18$ |  | $1.24$ | $0.84$ | $0.84$ | $1.23$ | $2.55$

(a) Configuration 1: CRDNN+RNN on LibriSpeech dataset

(b) Configuration 2: CRDNN+Transformer on LibriSpeech dataset

(c) Configuration 3: CRDNN+RNN on CommonVoice-it dataset

(d) Configuration 4: Wav2Vec2+RNN on CommonVoice-it dataset

Figure 5: SpeechBrain’s output error (WER, lower indicates better) changes
when rejecting a certain number of uncertain/OOD data.

Based on our findings in RQ2-4, we use the OOD score measured by KDE on module
2 and the uncertainty score calculated by VRO on module 2 when investigating
this research question. In practice, we use these strategies to reject 1%, 3%,
and 5% unreliable predictions for different configurations of the SpeechBrain
system accordingly. While reporting all the results might take too much space,
here we mainly report the full results of rejecting 1% unreliable predictions
in Table 26 and Table 27. As we can see, the system’s performance can be
improved by these strategies to different degrees. Overall, we find that the
combinations of OOD and uncertainty (S3, S4, and S5) usually perform better
than only using one of them (S1 and S2). Among S3, S4, and S5, the improvement
through S5 (reject either uncertain predictions or OOD data simultaneously) is
the most significant one compared with other strategies. For instance, in
configuration 1, S5 can improve the system’s performance by up to 37.9% (when
against level 1 Speed corruption). We also observe that S1 performs better
than S2 in some configurations (configuration 1, 3, and 4), while S2 performs
better than S1 in configuration 2. These results also suggest that neither OOD
nor uncertainty score is always better than the other one, considering a
combination of them could be a better choice. We further show the system’s
performance improvement by S5 compared with S1 when rejecting a different
number of unreliable predictions (1%, 3%, and 5%) in Fig. 5. Compared with S1
(dashed blue, orange, and green line vs. the solid red line), S5 improves
system performance (solid blue, orange, and green line vs. the solid red
line).

We also find that these strategies are usually more effective in improving
system performance against lower severity level (_i.e_., level-1) corruption
compared with higher ones. For instance, in configuration 1, S1 can usually
improve the system performance by over 27% when against level-1 corruption.
However, the improvement on level-2 and level-3 ones is limited, especially on
level-3. When using a combination of OOD and uncertainty, _i.e_., S5, the
system performance improvement is still limited when against level-3
corruption. These results align with our findings in RQ4, implying that OOD
detection techniques and uncertainty estimation for AI systems can be further
improved against highly corrupted data.

Answer to RQ4: We demonstrate that simple strategies of combining OOD
analysis- and uncertainty-based methods can improve the system’s performance
and reliability. In the SpeechBrain system, we find that only rejecting 1%
uncertain/OOD data can improve the system’s performance by up to 37.8%. These
results could motivate the need for more advanced research strategies for AI
system risk assessments.

##  5 Discussions and Future Opportunities

In this section, we discuss the implications of our empirical studies as well
as the identified future opportunities.

###  5.1 Common corruption patterns bring risks to AI systems that should be
seriously treated towards enhancing the system quality.

Common corruption patterns have been shown threatening to the performance of
unit-level AI modules (_i.e_., a single DNN model), and performance on
corrupted data thus becomes an important indicator of analyzing a single DNN’s
robustness. Yet limited investigation has been made on how these corruption
patterns can put AI systems at risk, in this study, we leverage common
corruption patterns from three different domains (_i.e_., image, text, and
audio) to create corrupted datasets with different levels of severity. Our
results from RQ0 show that, even at the system level, these common corruption
patterns still pose threats to the system’s quality. For instance, in the
SpeechBrain system (Table 7), even a low level of severity could significantly
decrease the system performance by 185.7% 111We manually check the audio clips
with level-1 corruption and find that such corruption cannot affect human’s
perception.. We also find that for an identical system, _e.g_., ByteTrack,
both fully AI-enabled configuration (_e.g_., YOLO+DeepSORT) and partial AI-
enabled one (YOLO+Track) can be threatened by common corruption patterns.