# Enhancing Deep Neural Networks Testing by Traversing Data Manifold

Yuanyuan Yuan, Qi Pang, and Shuai Wang  The Hong Kong University of Science
and Technology
<EMAIL_ADDRESS>
###### Abstract

To launch testing toward deep neural networks (DNNs), we often need to perform
massive mutations on the inputs and validate the mutated outputs. DNNs often
accept high-dimensional media data (e.g., photos, text, and audio) as inputs
and make judgments based on the perceptual content (e.g., a cat) of the media
data. The majority of available mutation methods either perform merely byte-
level mutations, which are likely to violate the format of perceptual content,
or apply a limited number of domain-specific filters (e.g., rain, fog). The
validity of mutated outputs is checked separately using domain-specific
criteria or heuristics, which are often poorly generalizable.

This study is inspired by the ability of data manifolds to capture critical
perceptions of high-dimensional media data in a low-dimensional space. We
reframe the mutation and validation of media data as the building and
traversal of data manifolds, thus addressing both difficult tasks in a unified
manner. Specifically, when high-dimensional media data are projected into a
low-dimensional manifold, the data can be mutated by traversing the manifold.
When contrasted with the input, the mutated data exhibit a gradual change in
the perceptual traits (e.g., left-oriented vs. right-oriented face) but
preserve the semantics-level constraints (i.e., a portrait remains a
portrait), thereby retaining the validity of mutated data by design.
Therefore, we rapidly stress DNNs by directly changing perceptual traits,
which are the focus of DNNs.

We develop DeepTraversal, a feedback-driven framework to test DNNs.
DeepTraversal first launches an offline phase to map media data of various
forms to manifolds. Then, in its online testing phase, DeepTraversal traverses
the prepared manifold space to maximize DNN coverage criteria and trigger
prediction errors. In our evaluation, DNNs executing various tasks (e.g.,
classification, self-driving, machine translation) and media data of different
types (image, audio, text) were used. DeepTraversal exhibits better
performance than prior methods with respect to popular DNN coverage criteria
and it can discover a larger number and higher quality of error-triggering
inputs. The tested DNN models, after being repaired with findings of
DeepTraversal, achieve better accuracy.

##  I Introduction

Despite the rapid advancement of deep learning techniques, deep neural
networks (DNNs), like traditional software, can make incorrect predictions
that can have fatal consequences. DNN models are currently used in a variety
of safety and security-critical applications, such as medical image processing
and self-driving cars. In such systems, incorrect edge-case behaviors are
shown to result in financial loss and serious dangers to public safety [55,
32]. For example, autonomous driving models are reported to have caused deadly
collisions [7, 21]. Machine translation systems can also produce errors that
might cause misunderstandings or ethical violations (e.g., discrimination)
[18, 56, 41, 25].

Recent research has conducted fuzz testing toward DNNs by randomly changing
their input data bytes (e.g., adding random noise to images) [48]. Due to the
format defined for high-dimensional media data, any mutation that violates the
format is likely to result in invalid input. Worse, media data (e.g., a
portrait) are meaningful, and semantics-agnostic mutations may corrupt
perceptions and produce meaningless data. In addition, adding “extreme weather
conditions” to image or employing knowledge transfer techniques [55, 63] have
recently been used to mutate media data. However, these methods are frequently
limited to a single scenario or data type, such as traffic scene images [45,
58, 16, 44], or natural language text [18, 56, 41, 25]. Further, to generate
semantically meaningful data, most existing works use heuristics or domain-
specific information, such as bounding the number of perturbed pixels [61] or
distance of features estimated by an oracle model [63]. These validation
schemes are often poorly generalizable and heavily reliant on pre-trained
oracle models.

Recent advances in representation learning show that data manifolds can
encapsulate key perceptions of high-dimensional media data in a low-
dimensional and continuous space [6, 66]. Thus, the mutation of an intrinsic
coordinate in the projected manifold can yield high-dimensional media data
with a gradual change in perceptual-level content (see Fig. 1). More
crucially, a manifold preserves semantics-level constraints (e.g., a cat is
still a cat), thus retaining the realism and validity of the modified media
data. We therefore present DeepTraversal, a manifold-based fuzz testing
framework for DNNs, which is inspired by the widespread success of manifold
learning in other disciplines. First, DeepTraversal launches an offline phase
to prepare a continuous, low-dimensional manifold $\mathcal{M}$ with an
encoder $G^{-1}$, which is mirrored from the generator component $G$ of a
generative adversarial network (GAN) [20]. Then, DeepTraversal launches an
online fuzzing phase, in which the prepared manifold, $\mathcal{M}$, is
traversed and the covered coordinates on the manifold are mapped back to the
high-dimensional space with $G$. Thus, media data that exhibit a gradual
change but are meaningful can be retrieved to stress DNNs. DeepTraversal
launches traversal on manifolds under the feedback of DNN coverage criteria
(e.g., neuron coverage [48]), aiming at maximizing the DNN coverage criteria
and detecting errors. DeepTraversal is a unified framework that can mutate
media data of many types, such as images, audio, and natural language text. It
can produce mutated media data that correspond to varied perceptual contents,
while retaining the validity of mutated data by design. Therefore,
DeepTraversal delivers format-agnostic media data mutation and does not
necessitate domain-specific knowledge, thus revealing a methodical and
successful new paradigm for DNN fuzz testing.

Figure 1: Two continuous and semantic-preserving mutations enabled by
DeepTraversal in an automated manner.

Ten popular DNNs that analyze media data in image, audio, and text formats are
evaluated. The working contexts range from generic (e.g., image
classification) to specific (e.g., autonomous driving). We compared the
fuzzing effectiveness of DeepTraversal with that of four de facto DNN fuzzing
frameworks — DeepHunter [61], TensorFuzz [45], DeepSmartFuzzer [14], and
DeepTest [55] — using five well-established DNN coverage criteria. We show
that DeepTraversal has considerably better performance than the previous works
in 31 out of 33 settings. In comparison with all previous works, DeepTraversal
also demonstrates promising results by recognizing more and diverse error-
triggering inputs. Data mutated by DeepTraversal are of excellent quality and
can be used to achieve better augmentation of DNN models, outperforming all
previous works. By mutating picture, text, and audio inputs, we demonstrate
the versatility of DeepTraversal in fuzzing DNN models used by autonomous
driving, machine translation, and audio classification scenarios. In summary,
the following major contributions are made by this work:

  * •

Conceptually, we propose a new method for mutating media data to fuzz DNNs,
which first involves the recasting of media data into manifolds. Then, by
traversing manifolds, we generate mutated media data that are gradually
altered but always legitimate. Two difficult challenges encountered by prior
efforts — mutating media data and affirming the legitimacy of mutated data —
are addressed with one stone.

  * •

Technically, DeepTraversal allows the mutation of media data in several
formats, without requiring any domain-specific knowledge or templates. To
launch an effective traversal on a manifold under the guidance of DNN coverage
criteria, DeepTraversal incorporates several design principles and
optimizations.

  * •

We evaluated DeepTraversal by using real-world DNNs that process images,
audio, and text. We show that high-quality, diversified media data are
generated, and that DeepTraversal achieves good performance w.r.t. various
criteria and significantly outperforms all previous frameworks.

##  II Background

By traversing manifolds corresponding to media data, DeepTraversal exhibits
versatility in mutating perceptions of media data in different formats, e.g.,
images, audio, and text. Given that most of the common DNNs can be categorized
into two types, feed-forward neural networks (FNNs) and recurrent neural
networks (RNNs), we first provide an introduction to how perceptions of media
data are extracted and processed by DNNs.

FNN. A Convolutional Neural Network (CNN) denotes one representative FNN. In
general, a CNN is trained to focus on the essential perceptual content of
images, which may include object types, colors, and motions. This training is
mainly accomplished through the use of the convolutional kernel, which
simulates how humans spot and recognize patterns in images, regardless of the
angle from which the images are seen. FNNs are widely used in image
understanding, e.g., classification, where perceptual content is retrieved for
coarse-grained labeling. Further, perceptual features are extensively tagged
with, for example, driver decision or disease information to facilitate
domain-specific activities such as self-driving and disease diagnostics.
Translation invariance is a crucial characteristic enforced by a CNN, which
means that perceptual items (e.g., a cat) may be recognized in an image even
if their appearance changes [33]. This characteristic improves a CNN’s ability
to process a wide range of images in real-world circumstances. However, as
discussed later in Sec. VII, translation invariance can hamper the efficiency
of previous DNN testing methods that alter images using affine transformations
[55]. In contrast, we will show how DeepTraversal can launch effective and
unified testing pipeline toward FNNs utilized in general image understanding
as well as in specific auto-driving and audio processing tasks by directly
altering perceptual content.

RNN. Derived from an FNN, an RNN leverages its internal state to process
variable-length sequences, e.g., natural language sentences. For such data, a
“sequence” usually contains more informative perceptions than individual
elements do [43]. RNNs are commonly used for text comprehension, such as
machine translation (MT), which is one of the most successful applications of
RNNs in natural language processing. Random mutation can easily damage
discrete data like text; therefore, previous RNN testing relied heavily on
predefined text-mutation templates [18, 56, 41, 25]. However, template-based
approaches have a limited generality and are likely to miss faults in real-
life corner cases. In contrast, DeepTraversal delivers perceptual-level
mutation by traversing manifolds of text data. This enables a thorough
exploration of the RNN input space while simultaneously ensuring the validity
of mutated text.

##  III Related Work

Figure 2: Images mutated by existing works and DeepTraversal (the last row).

A key challenge in testing DNNs is the systematic mutation of DNN inputs,
which are often high-dimensional media data. It is also crucial to keep the
validity of the modified data intact. We now review prior efforts that have
motivated our study.

###  III-A Media Data Mutation

Fig. 2 compares image mutation launched by existing methods and by
DeepTraversal. We summarize the application of existing mutation methods to
images and discuss their extension to text:

Pixel-Level Transformation changes image brightness or contrast, or adds
noise. DeepXplore [49] uses pixel-wise mutations in a whitebox context under
the guidance of target DNN’s gradients. It comprises two schemes: randomly
adding black blocks or perturbing user-specified regions with noise generated
from gradients. We examined its codebase [1]: the first scheme is completely
random, whereas the second scheme requires a manually specified region. Pixel-
level mutations can be ineffective as images usually have many pixels. Both
the approaches may damage the image naturalness, as seen in Fig. 2 (e.g.,
tearing objects in a photo).

Affine & Convolutional Transformations translates/rotates/scales/shears an
image while preserving the collinearity after mutation, as shown in Fig. 2. As
discussed in Sec. II, the translation invariance of a CNN may hinder affine
transformations. Convolutional transformations perform holistic modification
of an image, such as blurring or adding domain-specific (e.g., rainy) filters.
Affine and convolutional transformations are described in DeepTest [55].
DeepSmartFuzzer [14] divides an image into sections and mutates them one at a
time. Modifying only one section may result in fragmented images. Overall,
despite offering more holistic mutations, these changes are usually limited
(e.g., to a few severe weather conditions) and require domain-specific
information (e.g., a traffic scene should be rotated by a small angle). Thus,
the tested DNNs’ “behaviors” are not completely studied.

Knowledge Transfer transforms data from one domain to another by leveraging
knowledge from another area. DeepRoad [63] and TACTIC [35] mutate images using
style transfer techniques [69, 38]. Their strength (and limits) stems from
cycle-consistency training [69], a popular learning paradigm for generative
models. For example, given street images taken on sunny and snowy days, these
techniques enhance sunny-street images by gradually changing the weather to
snowy, and vice versa. Weather conditions provided by “cycle-consistency” are
frequently more realistic than those added by convolutional transformations
and are not confined to a few available templates. However, the availability
of photos from two domains limits this strategy (e.g., photos taken from two
weather conditions are needed). “Knowledge transfer” also fails when images in
two domains have significant geometric alterations [67]. This method has only
been used to test autonomous driving [63]. It is difficult to extend the
method to arbitrary real-life scenarios.

Text. Existing works heavily study mutating images. However, real-world DNN
models analyze various types of media data, such as text. A “pixel-level”
mutation similar to that for images is impractical in the case of text because
random perturbation of words will not retain grammatical or linguistic
coherence. Most text mutations are performed using pre-defined templates or
heuristics [18, 56, 41, 25].

###  III-B Media Data Validation

The mutation techniques discussed above can generate unrealistic data, and
therefore, validation is required for existing methods. A grammatical check
can validate natural language sentences. However, validating mutated images is
more difficult and often involves heuristics: TensorFuzz [45] adds pixel-wise
noise to images. It enforces the validity of mutated images by truncating
accumulated noise to a user-defined range. DeepTest [55] limits the parameters
of each transformation to a specific range. DeepTest also assesses mean
squared error (MSE) to rules out likely broken images. However, as found in
Sec. VII-A, many of its mutated images, though passed its validation, are
barely identifiable to humans. DeepHunter [61] allows only one-time affine
transformation on each image. It uses thresholds to bound the maximal #mutated
pixels and mutated pixel values for pixel-level transformations. However, the
quality of mutated images is very sensitive to the thresholds, which are
difficult to determine without experience. DeepRoad [63] ensures authenticity
by rejecting an image with features that are significantly different from
features in training data. The features are extracted by a pre-trained VGG
model [54]. Training VGG is computing-intensive. It also relies on the
availability of the target DNN’s training dataset, which might not be
available in all cases.

##  IV A Manifold View of DNN Testing

As stated in Sec. II, DNNs extract and analyze media data (e.g., face
attributes of a portrait). Manifolds capture crucial media data perceptions in
a low-dimensional, continuous space. We thus aim to launch DNN testing by
traversing the low-dimensional manifold space and mapping the footprints back
to a high-dimensional space to yield data having a gradual change. As
manifolds encapsulate semantic restrictions, traveling within manifolds
preserves the perceptual coherence of mutated data. Thus, data manifolds
address media data mutation and validation as a whole. We now introduce the
concept of manifolds and outline our framework.

Manifold. Manifold explains the feasibility of dimensionality reduction [34].
The key premise of manifold is the manifold hypothesis, which states that
real-world data in high-dimensional space concentrates near a low-dimensional
manifold [17]. That is, real-world data often lie in a manifold $\mathcal{M}$
of much lower dimensionality $d$, which is embedded in its high-dimensional
space $\mathcal{R}$ of dimensionality $D$ ($d\ll D$). Overall, data manifold
can be constructed by finding a projection
$f:\mathcal{R}\rightarrow\mathcal{M}$ that converts data $x\in\mathcal{R}$
into $z$ in an intrinsic coordinate system of $\mathcal{M}$.111“Intrinsic
coordinate” denotes the coordinate system of the low-dimensional manifold
space for each high-dimensional data sample [37, 11]. Thus, $f^{-1}$ projects
$f(x)\in\mathcal{M}$ back to the representation $x$ in the high-dimensional
space $\mathcal{R}$. It is worth noting that the “closeness” of data in
$\mathcal{R}$ is preserved in $\mathcal{M}$. More specifically, data points in
connected regions of $\mathcal{M}$ share similar properties, whereas data of
different classes (e.g., dog vs. vehicle images) lie in distinct manifolds.

Manifold Construction. PCA [2] approximates data manifold via linear
functions. The principal components of each data point $x$ should uniquely
locate its projection on $\mathcal{M}$ [6]. However, given most real-world
mappings between $\mathcal{R}$ and $\mathcal{M}$ as non-linear, various
approaches (e.g., Local Linear Embedding) are proposed to project data $x$ to
$\mathcal{M}$ with non-linear functions [4, 51, 5]. Despite the effectiveness
of non-linear approximation, these approaches are non-parametric, thus mapping
new data point in $\mathcal{R}$ are infeasible. Recent advance in DNNs,
particularly, generative models [68], enables construction of parametric and
non-linear projections $f_{\theta}:\mathcal{R}\rightarrow\mathcal{M}$.

Manifold Traversal. DeepTraversal benefits from the fact that mutating media
data $x$ can be carried via mutating $x$’s coordinate $z=f(x)$ on
$\mathcal{M}$. One key property of manifold is that it encodes constraints
that form perceptually meaningful media data [6]. For instance, the manifold
$\mathcal{M}_{p}$ of portraits, a connected and high-density region, retains
constraints over certain pixels that jointly make up perceptions like head,
ear and nose. A “face-like” photo may always be produced by traversing the
coordinate system of $\mathcal{M}_{p}$ and further mapping
${f(\hat{x})}\in\mathcal{M}_{p}$ back to $\hat{x}\in\mathcal{R}_{p}$. Mutating
$f(x)\rightsquigarrow{f(\hat{x})}$ changes certain perceptions on $\hat{x}$,
e.g., a left-oriented face $x$ turns right in $\hat{x}$. By constantly
traversing $\mathcal{M}$ with steps $[z,z_{1},z_{2},\cdots,z_{k}]$, sequence
$[x,x_{1},x_{2},\cdots,x_{k}]$ gradually change while retaining perceptual
constraints. We present two examples in Fig. 1.

Figure 3: Visualization of the constructed manifold.

Sample Manifold. Fig. 3 depicts the variety of Border Collie photos on a
manifold $\mathcal{M}$. Using DeepTraversal, we project a set of Border Collie
photos into coordinate $z$ on $\mathcal{M}$ of 120 dimensions. To ease the
presentation, we further pick the first two $z$-values, $d_{1}$ and $d_{2}$,
to form a 2D manifold on Fig. 3. We then pick coordinates on Fig. 3, marked in
red cycles, and use DeepTraversal to map them to images. Despite its two-
dimensionality, the visualization in Fig. 3 is informative to clarify the
concept of manifold. The hue of the background grass gathers images. Also,
most Collies stand when $d_{2}<0$ and lie down if $d_{2}>0$ (particularly when
$d_{1}=0$). The value of $d_{1}$ seems to be related with the orientation
since most of the Collies look left when $d_{1}<0$, and vice versa.

We clarify that this manifold is more comprehensive than training data. If we
only have images whose projected $d_{1}$ and $d_{2}$ have the same sign, i.e.,
<lying, toward right> Collies and <standing, toward left> Collies, manifold
$\mathcal{R}$ in Fig. 3 can still be created and enables the construction of
<lying, toward left> Collies and <standing, toward right> Collies. This
reveals the feasibility of launching fully fledged DNN testing by
systematically exploring manifold constructed by its normal inputs.

### High-Level Research Overview

As aforementioned, manifold can be constructed by finding a projection
$f_{\theta}:\mathcal{R}\rightarrow\mathcal{M}$ that converts media data
$x\in\mathcal{R}$ into $z=f_{\theta}(x)$ on $\mathcal{M}$. That said, mapping
$f_{\theta}(x)\in\mathcal{M}$ back to $x\in\mathcal{R}$ requires the
reconstructed data to be natural and realistic. Also, given usually limited
data samples, $f_{\theta}:\mathcal{R}\rightarrow\mathcal{M}$ is often
approximated on a limited number of data. This raises practical challenge to
construct an expressive manifold that can facilitate synthesizing diverse
$x\in\mathcal{R}$.

It is shown that employing neural networks, particularly generative models,
can construct manifolds [66, 27, 42]. Generative adversarial network (GAN)
[20] employs generator network $G$ and discriminator network $D$, with the
objective to make data approximated by $G$ indistinguishable under the view of
$D$. GAN minimizes the “distance” between natural and synthesized data
distributions. It seeks a Nash equilibrium in a two-player zero-sum game:

$\min\limits_{G}\max\limits_{D}\mathbb{E}_{x\sim q_{(x)}}[\log
D(x)]+\mathbb{E}_{z\sim p(z)}[\log(1-D(G(z)))]$

where $z$ is the latent vector of much lower dimensions and is sampled from a
continuous distribution $p(z)$, which is typically set as $\mathcal{N}(0,I)$.
$G$ implicitly encodes constraints formed by natural data.

DeepTraversal is based on GAN. We first launch an offline phase to train GAN,
including generator network $G$ and discriminator network $D$, over a set of
media data. The mirror of the generator network, $G^{-1}$, forms an “encoder”
$f_{\theta}:\mathcal{R}\rightarrow\mathcal{M}$. Then, during the online
testing phase, DeepTraversal maps a seed input $x$ onto its coordinate $z\sim
p(z)$ in the manifold space. Thus, mutating $z\rightsquigarrow z^{\prime}$
will result in footsteps in the manifold, where each footprint will be fed to
$G$, a decoder $f_{\theta}^{-1}:\mathcal{M}\rightarrow\mathcal{R}$, that maps
$z^{\prime}$ back to realistic and valid high-dimensional media data
$G(z^{\prime})$. Since $p(z)$ is continuous, mutating $z\sim p(z)$ forms a
traversal on manifold $\mathcal{M}$.

Figure 4: Comparison between DeepTraversal and existing works by projecting
their mutated images on manifold. Figure 5: Workflow of DeepTraversal.

A Schematic Comparison of Mutation Schemes. DeepTraversal mutates media data
by traversing the created manifold $p(z)$. Fig. 4 compares DeepTraversal and
existing works reviewed in Sec. III. On a data manifold, a seed image $i$ is
mutated when DeepTraversal walks toward the bottom left and the top right.
DeepTraversal, by mapping each footstep back to a high-dimensional space,
outputs a succession of mutated images with gradually-changed perceptions to
stress DNNs.

Existing methods directly mutate media data. Their mutated images typically
surround the seed $i$ in an $\varepsilon$-radius sphere. Some mutated images
breach the perception constraints (e.g., by adding too much random noise),
thus residing outside the manifold (shown in red in Fig. 4; the red dot is on
the sphere and above the manifold). Although these mutated images may trigger
DNN mis-behaviors, they typically stress only a limited range of behaviors of
DNNs, given that not all perceptions focused by the target DNN are covered.
Worse, we emphasize that the $\varepsilon$-radius must be reasonably small, as
images may become unidentifiable, with disrupted senses, as $\varepsilon$
rises. Sec. III-B described how existing methods retain “validity”; e.g.,
DeepHunter allows only one affine transformation per image. Thus, existing
methods are confined within limited perceptions by design. In contrast,
DeepTraversal thoroughly mutates perceptions to generate more diverse media
data, as will be shown in Sec. V.

Objective-guided Generation/Searching vs. Manifold-bounded Mutation. Other
methods have used generative models like VAEs [31] to generate particular data
points that meet predefined objectives [10, 9, 22]. Unlike DeepTraversal,
those works use an objective-directed generation pipeline. That is, they seek
to generate a specific input that fulfills predefined goals and frequently use
the target DNN’s gradients as feedback. To synthesize outputs, the generative
model is iteratively steered online. Additionally, one recent work [30] adopts
VAE to project all images into one latent space, such that by searching on the
latent space, which is more efficient than searching on the pixel space, it
can generate diverse images to test DNNs. We point out that the latent space
does not capture manifolds, given that media data of different classes stay in
distinct manifolds. Merely sampling points from the whole latent space will
produce invalid media data, for instance, an intermediate image between “dog”
and “butterfly”. It’s infeasible to give labels for such images, making this
scheme impractical for DNN testing.

DeepTraversal aims to significantly mutate media data. A generative model is
exclusively adopted offline to prepare a manifold. Then, during the online
phase, DeepTraversal traverses the prepared manifold by starting at seed
$z_{0}$ to produce modified data with gradually changed perceptions.
DeepTraversal follows a standard feedback-driven testing procedure, using DNN
coverage criteria as the feedback. Unlike generation-based methods,
DeepTraversal systematically explores the input space of DNNs instead of
generating edge cases with predefined goals.

##  V Design of DeepTraversal

Fig. 5 illustrates the high-level workflow of DeepTraversal, in terms of
manifold construction (Fig. 5(a)) and feedback-driven testing (Fig. 5(b)). We
now describe each phase in details.

###  V-A Offline Manifold Preparation

Requirement for $\mathcal{S}$. Fig. 5(a) illustrates the offline phase of
DeepTraversal which prepares a manifold over dataset $\mathcal{S}$.
DeepTraversal mutates media data of different types. We use a GAN to create a
specific manifold over $\mathcal{S}$, which can be images, audio, or text. We
assume that $\mathcal{S}$ represents all accessible data for DNNs testing. It
can be model training data or public datasets, e.g., ImageNet [15].

Design Focus. We do not claim novelty in model design. Approximation of data
manifolds with generative models such as GAN has been widely studied in the AI
community [66, 27, 42]. Further, we do not propose novel generative models for
manifold approximation. As described in Sec. VII, DeepTraversal is equipped
with standard generative models based on the input data type. Though standard
models are sufficient, DeepTraversal may be equipped with advanced generative
models [23, 46], if needed.

Data Representation. DeepTraversal mutates media data of two types: continuous
and discrete. Image and audio data are continuous since they are represented
as matrices of floating numbers. Textual data comprise a sequence of words.
They are discrete because there is no “intermediate word.” For continuous
data, we compute the manifold directly (see below). It is, however, impossible
to directly compute the manifold of discrete text by using a GAN. ARAE [65] is
a prominent approach that uses adversarial training to convert discrete text
into a continuous representation, thus enabling the processing of text data by
using a GAN.

Preparing Manifold. To approximate the manifold $\mathcal{M}_{\mathcal{S}}$ of
$\mathcal{S}$, we first launch offline GAN training over $\mathcal{S}$. The
offline training is completed when $G$ can constantly generate realistic data
that is indistinguishable toward $D$. We then keep the well-trained $G$ as
$f_{\theta}^{-1}$, which will be used to map low-dimensional $z$ on
$\mathcal{M}_{\mathcal{S}}$ back to media data $G(z)\in$
$\mathcal{R}_{\mathcal{S}}$. We use different $G$ over manifold constituted
from media data of different types, e.g., image or text. Further, since media
data may have different sizes, we configure different dimensionality for the
latent space $z$ (see details in Sec. VI).

As introduced in Sec. IV, we mirror $G$ to form an encoder $E=G^{-1}$ that
prepares $\mathcal{M}_{\mathcal{S}}$. $E$ is further trained in this offline
phase to minimize the reconstruction error $G(E(s))$ with parameters of $G$
fixed. We then use $E$ to map our seed corpus $C$, denoting a set of media
data, to a set of low-dimensional coordinates $\mathcal{C}$ on
$\mathcal{M}_{\mathcal{S}}$. Our online testing is launched using each
coordinate $c\in\mathcal{C}$. Note that $E$ is only used for initializing
$\mathcal{C}$ in the offline phase, whereas $G$ will be kept to map each
mutated $c^{\prime}$ back to high-dimensional data $G(c^{\prime})$.

Network Structure. Both $G$ and $D$ are composed of stacked blocks [50]. Each
block has a convolutional layer, a batch normalization (BN) layer [29] and a
non-linear function. The BN layer acts as a regularizer by normalizing a batch
of input for subsequent layers as

$x=\frac{x-E[x]}{\sqrt{Var[x]}}\gamma+\beta$

with learned gain $\gamma$ and bias $\beta$. It alleviates the problem of
internal covariate shift, which refers to the varied distribution of input for
each layer during training, thus guaranteeing $G$ and $D$ to be of saturating
nonlinearity [29].

Handling Distinct Manifolds. Data of different classes lie in distinct
manifolds (e.g., dog vs. vehicle images); therefore, it is generally
infeasible to design a universal model to construct distinct manifolds.
Without training a separate model for each manifold (which is computation-
intensive), we use class-conditional batch normalization (BN) to alleviate
this hurdle [13]. The gain and bias in each BN layer depend on the manifold to
be constructed. As illustrated in Fig. 5(a), we only need to switch between BN
layers to transit across distinct manifolds.

###  V-B Online Feedback-Driven Manifold Traversal

Fig. 5(b) illustrates the online phase of DeepTraversal, and the key algorithm
is presented in Fig. 5(c). The offline phase has mapped a seed corpus $C$ to
coordinates $\mathcal{C}$ on the manifold. A priority queue $\mathcal{Q}$ is
initialized using all elements with equal priority in $\mathcal{C}$; each seed
$c$ in $\mathcal{Q}$ may be considered as the starting point of manifold
walking for several iterations, unless its priority is lower than a threshold.
The reason for retaining each seed coordinate $c$ for multiple mutations is
that the mutation of different perceptions on the same data (e.g., images) is
likely to activate different neurons. We now explain the online phase by
traversing within the manifold in Fig. 5(b).

Exploration vs. Exploitation. Our traversal is inspired by “exploitation vs.
exploration” [60], a popular strategy that enables random exploration (to
uncover new knowledge) and exploits knowledge learned from previous
traversals. Let $\mathcal{T}$ be a covariance matrix that records the previous
trajectory taken on a manifold during the current online phase, which will be
explained soon. At time step $t$, we may (with probability $\lambda$) decide
to mutate a seed $c$ with the highest priority in $\mathcal{Q}$ (lines 2–5 in
Fig. 5(c)). This way, $c$ traverses to $c_{t}$, with the trajectory as
$c\mathcal{T}$. We may also randomly traverse to $c_{t}$ sampled on the
manifold with a probability $1-\lambda$ (line 6).

Random exploration is likely to trigger new findings that impede DNN
predictions. Nevertheless, through continuous explorations, $\mathcal{T}$
gradually converges to directions that imply effective mutations (i.e.,
revealing new DNN coverage). Thus, we anticipate that mutating according to
$\mathcal{T}$ may lead to a more oriented exploration of the manifold and
achieve higher coverage within a given time budget. Consider the manifold in
Fig. 5(b). Suppose that $c$ has been mutated twice using $\vec{v}_{1}$ and
$\vec{v}_{2}$, which induces the activation of distinct groups of neurons in
the target DNN, namely $g_{1}$ and $g_{2}$. As DeepTraversal maintains
$\mathcal{T}$, subsequent mutations over $c$ will use
$\vec{v}_{1}+\vec{v}_{2}$, and therefore, mutations directing to
$\vec{v}_{1}+\vec{v}_{2}$ are likely to stress neurons that interleave with
$g_{1}$ and $g_{2}$.

Not all mutations can increase DNN coverage; therefore, mutating $c$ many
times is unlikely to reveal new coverage. Given a priority queue
$\mathcal{Q}$, we lower the priority of $c$ progressively (line 5) and spend
more effort on other seeds in $\mathcal{Q}$.

Starting from 0, $\lambda$ is increased by $\delta$ each step if coverage
increases (line 13), until it reaches the predefined threshold $\Lambda$. This
will gradually reduce the effect of randomness in determining traversal
directions. Moreover, $\lambda$ is not allowed to exceed $\Lambda$
($\Lambda<1$) to retain chances of finding new useful directions on the
manifold.

Trajectory-Aware Mutation. Given each coordinate $c_{i}$ on the manifold as a
vector, we keep track of all covered footsteps by using a covariance matrix,
$\mathcal{T}$. In lines 3–5, DeepTraversal mutates $c$ using
$\mathcal{T}_{t-1}$, which records the footsteps taken on this manifold until
time $t-1$. $\mathcal{T}$, as a matrix, is initialized to 0. If the media data
$G(c_{t})$ derived from step $c_{t}$ increases the DNN testing criteria (line
7), DeepTraversal adds $c_{t}$ to $\mathcal{Q}$ and updates $\mathcal{T}$ as
follows:

$\mathcal{T}_{t}=\frac{(t-1)\mathcal{T}_{t-1}}{t}+\frac{(t-1)(\mu_{t-1}-c_{t})(\mu_{t-1}-c_{t})^{\top}}{t^{2}}$

where $\mu_{t-1}=\frac{1}{t-1}\sum^{t-1}_{i=1}c_{i}$ and will be updated as
$\mu_{t}=(1-1/t)\mu_{t-1}+c_{t}/t$. From a holistic view, $\mathcal{T}$
describes the direction and distance on the manifold that are prone to
increase DNN testing criteria, according to experience collected until time
$t$.

$\mathcal{T}$ is initialized for each online phase. Launching effective
traversal (i.e., better enhancing DNN testing criteria) requires a
$\mathcal{T}$ that is specific for a manifold and the target DNN. For
instance, to test an image classifier $m_{1}$ that relies on the ear shape to
classify dogs, it would be beneficial to mutate perceptions related to the ear
within manifold $\mathcal{M}_{dog}$ over dog images. In contrast, to test an
object detector $m_{2}$, mutating perceptions related to motions (e.g.,
standing vs. lying) within $\mathcal{M}_{dog}$ may better stress $m_{2}$.
Furthermore, Sec. II has introduced the translation invariance property of a
CNN. Thus, $\mathcal{T}$ helps to adaptively recognize certain perceptions
that are prone to enhance DNN coverage and reveal new behaviors, and it saves
efforts mutating certain perceptions that have been shown as less effective
based on current online phase experience.

###  V-C Testing Oracle and Testing Criteria

DeepTraversal is orthogonal to specific testing oracles and criteria. Its
manifold-based data mutation can boost various testing oracles formed by
adtanved program testing paradigms such as metamorphic testing and
differential testing toward DNN models. In evaluation, we show that
DeepTraversal can outperform existing works with respect to common testing
criteria.

DeepTraversal primarily performs intra-manifold traversal for images and
audio. All mutated inputs should share the same label (e.g., a dog is always a
dog). The intra-manifold setting eases the process of forming a testing
oracle: we detect DNN defects as long as inconsistent predictions (different
labels) occur. Nevertheless, for text data, the manifold is generally higher-
level and indicates grammatical coherence. Therefore, all grammatically
coherent English text can be viewed as lying on the same manifold. Similarly,
when testing auto-driving models whose inputs are densely-annotated driving
scenes, traversing a manifold created over normal driving scene images may
yield images with varied annotations, e.g., driving scenes requiring
inconsistent steering angles. In these cases, we use other oracles (e.g.,
formed by differential testing); see Sec. VII-C.

In Sec. VII-D, we also form a differential testing scheme by comparing the
model predictions across a DNN model and its quantized version. We test these
DNN models using the same mutated inputs and compare the consistency of their
predictions.

##  VI Implementation & Evaluation Setup

Our entire codebase has about 12,690 LOC (counted by cloc [12]). DeepTraversal
is written in PyTorch 1.9.0. Both online and offline phases of DeepTraversal
are launched on one Intel Xeon CPU E5-2683 with 256GB RAM and one Nvidia
GeForce RTX 2080 GPU.

Comparison. The target DNNs and the involved datasets are introduced in each
experiment in Sec. VII. DeepTraversal is compared with representative tools
reviewed in Sec. III. To generate diverse and valid images at their best
effort, all of these tools are configured with some key hyper-parameters. We
list their setups as follows:

DeepHunter [61] requires #mutated pixels is less than $\alpha\times$ #total
pixels, or the maximal value of changed pixels is less than $\beta\times 255$.
We follow its default setting where $\alpha=0.02$ and $\beta=0.2$.

TensorFuzz [45] truncates the accumulated noise within $L_{\infty}$. We use
its shipped configuration which sets $L_{\infty}$ to $0.4$.

DeepSmartFuzzer [14] performs mutation on one small region of an image each
time. We follow its default setting where #regions is nine. Since it does not
explicitly preserve validity of mutated images, we use the image validation
scheme of DeepHunter.

DeepTest [55] rules out likely-broken mutated images based on the mean squared
error (MSE). Given an image $i$ and the mutated output $i^{\prime}$, we only
keep $i^{\prime}$ satisfying $MSE(i,i^{\prime})<1000$. Setting MSE as $1000$
denotes a large and tolerable configuration.

DeepRoad [63] is specifically designed for testing autonomous driving. We
reuse its released model to mutate driving scenes.

Coverage Criteria. We benchmark testing tools using the following coverage
criteria.

Neuron Coverage (NC) [48] rescales neuron outputs from the same layer to
$[0,1]$: a neuron is activated if its scaled output is greater than a
threshold $T$. Same with [61, 14], we set $T$ as $0.75$.

K-Multisection Neuron Coverage (KMNC) [39] first decides the normal output
range of each neuron using all training data of the target DNN. It then
divides output range of each individual neuron into $K$-sections. The coverage
is computed as how many (ratio) neuron outputs lie in the total #neuron
$\times K$ sections. Following DeepHunter, we set $K$ as $1000$.

Neuron Boundary Coverage (NBC) [39] calculates coverage by counting #neuron
whose output lies outside the range decided in KMNC.

Strong Neuron Activation Coverage (SNAC) [39] deems a neuron as activate if
its output is greater than the upper bound of the output range decide in KMNC.

Top-K Neuron Coverage (TKNC) [39] measures how many (ratio) neurons in a layer
appear in the top-$K$ outputs for once. We set $K$ to $10$.

Setup. For the current implementation of DeepTraversal, we set $\delta$ and
$\Lambda$ as 0.0005 and 0.8, as two parameters in our traversal (see Fig.
5(c)), respectively. As mentioned in Sec. V-A, media data of different sizes
are mapped to manifold of different dimensions. Also, in Sec. VII-A, we find
implementation bugs in existing tools that will influence their results. While
these do not affect their key findings, we spend time to debug and patch their
tools to compare with DeepTraversal.

##  VII Evaluation

We evaluated DeepTraversal by investigating the following research questions:

RQ1: Is DeepTraversal more effective than existing tools?

RQ2: How does DeepTraversal perform in fault identification?

RQ3: How is the generalizability of DeepTraversal?

RQ4: Can DeepTraversal be extended to other use cases?

RQ5: Is the design consideration of DeepTraversal useful?

###  VII-A Effectiveness

To answer RQ1, we compare DeepTraversal with existing tools using models and
datasets of different scales by assessing the 1) increased neuron coverage and
2) quality of mutated outputs. As in previous studies [61, 14, 45], we focused
on image classification. Target models are listed in Table I. The topological
structures of modern DNNs often combine sequential and non-sequential
connections. We chose VGG and ResNet due to their representative and distinct
network topologies. VGG is built from sequentially stacked modules, whereas
ResNet is not, due to the “short-cut” link [24]. MobileNet is a popular DNN
for mobile devices. Table II lists the data used in this evaluation. ImageNet
is of a considerably larger scale than CIFAR10 in terms of the number of
images and average image sizes. Images from both the datasets are in common
formats, i.e., JPEG for ImageNet and PNG for CIFAR10, since converting an
image from floating-point matrix will induce precision loss, which will be
potentially augmented in later computation, leading to flipped prediction.

The GAN model used at this step is from [8]. All models trained on CIFAR10
reach competitive test accuracy over 93%. Models trained on ImageNet are
shipped by PyTorch. Since we use all training images from both datasets to
form manifolds, to be fair for existing tools, we use all training images as
seeds. We let all tools run six hours for ImageNet-trained DNNs and four hours
for CIFAR10-trained DNNs. We take a batch of images to calculate coverage and
set all tools to perform at most $TRY\\_NUM$ mutations on an image in one
iteration. We set $BATCH\\_SIZE$ to $32$ and $TRY\\_NUM$ to $50$ [61].

TABLE I: Target DNN Models used in Sec. VII-A.

Model | #Neuron | #Layer | Remark
---|---|---|---
ResNet50 [24] | 26570/27560* | 54 | Non-sequential topology
VGG16_BN** [54] | 12426 | 16 | Sequential topology
MobileNet-V2 [28] | 17066 | 53 | Mobile devices

  * •

*ImageNet version. **All three models have batch normalization layers.

TABLE II: Datasets used in Sec. VII-A.

Dataset | #Label | #Image | Size (pixel) | Format
---|---|---|---|---
CIFAR10 | 10 | 50,000 | 32 $\times$ 32 | PNG
ImageNet* | 100 | 100,000 | 128 $\times$ 128 | JPEG

  * *

*To speed up the progress, we randomly select 100 classes from ImageNet and rescale the image to 128 $\times$ 128.

TABLE III: Neuron coverage(%). “Init.” denotes coverage achieved by seeds.
Results are averaged over 5 runs with maximum standard deviation $<10^{-3}$
(i.e., $0.0x\%$).

Criteria | Testing Tool | ImageNet | CIFAR10 | CIFAR10 | CIFAR10
---|---|---|---|---|---
ResNet-50 | ResNet-50 | VGG-16 | MobileNet-V2
NC | Init. | 59.73 | 55.39 | 41.75 | 60.61
DeepTraversal | 66.91 | 64.37 | 50.71 | 64.01
DeepHunter | 65.21 | 61.85 | 48.27 | 62.13
TensorFuzz | 60.41 | 56.55 | 43.03 | 62.33
DeepSmartFuzzer | 63.02 | 60.85 | 47.78 | 62.45
DeepTest | 60.63 | 57.18 | 44.04 | 60.72
KMNC | Init. | 0.59 | 1.43 | 1.10 | 1.40
DeepTraversal | 3.04 | 2.98 | 2.83 | 2.82
DeepHunter | 7.81 | 7.29 | 6.35 | 7.19
TensorFuzz | 3.06 | 2.94 | 2.61 | 2.81
DeepSmartFuzzer | 7.89 | 6.85 | 5.82 | 6.82
DeepTest | 3.05 | 2.98 | 2.83 | 2.80
SNAC | Init. | 0 | 0 | 0 | 0
DeepTraversal | 2.63 | 3.27 | 7.44 | 2.38
DeepHunter | 2.19 | 3.15 | 6.28 | 2.08
TensorFuzz | 0.47 | 3.23 | 5.50 | 2.48
DeepSmartFuzzer | 0.85 | 3.07 | 5.99 | 2.24
DeepTest | 0.92 | 1.54 | 1.78 | 1.93
NBC | Init. | 0 | 0 | 0 | 0
DeepTraversal | 2.77 | 5.99 | 3.83 | 2.84
DeepHunter | 2.85 | 5.28 | 3.25 | 2.52
TensorFuzz | 0.48 | 2.91 | 2.76 | 2.43
DeepSmartFuzzer | 0.90 | 4.82 | 3.34 | 2.39
DeepTest | 1.51 | 3.14 | 1.03 | 1.63
TKNC | Init. | 52.25 | 23.21 | 23.24 | 26.64
DeepTraversal | 57.43 | 26.24 | 32.59 | 28.13
DeepHunter | 55.94 | 25.38 | 29.14 | 27.79
TensorFuzz | 52.77 | 23.51 | 24.15 | 26.91
DeepSmartFuzzer | 54.77 | 24.92 | 28.62 | 27.51
DeepTest | 52.61 | 23.68 | 25.35 | 26.83

Figure 6: Qualitative evaluation of existing tools and DeepTraversal. Figure
7: Coverage increase.

Increased Neuron Coverage. Table III reports the neuron coverage increase.
DeepTraversal outperformed earlier methods in most cases. This result shows
that mutations launched by DeepTraversal are better for exploring DNNs. In
Fig. 7, we show how neuron coverage in ResNet50 increases with NC as the
objective. DeepTraversal consistently achieved higher coverage during the
testing process. DeepTest terminates earlier than the other methods because it
has exhausted all seeds: it discards a seed even if the seed increases
coverage.

For KMNC in Table III, DeepHunter and DeepSmartFuzzer perform better than
DeepTraversal. Recall Fig. 4 illustrates that existing tools explore an
$\epsilon$-radius sphere surrounding a seed. KMNC, in general, divides neuron
output range determined by the initial seeds into $K$-sections. Therefore, by
intensively exploring the sphere near the seeds, DeepHunter and
DeepSmartFuzzer can cover most sections checked by KMNC. However, for criteria
that reflect the “corner-case” behavior of test inputs, such as SNAC and NBC,
DeepHunter and DeepSmartFuzzer have small increase.

In contrast, given that DeepTraversal traverses the entire manifold (not just
a sphere surrounding a seed), DeepTraversal is shown to perform much better
for SNAC and NBC. TKNC has similar coverage trending to that of NC because
both focus on certain neurons with the highest outputs.

Remarks. DeepTraversal exhibits relatively low increase on KMNC, SNAC, and
NBC. Note that the outputs of modern DNNs were close to zero (similar to a
normal distribution) due to input normalization and the presence of BN layers
(which convert neuron outputs to zero-centered small values). Such small and
normalized neuron outputs can hardly reflect changes on these three fine-
grained coverage criteria, particularly for SNAC and NBC which looks for edge-
case outputs.

Also, ImageNet, the model training dataset, is very comprehensive. Therefore,
most neuron outputs are already within the range of training data, which
induces a high “Init.” score, but imposes challenges for testing tools
(including DeepTraversal) to further increase the coverage.

TABLE IV: Quality of mutated ImageNet images.

| DeepTraversal | DeepHunter | TensorFuzz | DeepSmartFuzzer | DeepTest
---|---|---|---|---|---
IS | 33.87$\pm$0.45 | 21.21$\pm$0.48 | 21.24$\pm$0.94 | 23.15$\pm$0.42 | 10.46$\pm$1.36
FID | 54.31 | 70.13 | 75.09 | 67.06 | 119.89

Quality of Mutated Images. Fig. 6 shows a comparison of mutated images. As
DeepHunter limits the number of affine transformations applied on one image,
its modified outputs are similar to the reference. TensorFuzz limits the
accumulated noise on one image within an $L_{\infty}$, thus preserving image
perception. However, TensorFuzz generates outputs that are less diversified
(see discussions about Table V below) because its mutation patterns are
limited. DeepSmartFuzzer deconstructs images as it performs region-level
mutations. DeepTest does not explicitly confine mutation operations. As shown
in Fig. 6, it can apply affine transformations repeatedly on one image, thus
resulting in an aberrant output. However, DeepTraversal generates diverse and
perceptually meaningful outputs.

Furthermore, DNNs are deployed under the i.i.d (independent and identically
distribute) assumption. That is, they require the test data and training data
follow an identical distribution. It’s trivial that DNNs make erroneous
predictions given out-of-distribution input. Thus, assessing the realism (or
naturalness) of mutated images is necessary. This is also reasonable since
DNNs are adopted in real-world scenarios. We are more interested in finding
erroneous behaviors given natural inputs.

For the mutated images, we use inception score (IS) [52] and Fr$\grave{e}$chet
Inception Distance (FID) [26], two popular criteria in the AI community, to
assess naturalness. A higher IS is better, while a lower FID suggests more
naturalness. The results are in Table IV. We interpret these quantitative
results as further demonstration that DeepTraversal can produce images of
higher visual quality than earlier techniques.

Answer to RQ1: DeepTraversal is more effective than existing tools on
throughput, increased coverage and quality of mutated images.

###  VII-B Fault Identification

To answer RQ2, this section compares DeepTraversal with existing tools in
terms of the number of triggered errors, the diversity of exposed erroneous
behaviors, and retraining with the error-triggering inputs.

TABLE V: Prediction inconsistent errors using NC as criterion.

| Tool | ImageNet | CIFAR10 | CIFAR10 | CIFAR10
---|---|---|---|---|---
ResNet-50 | ResNet-50 | VGG-16 | MobileNet-V2
#Error | DeepTraversal | 19679 | 4073 | 3004 | 799
DeepHunter | 7455 | 2011 | 1714 | 631
TensorFuzz | 783 | 40 | 12 | 8
DeepSmartFuzzer | 3963 | 283 | 219 | 78
DeepTest | 614 | 697 | 690 | 162
#Class/ Entropy ($\times$100) | DeepTraversal |  849/81.30 |  10/99.05 |  10/98.98 |  10/98.84
DeepHunter | 691/87.93 | 10/73.91 | 10/69.13 | 10/69.04
TensorFuzz | 297/94.57 | 9/91.85 | 5/84.37 | 5/92.83
DeepSmartFuzzer | 524/89.68 | 10/94.76 | 9/93.65 | 10/91.29
DeepTest | 306/95.79 | 10/68.08 | 9/62.72 | 7/61.25

TABLE VI: Retraining Accuracy (%) of ResNet50 trained on CIFAR10. Results are
averaged over 5 runs with maximum standard deviations $<10^{-4}$ (i.e.,
$0.00x\%$).

Init. | DeepTraversal | DeepHunter
---|---|---
93.77 |  94.07$\uparrow$ 93.96$\uparrow$ | 93.86$\uparrow$
TensorFuzz | DeepSmartFuzzer | DeepTest
93.83$\uparrow$ | 93.73$\downarrow$ | 93.24 $\downarrow$

#Triggered Errors. Our testing oracle asserts whether model outputs are
consistent (Sec. V-C). Table V lists #errors when NC was used as a criterion.
We only count an image as an error-triggering image if it is format valid
(i.e., pixel value $\in[0,255]$). DeepTraversal triggered more errors for all
settings, nearly doubling the amount of #errors found by DeepHunter. More
errors are found for the ImageNet cases. ImageNet is a much larger dataset
with 1,000 classes. Performing image classification on ImageNet is much harder
than classification over CIFAR10. This implies that model quality in our
ImageNet setting is unavoidably lower, more “vulnerable” to testing tools.

Erroneous Behavior. Aside from the number of errors, we are interested in
their diversity. The diversity can be assessed through a two-step approach.
All tested models perform multi-label classification. Therefore, we first
compare the number of classes, denoted as $|C|$, covered by erroneous
predictions. If two testing tools yield equal $|C|$, we further compare the
scaled information entropy of the erroneous predictions. The scaled entropy is
$-\frac{1}{|C|}\sum p_{c}\log p_{c}$, where $p_{c}$ is the proportion of
outputs incorrectly predicted to be class $c$. Scaled entropy is in the range
of $[0,1]$, and a higher entropy suggests more diversity, i.e., less bias for
particular classes. Table V shows that DeepTraversal outperforms others. We
find that the scaled entropy is close to 1, indicating that findings of
DeepTraversal look unbiased and are evenly distributed across all classes for
all three models trained on CIFAR10. We interpret these results as promising,
which show that DeepTraversal can comprehensively explore the probable actions
of DNNs, revealing more diverse flaws.

Retraining with Error-Triggering Images. To “debug” DNNs, the common approach
is to retrain them using correctly-labeled error-triggering inputs [59, 64].
Table VI shows the accuracy of ResNet50 retrained on CIFAR10 that is augmented
with correctly-labeled error-triggering inputs. For each tool, we take all of
their discovered error-triggering inputs to retrain the model. Furthermore,
since DeepTraversal finds much more error-triggering inputs, to make the
comparison fair (DeepHunter has 2,011 error-triggering inputs), we randomly
selected $2,000$ error-triggering inputs of DeepTraversal for retraining
(93.96). As the tested model already has competitive accuracy, retraining
improves only modestly.

DeepTraversal surpasses others in both settings (94.07 and 93.96), indicating
that the flaws detected by DeepTraversal are of high quality and diversity. We
also emphasize that real-world DNNs may have been trained using standard data
augmentation approaches like affine transformations [53]. Therefore, faults
triggered by DeepTraversal are more desirable and orthogonal to those that may
have been included in the augmented training data. It also explains why
TensorFuzz retraining had the similar accuracy as DeepHunter, even though
TensorFuzz triggered fewer faults. Retraining on datasets augmented by
DeepSmartFuzzer and DeepTest resulted in a decrease in accuracy. The region-
wise mutation can break objects in images (see Fig. 6), and DeepTest may apply
several affine modifications to an image, thus making it less recognizable.

Not all faults are valuable; retraining on some broken error-triggering data
may result in performance loss. Nevertheless, DeepTraversal mutates
perceptions of images while retaining the perceptual constraints, i.e., a cat
is still a cat. This evaluation illustrates the merit of manifold-based
mutation and its effectiveness in error detection and model enhancement. It
further amplifies that the perceptual constraints are retrained given that
retraining on the produced data can notably enhance DNNs and help in largely
outperforming others.

Answer to RQ2: DeepTraversal performs the best on fault identification in
terms of the number, diversity and quality of error-triggering inputs.

###  VII-C Generalizability

TABLE VII: Datasets and models for answering RQ3.

Task | Data Type | Dataset | Models
---|---|---|---
Autonomous driving |  |  | $\text{Dave}_{\text{orig}}$
Image | CityScape | $\text{Dave}_{\text{norm}}$
|  | $\text{Dave}_{\text{dropout}}$
Machine translation | Text | SNLI | Transformer
Audio classification | Audio | SC09 | CNN

TABLE VIII: Quantitative results for answering RQ3.

Task | Tool | Increased Coverage(%) | #Fault
---|---|---|---
Autonomous driving | $\text{DeepRoad}_{\text{rain}}$ | (+6.15, +0.58, +0.82) | 2375
$\text{DeepRoad}_{\text{snow}}$ | (+7.36, +16.14, +2.32) | 2213
DeepTraversal | (+18.64, +25.94, +9.95) | 15547
Machine translation | DeepTraversal | +9.96 | 4011
Audio classification | DeepTraversal | +4.04 | 1449

This section benchmarks the generalizability of DeepTraversal. Table VII lists
the models and datasets. Cityscapes has 2,795 driving scenes from the drivers’
view. SC09 features 18,620 sound clips of human voices saying the numbers from
0 to 9. Each sound clip has a duration of about one second and one number is
said. The Stanford Natural Language Inference (SNLI) dataset contains 570,000
human-written English sentences. SNLI is commonly used for natural language
inference and is grammatically coherent. Three variants of the DAVE model [48]
were used to test autonomous driving. Given a driving scene, DAVE predicts a
steering angle. Transformer [57] is a state-of-the-art model for machine
translation (MT). The audio classifier is a CNN that takes the log-Mel
spectrum representation (see below) of audios as inputs. All testing campaigns
run for six hours.

Autonomous Driving. It is infeasible to keep driving attributes such as
steering angles consistent when all the driving scenes are in the same
manifold. As a result, we set up a differential testing scenario to form our
oracle, in which steering angles predicted by three DNNs must match. The
coverage criterion for this setting is the sum of the three target DNNs’ NC.
The GAN architecture from [3] is used. The tested DNNs are provided by
DeepXplore [48] and DeepTest [55] with pre-trained parameters.

We compared DeepTraversal with DeepRoad. The mutated outputs are presented in
Fig. 8. In addition to adjusting the road direction, DeepTraversal may also
change the position/color of the car and background buildings, which
introduces diverse driving scenes. We find that DeepRoad breaks original
images when changing weathers. Therefore, its metamorphic oracle on steering
angle consistency does not always hold. We also assess DeepRoad in the
differential testing setting. In Table VIII, the growth in neuron coverage and
the number of faults are listed. DeepRoad uses additional data from other
domains to deliver “style translation.” For instance, DeepRoad with a “rainy”
scheme requires 1) seed driving scenes and 2) topologically similar scenes
that offer rainy and sunny scenarios. Although DeepRoad is enhanced with
additional knowledge, DeepTraversal outperformed it in both coverage and
#faults.

Figure 8: (Mutated) driving scene images.

Audio. We first use log-Mel spectrum (LMS), a 2D lossless encoding/decoding
method to convert raw audio into 2D images. These 2D images are mutated by
DeepTraversal, and then converted back to stress target DNNs. Our GAN model
was built from stacked CNNs. Audio clips of the same number being spoken lie
in the same manifold. The target DNN was a pre-trained CNN with $93.96\%$
accuracy on the SC09 dataset. NC is used as the coverage criterion. The
increased coverage and #faults are reported in Table VIII. We further retrain
the tested model on SC09 augmented with error-triggering inputs. The test
accuracy improved to $94.52\%$, demonstrating the high quality of the mutated
(error-triggering) audio recordings.

Figure 9: Mutated sentences from the same seed.

Text. DeepTraversal enables the mutation of natural language texts in a
grammatically coherent manner, without prior knowledge or predefined templates
[18, 56, 41]. The procedure is the same as that of image or audio mutations,
except that we first use ARAE to encode discrete text into a continuous
representation (discussed in Sec. V-A). We used NC as the testing criteria.

We test two jointly trained MT models, namely $\vec{E2G}$ and $\vec{G2E}$,
that translate English (German) to German (English). Since MT primarily
focuses on grammatical norms of languages, all English text can be viewed as
being in one manifold. DeepTraversal could build grammatically coherent yet
different English sentences by traversing the constructed manifold.
Empirically, we use the fluency score, $(1+H(W))^{-1}$, to assess coherence of
mutated sentence $W$, where $H(W)$ refers to $-\frac{1}{|W|}\sum_{w_{i}}\log
P(w_{i}|w_{<i})$ and $w_{i}$ is the $i$th word in $W$. Fluency score, higher
the better, ranges from $[0,1]$. The AI community uses fluency score and its
derivatives to assess grammatical coherence [19, 40]. All mutated sentences
achieve a fluency score of $0.32$, suggesting strong coherence, compared to
0.35 for normal phrases in SNLI. We present several sample sentences in Fig.
9, which are mutated from the same seed. They are grammatically diverse and
coherent, despite the fact that DeepTraversal does not specify or require
templates to mutate texts.

It is generally hard to decide if MT makes translation errors. We form the
testing oracle by assessing the consistency of $\vec{E2G}$ and $\vec{G2E}$. We
convert an English sentence $eng$ to a German sentence $ger$ using $\vec{E2G}$
and then back to an English sentence $eng^{\prime}$ using $\vec{G2E}$;
$eng^{\prime}$ should be highly similar with $eng$. An inconsistency indicates
that at least one of the models makes incorrect translation.

We assess the similarity of $eng$ and $eng^{\prime}$ via BLEU (Bi-Lingual
Evaluation Understudy) [47]. The BLEU score is highly correlated with human
assessment and ranges from $[0,1]$ (higher is better). We denote $eng$ as a
fault-triggering input if the BLEU score between $eng$ and $eng^{\prime}$ is
$0$. The sentence “A black dog carrying a metal to a man” is a fault-
triggering sentence produced by DeepTraversal. This sentence is translated as
“Ein schwarzer Hund h$\ddot{\text{a}}$lt einen Mann an” and is back-translated
to “A black dog is holding a man”. Table VIII reports the increased neuron
coverage and #faults. DeepTraversal triggers a great number of faults in six
hours. The results are encouraging, demonstrating DeepTraversal’s adaptability
in managing diverse media data types in a unified and effective manner.

Answer to RQ3: DeepTraversal is generalizable given that, enabled by the
semantic-level mutation, it can boost DNN testing on images, audio and text
data and different working scenarios.

###  VII-D Use Case

We answer RQ4 by extending DeepTraversal to black-box setting where we trigger
inconsistency between a DNN and it’s quantized variants.

TABLE IX: Results for answering RQ4.

Quantized layer | Val acc. (%) | #Iteration | #Fault
---|---|---|---
Conv2D | 58.37 | 33,490 | 139,122
Linear | 58.32 | 34,751 | 146,793
Linear + Conv2D | 58.32 | 38,627 | 165,065

We employ a MobileNetV2 pretrained on ImageNet (officially provided by
Pytorch) and the quantization schemes are listed in Table IX. All three
schemes quantize the layer parameters from 32-bit floating points to 8-bit
integers. We present the validation accuracy (on the randomly select $100$
classes from ImageNet) in Table IX. Since the original model has $58.59\%$
validation accuracy, the accuracy drop induced by quantization is neglectable.
The quantized models are thus deemed as equivalent to the unquantized one. We
use DeepTraversal to trigger inconsistency by semantic-level mutation.

Unlike in previous sections where we employ DeepTraversal in gray-box setting,
quantized models are typically deployed on mobile devices, it’s usually
infeasible to access the intermediate neuron outputs. More specifically, we
only have the final output, i.e., a predicted label and the associate
probability. Simply using “different predictions” as the testing objective is
not suitable since we have no “direction” for accumulating mutations.
Fortunately, the predicted probability is a continuous value in $[0,1]$, we
thus set the testing objective as “maximizing the difference of predicted
probabilities”. We launch the testing campaigns for 6 hours. The #iteartion
and #faults are listed in Table IX. DeepTraversal produces a vast number of
inconsistency-triggering inputs which reveals the security concerns about
model quantization.

Answer to RQ4: DeepTraversal can be extended to other use cases with promising
performance.

###  VII-E Ablation Study

We answer RQ5 by launching ablation studies on the feedback-driven manifold
traversal of DeepTraversal.

TABLE X: Ablation Results.

NC(%) |  ImageNet ResNet50 |  CIFAR10 ResNet50 |  CIFAR10 VGG16 |  CIFAR10 MobileNet-V2
---|---|---|---|---
DeepTraversal | 66.91 | 64.37 | 50.71 | 64.01
DeepTraversal w/o $\mathcal{T}$ | 62.48 | 60.94 | 48.42 | 62.44
DeepHunter | 62.52 | 60.62 | 47.40 | 61.26

Table X reports ablation evaluation to compare DeepTraversal, random traversal
on manifold and DeepHunter (the best in our competitors). We find that 1)
traversal in manifolds, though randomly, is already comparable to DeepHunter,
which indicates that semantic-level mutation is necessary, and 2)
DeepTraversal largely outperforms random traversal, which proves the
superiority of our traversal algorithm, given that $\mathcal{T}$ encodes the
semantic-change that is more sensitive to the tested DNN.

Answer to RQ5: the design consideration of DeepTraversal, especially the
traversal algorithm, benefits its performance.

##  VIII Limitation

DeepTraversal traverses manifold and yield valid mutation outputs. For
instance, a manifold constructed over cat images always emits cat images. As
noted in Sec. V-C, mutations on manifold may break fine-grained annotations.
For instance, to preserve fine-grained annotations (e.g., blue-eyed Collies),
a training dataset with only blue-eyed Collies is necessary to train the
manifold. By including both blue- and brown-eyed Collies in the dataset, even
if there are only a few brown-eyed samples, the manifold will contain both
perceptions. Also, DeepTraversal is not intended to test media libraries
(e.g., libjpeg [36]). In short, media libraries typically process data bytes
of media data rather than perceptions. Media libraries can be tested by
software fuzzing tools, e.g. AFL [62].

##  IX Conclusion

This research provides a multifaceted strategy for mutating media data. We
convert high-dimensional media data into a low-dimensional manifold, and
traverse the full space of manifold to produce mutated data with gradually-
changed perceptions. Our evaluation of mutating media data in various forms
demonstrates our approach’s high effectiveness, exceeding prior works in both
general image classification and domain-specific scenarios.

## References

  * [1] Code of deepxplore.  https://github.com/peikexin9/deepxplore.
  * [2] Hervé Abdi and Lynne J Williams.  Principal component analysis.  Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010.
  * [3] Samaneh Azadi, Michael Tschannen, Eric Tzeng, Sylvain Gelly, Trevor Darrell, and Mario Lucic.  Semantic bottleneck scene generation.  arXiv preprint arXiv:1911.11357, 2019.
  * [4] Mukund Balasubramanian, Eric L Schwartz, Joshua B Tenenbaum, Vin de Silva, and John C Langford.  The ISOMAP algorithm and topological stability.  Science, 295(5552):7–7, 2002.
  * [5] Mikhail Belkin and Partha Niyogi.  Laplacian eigenmaps for dimensionality reduction and data representation.  Neural computation, 15(6):1373–1396, 2003.
  * [6] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [7] Neal E. Boudette.  Tesla’s self-driving system cleared in crash.  https://www.nytimes.com/2017/01/19/business/tesla-model-s-autopilot-fatal-crash.html.
  * [8] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2018.
  * [9] Taejoon Byun and Sanjai Rayadurgam.  Manifold for machine learning assurance.  In 2020 IEEE/ACM 42nd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pages 97–100. IEEE, 2020.
  * [10] Taejoon Byun, Abhishek Vijayakumar, Sanjai Rayadurgam, and Darren Cofer.  Manifold-based test generation for image classifiers.  In 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), pages 15–22. IEEE, 2020.
  * [11] Jose A Costa and Alfred O Hero.  Geodesic entropic graphs for dimension and entropy estimation in manifold learning.  IEEE Transactions on Signal Processing, 52(8):2210–2221, 2004.
  * [12] Al Danial.  CLOC.  https://github.com/AlDanial/cloc, 2017.
  * [13] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville.  Modulating early visual processing by language.  arXiv preprint arXiv:1707.00683, 2017.
  * [14] Samet Demir, Hasan Ferit Eniser, and Alper Sen.  DeepSmartFuzzer: Reward guided test generation for deep learning.  arXiv preprint arXiv:1911.10621, 2019.
  * [15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR09, 2009.
  * [16] Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M. Rao, R. P. Jagadeesh Chandra Bose, Neville Dubash, and Sanjay Podder.  Identifying implementation bugs in machine learning based image classifiers using metamorphic testing.  In ISSTA, 2018.
  * [17] Xing Fan, Wei Jiang, Hao Luo, and Mengjuan Fei.  Spherereid: Deep hypersphere manifold embedding for person re-identification.  Journal of Visual Communication and Image Representation, 60:51–58, 2019.
  * [18] Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou.  Fairness testing: testing software for discrimination.  In ACM ESEC/FSE, pages 498–510. ACM, 2017.
  * [19] Tao Ge, Furu Wei, and Ming Zhou.  Fluency boost learning and inference for neural grammatical error correction.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1055–1065, 2018.
  * [20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [21] David Grossman.  Uber self-driving car kills pedestrian in arizona.  https://www.popularmechanics.com/technology/infrastructure/a19482100/uber-self-driving-car-kills-pedestrian-in-arizona/, 2018\.
  * [22] Ge Han, Zheng Li, Yunqing Wei, Chengyu Hu, Shanqing Guo, Yang Zhang, Xin-Shun Xu, and Tang Peng.  FuzzGAN: A generation-based fuzzing framework for testing deep neural networks.  https://g3h4n.github.io/publication/FuzzGAN, 2019.
  * [23] GM Harshvardhan, Mahendra Kumar Gourisaria, Manjusha Pandey, and Siddharth Swarup Rautaray.  A comprehensive survey and analysis of generative models in machine learning.  Computer Science Review, 38:100285, 2020.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [25] Pinjia He, Clara Meister, and Zhendong Su.  Structure-invariant testing for machine translation.  In ICSE, 2020.
  * [26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [27] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce.  Learning motion manifolds with convolutional autoencoders.  In SIGGRAPH Asia 2015 Technical Briefs, pages 1–4. 2015.
  * [28] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [29] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [30] Sungmin Kang, Robert Feldt, and Shin Yoo.  Sinvad: Search-based image space navigation for dnn image classifier test input generation.  In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pages 521–528, 2020.
  * [31] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [32] Damon Lavrinc.  This is how self-driving cars suck in the rain.  https://jalopnik.com/this-is-how-bad-self-driving-cars-suck-in-the-rain-1666268433, 2018\.
  * [33] Yann LeCun, Fu Jie Huang, and Leon Bottou.  Learning methods for generic object recognition with invariance to pose and lighting.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II–104. IEEE, 2004.
  * [34] John A Lee and Michel Verleysen.  Nonlinear dimensionality reduction.  Springer Science & Business Media, 2007.
  * [35] Zhong Li, Minxue Pan, Tian Zhang, and Xuandong Li.  Testing dnn-based autonomous driving systems under critical environmental conditions.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6471–6482. PMLR, 18–24 Jul 2021.
  * [36] libjpeg.  Main libjpeg-turbo repository, 2020.
  * [37] Tong Lin and Hongbin Zha.  Riemannian manifold learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5):796–809, 2008.
  * [38] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In Advances in neural information processing systems, pages 700–708, 2017.
  * [39] Lei Ma, Felix Juefei-Xu, Jiyuan Sun, Chunyang Chen, Ting Su, Fuyuan Zhang, Minhui Xue, Bo Li, Li Li, Yang Liu, et al.  DeepGauge: Comprehensive and multi-granularity testing criteria for gauging the robustness of deep learning systems.  In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018.
  * [40] Pingchuan Ma and Shuai Wang.  Mt-teql: Evaluating and augmenting consistency of text-to-sql models with metamorphic testing.  arXiv preprint arXiv:2012.11163, 2020.
  * [41] Pingchuan Ma, Shuai Wang, and Jin Liu.  Metamorphic testing and certified mitigation of fairness violations in nlp models.  In IJCAI, pages 458–465, 2020.
  * [42] Francisco J Martinez-Murcia, Andres Ortiz, Juan-Manuel Gorriz, Javier Ramirez, and Diego Castillo-Barnes.  Studying the manifold structure of alzheimer’s disease: A deep learning approach using convolutional autoencoders.  IEEE journal of biomedical and health informatics, 24(1):17–26, 2019.
  * [43] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur.  Recurrent neural network based language model.  In Eleventh annual conference of the international speech communication association, 2010.
  * [44] Shin Nakajima and Tsong Yueh Chen.  Generating biased dataset for metamorphic testing of machine learning programs.  In IFIP-ICTSS, 2019.
  * [45] Augustus Odena and Ian Goodfellow.  Tensorfuzz: Debugging neural networks with coverage-guided fuzzing.  arXiv preprint arXiv:1807.10875, 2018.
  * [46] Zhaoqing Pan, Weijie Yu, Xiaokai Yi, Asifullah Khan, Feng Yuan, and Yuhui Zheng.  Recent progress on generative adversarial networks (gans): A survey.  IEEE Access, 7:36322–36333, 2019.
  * [47] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [48] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana.  DeepXplore: Automated whitebox testing of deep learning systems.  In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, pages 1–18, New York, NY, USA, 2017. ACM.
  * [49] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana.  DeepXplore: Automated whitebox testing of deep learning systems.  In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, pages 1–18, 2017.
  * [50] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [51] Sam T Roweis and Lawrence K Saul.  Nonlinear dimensionality reduction by locally linear embedding.  science, 290(5500):2323–2326, 2000.
  * [52] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29:2234–2242, 2016.
  * [53] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of Big Data, 6(1):1–48, 2019.
  * [54] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [55] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray.  DeepTest: Automated testing of deep-neural-network-driven autonomous cars.  ICSE ’18, 2018.
  * [56] Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay.  Automated directed fairness testing.  ASE, 2018.
  * [57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [58] Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang, and Peixin Zhang.  Adversarial sample detection for deep neural network through model mutation testing.  ICSE, 2019.
  * [59] Shuai Wang and Zhendong Su.  Metamorphic object insertion for testing object detection systems.  In ASE, 2020.
  * [60] Christopher JCH Watkins and Peter Dayan.  Q-learning.  Machine learning, 8(3-4):279–292, 1992.
  * [61] Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Hongxu Chen, Minhui Xue, Bo Li, Yang Liu, Jianjun Zhao, Jianxiong Yin, and Simon See.  Coverage-guided fuzzing for deep neural networks.  arXiv preprint arXiv:1809.01266, 2018.
  * [62] Michał Zalewski.  American Fuzzy Lop (AFL).  https://lcamtuf.coredump.cx/afl/, 2021.
  * [63] Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid.  DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems.  In ASE, 2018.
  * [64] Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen.  Autotrainer: An automatic dnn training problem detection and repair system.  In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 359–371, 2021.
  * [65] Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun.  Adversarially regularized autoencoders.  In International conference on machine learning, pages 5902–5911. PMLR, 2018.
  * [66] Bo Zhu, Jeremiah Z Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen.  Image reconstruction by domain-transform manifold learning.  Nature, 555(7697):487–492, 2018.
  * [67] Jun-Yan Zhu.  CycleGAN Failure Cases.  http://github.com/junyanz/CycleGAN#failure-cases, 2021.
  * [68] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros.  Generative visual manipulation on the natural image manifold.  In European conference on computer vision, pages 597–613. Springer, 2016.
  * [69] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.