Rethinking Diversity in Deep Neural Network Testing
Motivated by the success of traditional software testing, numerous diversity
measures have been proposed for testing deep neural networks (DNNs). In this
study, we propose a shift in perspective, advocating for the consideration of
DNN testing as directed testing problems rather than diversity-based testing
tasks. We note that the objective of testing DNNs is specific and well-defined:
identifying inputs that lead to misclassifications. Consequently, a more
precise testing approach is to prioritize inputs with a higher potential to
induce misclassifications, as opposed to emphasizing inputs that enhance
"diversity."
  We derive six directed metrics for DNN testing. Furthermore, we conduct a
careful analysis of the appropriate scope for each metric, as applying metrics
beyond their intended scope could significantly diminish their effectiveness.
Our evaluation demonstrates that (1) diversity metrics are particularly weak
indicators for identifying buggy inputs resulting from small input
perturbations, and (2) our directed metrics consistently outperform diversity
metrics in revealing erroneous behaviors of DNNs across all scenarios.
Zi Wang, Jihye Choi, Ke Wang, Somesh Jha
# Rethink Diversity in Deep Learning Testing

###### Abstract

Deep neural networks (DNNs) have demonstrated extraordinary capabilities and are an integral part of modern software systems. However, they also suffer from various vulnerabilities such as adversarial attacks and unfairness. Testing deep learning (DL) systems is therefore an important task, to detect and mitigate those vulnerabilities. Motivated by the success of traditional software testing, which often employs diversity heuristics, various diversity measures on DNNs have been proposed to help efficiently expose the buggy behavior of DNNs. In this work, we argue that many DNN testing tasks should be treated as directed testing problems rather than general-purpose testing tasks, because these tasks are specific and well-defined. Hence, the diversity-based approach is less effective.

Following our argument based on the semantics of DNNs and the testing goal, we derive \(6\) metrics that can be used for DNN testing and carefully analyze their application scopes. We empirically show their efficacy in exposing bugs in DNNs compared to recent diversity-based metrics. Moreover, we also notice discrepancies between the practices of the software engineering (SE) community and the DL community. We point out some of these gaps, and hopefully, this can lead to bridging the SE practice and DL findings.

 software testing, deep learning, gradient methods

## I Introduction

DL systems have demonstrated remarkable performance across various machine learning tasks and also revolutionized modern society [1, 2]. More recently, large language models [3] and stable diffusion models [4] have shown impressive, or even superhuman capabilities on many generative tasks. Despite the wide application of DL systems nowadays, their decision process remains quite black-box and less explainable to humans, and they are also shown to be vulnerable to various bugs [5, 6, 7]. With the wide deployment of DL systems, especially in some safety-critical applications, it is important to ensure the security and trustworthiness of those systems. However, DL systems are also software artifacts, thus amenable to SE approaches.

Testing, one of the classical SE techniques, has been practiced for decades to detect erroneous behaviors of traditional software [8]. By serving numerous inputs to the programs, testing can expose bugs when the execution deviates from the specification. Moreover, the inputs can serve as concrete examples for developers to demonstrate the existence of defects and to understand and debug the program.

In traditional software testing, tools often employ the adaptive testing scenario: a repeated loop involving input mutation, program execution, and evaluation of fitness [9, 10]. Inputs with higher fitness scores are prioritized or given more mutation resources for the next loop iterations. The goal is to expose bugs fast and this fitness quantifies how likely an input can expose bugs. As a result, to design an effective testing framework, an important task is to create a good metric that can prioritize potential defective inputs. Motivated by the idea of adaptive testing, to extend its utility toward detecting DNN bugs, a natural question arises as follows:

What is a good metric that can prioritize potential erroneous inputs for DNNs?

There are two types of testing frameworks: _diversity_-based testing and _directed_ testing. Diversity-based testing relies on the assumption that diversified tests can execute the program thoroughly, and often expose corner cases that are overlooked by developers. This methodology is suitable for general-purpose testing, where tools aim to expose as many bugs as possible [8, 9]. On the other hand, the goal of directed testing is more specific; such as detecting a _particular type of bug_ rather than any bug. Driven by the specific goal, directed testing prioritizes inputs that are more potential to expose particular bugs rather than inputs that improve the "diversity" [10, 11]. For example, standard fuzz testing prioritizes tests that trigger unseen coverage. However, if the tool is to identify concurrency errors, test executions that are close to the potential data-race code in the program should be considered more promising to expose concurrency bugs [12].

In recent years, researchers have proposed methods to test DL systems. Many of them are motivated by traditional software testing practices, in particular, to encourage the diversity of inputs. One of the common diversity measures in software testing is code coverage of execution from the inputs [8]. Researchers were motivated by this measure and defined similar metrics for DNNs. They treated neurons as the basic computational components, similar to basic blocks in programs, and defined diversity metrics in terms of neuron activation patterns [13], namely, _Neuron Coverage_ (NC). However, later works argued that NC is _not_ a meaningful metric [14]. Till these days, the usage of neuron activation diversity remains controversial: some recent works re-confirm that NC does not bring significant advantage [15] while some others still design
diversity metrics based on neuron activations [16].

Contrary to existing works that design different diversity metrics for DL testing, we argue that many DL testing tasks in the literature [13, 16] should be treated as directed testing tasks because they are well-defined and specific. We refer to these bugs under detection as _functionality_ bugs because these bugs violate the DNN functionality. Also, the mathematical semantics of DNNs are simple compared to traditional software, so one can quantify how likely one input is to change the DNN's functionality, and this provides a measure for testing. Therefore, test generation should be guided by this directed measure rather than a diversity measure.

Moreover, we suggest that the testing can be even accelerated by exploiting an intrinsic nature of DL systems: differentiability. The differentiability of DNNs allows us to derive fast and precise linear approximations in a small neighborhood using gradients. As we will explain, state-of-the-art \(\ell_{p}\)-adversarial attacks from DL literature, projected gradient descent (PGD) [17], can also be categorized into our testing framework. Moreover, we provide a careful mathematical analysis of gradient-based methods, which validates these methods for \(\ell_{p}\)-transformations, but also _casts doubt_ on their applications to other types of transformations.

In fact, we notice that there are discrepancies between the practices in the SE and DL communities. Even though DL systems are also programs and many SE techniques can be applied, DL has also been studied extensively outside SE due to its capability and popularity. Many practices in both communities are similar at a high level but still different in many ways. We point out some of these gaps and hopefully, this work can help bridge some of the gaps.

To summarize, we make the following contributions:

* We provide a novel conceptual framework for many DL testing tasks, and demonstrate that some state-of-the-art DL practices are the natural consequences of this framework. We believe that our work can help clarify the controversial DNN testing topic, i.e., NC maximization; and bridge the gaps between practices in the SE and DL communities. In fact, we point out that the NC maximization metric generates out-of-distribution (OOD) inputs using the DL glossary.
* To instantiate our framework, we provide several concrete testing metrics with careful mathematical analysis. Our unified testing framework allows us to handle various types of input transformations, and past works were unable to. Moreover, our analysis validates the application of gradients on \(\ell_{p}\)-adversarial attacks but also calls for rigorous scrutiny of gradient methods on other types of input transformations.
* We empirically validate our framework with several carefully designed experiments Moreover, we also compare our framework with several diversity-based methods. The result shows that our metrics are more effective in exposing the functionality bugs among all settings, and also reveals that the diversity metrics are weak indicators for bugs under small \(\ell_{p}\)-perturbations.

## II background

### _Mathematical notations and definitions_

Let \([n]=\{1,\ldots,n\}\). For a vector \(v\in\mathbb{R}^{n}\), \(\text{sign}(v)\in\{-1,0,1\}^{n}\) such that

\[\text{sign}(v)_{i}=\left\{\begin{array}{ll}1,&v_{i}>0\\ 0,&v_{i}=0\\ -1,&v_{i}<0\end{array}\right.\]

\(A^{T}\) denotes the transpose of a matrix \(A\). Let \(||v||_{p}\) denote the \(\ell_{p}\) norm of a vector \(v\):

\[||v||_{p}=\sqrt[p]{\sum_{i=1}^{n}|v_{i}|^{p}}.\]

The canonical Euclidean norm is then \(||v||_{2}\), and another commonly considered norm on the input is the \(\ell_{\infty}\)-norm:

\[||v||_{\infty}=\max_{i\in[n]}|v_{i}|.\]

For two functions \(f\) and \(g\), \(f\circ g(x)=f(g(x))\) denotes the composition of \(f\) and \(g\). If \(f:\mathbb{R}^{n}\rightarrow\mathbb{R}\) is differentiable, we use \(D^{i}f\) to denote the \(i\)-th order differentiation of partial derivatives, e.g., \(D^{1}f\) is the gradient of \(f\), and \(D^{2}f\) is the Hessian of \(f\). As a convention, we omit \(1\) when we denote gradient, i.e., we use \(Df\) to denote the gradient of \(f\).

For an infinitely differentiable multivariate function \(f:\mathbb{R}^{n}\rightarrow\mathbb{R}\), the Taylor expansion \(T\) of \(f\) at \(a\in\mathbb{R}^{n}\) is

\[T(a+\Delta x)=f(a)+(\Delta x)^{T}Df(a)+\frac{1}{2}(\Delta x)^{T}D^{2}f(a)( \Delta x)+\ldots,\]

Therefore, \(T(a+\Delta x)\) is an infinite series. If \(f\) is analytic, then \(T(a+\Delta x)\) converges to \(f(a+\Delta x)\)[18]. In this work, we only work with the first two terms of \(T\), i.e., we only consider

\[f(a)+(\Delta x)^{T}Df(a). \tag{1}\]

Given the information of \(f\) at \(a\), i.e., \(f(a)\) and \(Df(a)\), Equation (1) is the linear approximation of \(f\) at \(a+\Delta x\). This can provide an estimation of \(f(a+\Delta x)\) if we do not want to evaluate \(f\) at \(a+\Delta x\) directly, i.e., \(f(\Delta x+a)\approx f(a)+(\Delta x)^{T}Df(a)\).

_Remark II.1_.: The quality of linear approximation depends on how large \(x\) is, and how close \(f\) is to a linear function.

### _Software testing_

Large software artifacts have been an integral part of modern society for decades and are evolving on a constant basis. Despite its ubiquity, software usually contains various defects and some of them can raise serious security concerns. Testing remains one of the most efficient ways to demonstrate the presence of bugs. Since modern software is complicated, bugs can occur anywhere in the program. For example, a buffer usage in the program could potentially introduce overflow and a pointer operation could lead to a NULL pointer dereference.

Due to the omnipresence of risky operations in the program, automated testing techniques are commonly guided by the diversity heuristics: a diverse set of inputs can execute
the program thoroughly and therefore result in more bugs revealed [8]. One of the diversity measures in testing is code coverage, which can have multiple granularities such as function coverage, basic block coverage, and path coverage. For example, the famous fuzzing tool, American Fuzzy Lop (AFL) is a state-of-the-art general-purpose fuzzer that employs the coverage maximization heuristics [9]. With a loop involving input mutation, execution, and evaluation of coverage-based diversity metric, AFL generates inputs aiming to execute different parts of the program and has successfully detected numerous bugs in various applications.

### _DNNs_

In this work, we mainly focus on feed-forward DNNs, but our methodology is quite general and architecture agnostic. Here we discuss some basics of DNNs that are needed to elaborate our testing approach.

A feed-forward DNN \(f:\mathbb{R}^{m}\rightarrow\mathbb{R}^{l}\) is a composition of affine transformations and non-linear activation functions:

\[f_{1}(x)=W^{(1)}x+b^{(1)};\ f_{i}(x)=W^{(i)}\sigma(x)+b^{(i)},i=2,\ldots,d.\]

where \(W^{(i)}\in\mathbb{R}^{n_{i+1}\times n_{i}}\) is the weight matrix between the layers, \(n_{1}=m\) and \(n_{d+1}=l\), \(d\) is the depth of the network, and \(b^{(i)}\in\mathbb{R}^{n_{i+1}}\) is the bias term. \(\sigma\), the activation, is an element-wise non-linear function. \(f=f_{d}\circ\cdots\circ f_{1}\).

_Remark II.2_.: In general, a neural network is highly non-linear. Therefore, for the linear approximation to be precise, we would need \(\Delta x\) to be small in Equation (1).

In this work, we only consider the standard classification tasks. \(f:\mathbb{R}^{m}\rightarrow\mathbb{R}^{l}\) has \(l\) outputs. Let \(f^{(i)}\) be the \(i\)-th output of \(f\). The classification of an input \(x\) is \(\mathcal{C}(f,x)=\arg\max_{k\in[l]}f^{(k)}(x)\). Suppose that the prediction of \(x\) is \(i\), then \(f^{(i)}(x)>f^{(j)}(x)\) for all \(j\neq i\). The output of \(f\) is called the logit score, and the classification model outputs the class with the highest logit score.

Compared with traditional software, which might contain complicated structures like pointer aliasing, various data structures and indirect function calls, DNNs are numerical programs with simple mathematical semantics. Therefore, the techniques designed to handle the complexity in traditional software might not be directly applicable to DNNs. In particular, we want to point out that diversity-based methods in software testing are not the ideal testing framework for many DNN testing tasks.

### _DNN testing_

The functionality of DNNs is to classify inputs correctly aligned with human perception. Therefore, functionality bugs in the classifier setting should be incorrect classification, and the testing goal is to generate inputs that are misclassified. However, obtaining tests and manually labeling them is usually a very expensive process. Researchers have proposed metamorphic testing [19], which augments existing inputs with metamorphic relations. Metamorphic testing generates new inputs by transforming existing ones while preserving their labels. For image inputs, many transformations do not change human judgment, such as small perturbations, image rotation, zooming, and brightness change. We refer to these transformations as semantics-preserving transformations. They can augment the original dataset by applying those transformations on labeled inputs. If any of those transformed inputs can alter the DNN's prediction, then we identify misclassified inputs. In this work, we focus on identifying misclassified inputs, which we refer to as _functionality testing_. Rather than manually collected and labeled, these inputs are generated from a few compositions of small semantics-preserving transformations. Many of the existing works can be categorized into functionality testing [5, 6, 13, 16], to change the DNN prediction by applying semantics-preserving transformations on the inputs.

### _Diversity in DNN testing_

Inspired by the success of traditional software testing, researchers have devised various diversity measures for DL system testing. One of the first diversity measures is NC [13], similar to the code coverage in traditional software testing, which measures how many neurons are activated above a specified threshold \(k\). The assumption is that the more neurons are activated, the more states of DNNs are explored. Later works refined the definition of NC and proposed several structural variants of NC, including the neuron boundary coverage and strong neuron activations [20]. More recently, [16] proposed to use intermediate layer activation divergence between metamorphic inputs as an indicator to detect prediction violations, arguing that inputs with different intermediate activation patterns should also have divergent predictions. [21] proposed a measure to compute the fault pattern of inputs based on the normalized logit score, and a test selection algorithm that chooses a subset of inputs with diverse fault patterns and higher prediction uncertainty.

### _Directed software testing_

Unlike general-purpose software testing tools such as AFL, directed testing prioritizes inputs that are close to the specific testing goal. For example, a data-race testing tool should prioritize inputs that are more likely to trigger concurrency bugs rather than inputs with unseen coverage [12]. In software testing, this proximity can often be measured statically by distance in code. For example, inputs that trigger concurrency bugs necessarily execute code where data race can occur, so the proximity can be measured by the distance between the execution trace and those race condition targets in the program. Mutating inputs close to those targets can generate inputs even closer to the targets and eventually reach the target and potentially trigger specific bugs [10].

Researchers have developed several directed testing tools, and shown that they are more capable of detecting specific bugs than general-purpose testing tools [11]. In fact, we argue that functionality testing of DNNs should be considered as directed testing tasks conceptually, because bugs within these tasks are well-defined and specific.
## III Main thesis

In this section, we elaborate on the main thesis of this work:

Functionality testing of DNNs is a directed testing task.

This is because the mathematical semantics of DNNs is simple and the testing goal becomes specific under the semantics. This thesis motivates us to devise a few concrete testing metrics. All of them are directness metrics for functionality testing, and the application scope of each metric depends on the underlying input transformation. In fact, we want to highlight that each metric has its own application scope, and misusing them outside their scope can degrade the performance.

### _Forward fitness_

We use \(x\) to denote a human-labeled input, and let \(x^{\prime}\) be a generated input from a few small semantics-preserving transformations, similar to input mutation in traditional software testing. We fix a DNN under test: \(f\). Let \(i\in[l]\) be \(x\)'s label. We assume that \(f\) classifies \(x\) correctly, i.e., \(C(f,x)=i\). The goal of testing is to identify the buggy \(x^{\prime}\) so that the \(f\)'s classification over \(x^{\prime}\) will change, i.e., \(C(f,x^{\prime})\neq C(f,x)\). We provide two different interpretations of bugs, each of which can provide a testing metric.

_Remark III.1_.: Here we assume that \(x\) is classified correctly by \(f\), i.e., \(C(f,x)\) matches the ground truth for \(x\). We make this assumption for the following reasons:

1. Modern network architectures have high capabilities in data fitting and can usually achieve high test accuracy.
2. Since our goal is to alter the prediction of \(f\) on \(x^{\prime}\), if we accomplish it, at least one of \(x\) or \(x^{\prime}\) is a bug. This is similar to the metamorphic testing methodology [16].

**Margin-based score.**\(C(f,x)=i\) implies \(f^{(i)}(x)>f^{(j)}(x),\forall j\in[l],j\neq i\). We define

\[g_{ji}=f^{(j)}-f^{(i)}.\]

where \(g_{ji}(x)<0\), \(\forall j\neq i\), because \(C(f,x)=i\). To change the prediction of \(f\) on \(x^{\prime}\) that is close to \(x\) semantically, we need \(f^{(j)}(x^{\prime})>f^{(i)}(x^{\prime})\) for some \(j\in[l],j\neq i\). Hence, the fitness function for functionality testing could be

\[\max_{j\neq i}(f^{(j)}(x^{\prime})-f^{(i)}(x^{\prime}))=\max_{j\neq i}g_{ji}(x ^{\prime}), \tag{2}\]

i.e., any transformed input \(x^{\prime}\) that has the greatest value of \(\max_{j\neq i}(f^{(j)}(x^{\prime})-f^{(i)}(x^{\prime}))\) is considered the most promising one to expose the bug. Intuitively, the transformed input with a higher margin difference between other classes and the correct label is considered better and has a greater chance to alter the prediction. We refer to Equation (2) as the _margin score_.

**Loss-based score.** When the network is trained, a loss is used to measure the distance between the logit score \(f(x)\) and the ground truth \(i\). The training process minimizes the loss to fit the data to the label, i.e., if this measure is small, the \(x\) is likely to be classified correctly. This naturally provides a measure for the bug: if the loss is big, then \(x\) is likely to be misclassified. In classification tasks, the conventional loss is the cross-entropy function [22, chapter 3]:

\[CE(f(x^{\prime}),i)=-\log\frac{e^{f^{(i)}(x^{\prime})}}{\sum_{j=1}^{n}e^{f^{( j)}(x^{\prime})}}. \tag{3}\]

We refer to this score as the _loss score_.

### _Gradient based fitness_

To collect the logit score of input \(x^{\prime}\), one has to execute the DNN on \(x^{\prime}\). This can be expensive, especially when there are many input transformations and the model is large. However, one distinguishable characteristic of DL systems compared to traditional software is that neural networks are differentiable. Recall that Equation (1) defines a linear approximation of a function via gradients. This allows us to quickly estimate the values of the functions without executing the neural networks.

More formally, suppose we have an input \(x\) and the transformed input \(x^{\prime}\), we can define \(\Delta x=x^{\prime}-x\). From Equation (1), we know that \(f(x^{\prime})\approx f(x)+(\Delta x)^{T}Df(x)\). This is particularly useful when the number of transformations is huge and even infinite in some cases. We compute the gradient with the original input once, then for every transformation, only need to compute the difference in the input domain.

**Approximating margin score.** For the margin score in Equation (2), the quantity we want to estimate is \(g_{ji}(x^{\prime})\). We can compute the the value of \(g_{ji}\) at \(x\), \(g_{ji}(x)\), and the gradient at \(x\), \(Dg_{ji}(x)\), then compute the quantity:

\[\bar{g}_{ji}(x^{\prime})=g_{ji}(x)+(\Delta x)^{T}Dg_{ji}(x).\]

\(\bar{g}_{ji}(x^{\prime})\) is a linear approximation of \(g_{ji}(x^{\prime})\). Instead of identifying the input \(x^{\prime}\) that has the greatest value of \(\max_{j\neq i}(f^{(j)}(x^{\prime})-f^{(i)}(x^{\prime}))\), we choose the input \(x^{\prime}\) that has the greatest value of

\[\max_{j\neq i}(\bar{g}_{ji}(x^{\prime})). \tag{4}\]

We refer to this score as the _backward margin score_, because we need the gradient to compute this approximation, and gradient computation requires backpropagation over the computation graph.

**Approximating loss score.** To approximate Equation (3) at \(x^{\prime}\), we have

\[\overline{CE}(f(x^{\prime}),i)=CE(f(x),i)+(\Delta x)^{T}\cdot D(CE)(f(x),i), \tag{5}\]

where \(D(CE)(f(x),i)\) is the gradient of the cross-entropy function at \(x\). We refer to this score as the _backward loss score_.

**Computational efficiency.** Using the linear approximation surrogate is particularly beneficial when the number of transformations is huge, and the number of model parameters is much larger than the dimensions of the inputs, which is always true for modern network architecture. Suppose there are \(N\) transformations for an input, a forward execution takes \(F\) time,
a backward gradient execution takes \(B\) time, which is usually similar to \(F\), and the inner-product computation takes \(I\) time. To compute the forward fitness score, we need \(N\) forward executions, which take \(NF\) time. To compute the backward fitness score, we need \(1\) forward execution to compute \(f(x)\) and \(1\) backward execution to compute the gradient \(Df(x)\), and then \(N\) inner-product computations to compute \((\Delta x)^{T}D(f)(x)\), which is usually very cheap compared to DNN executions. Brought together, the backward fitness score takes \(F+B+NI\) time to finish.

**Analytical solution.** The backward gradient surrogate can be more valuable when the transformation set is continuous and has simple nice geometry. Because the set is continuous, theoretically, there are infinitely many transformations in the set. Therefore, forward-executing all transformed inputs becomes infeasible. However, if we use the gradient to approximate the function value, we can sometimes derive the optimal transformation among all possible ones to maximize the linear approximation Equation (1) analytically. For instance, if the transformation set is \(\ell_{2}\) or \(\ell_{\infty}\)-transformations, the geometry of the sets is either Euclidean balls or hypercubes. To maximize the gradient inner product, one only needs to project the gradient to the \(\ell_{p}\)-balls. Let \(h\) be the gradient. If the perturbation set is an \(\ell_{2}\)-ball with radius \(\epsilon\), the projection of \(h\) inside the ball is:

\[\frac{h}{||h||_{2}}*\epsilon; \tag{6}\]

if the perturbation set is an \(\ell_{\infty}\)-ball with radius \(\epsilon\), the projection of \(h\) is:

\[\epsilon*\text{sign}(h). \tag{7}\]

This is the core idea of PGD, the state-of-the-art \(\ell_{p}\)-attacks, where the gradient comes from the loss-based score, similar to Equation (5). Thus, when using gradient methods and the underlying transformation includes small \(\ell_{p}\)-perturbations, we also use the gradient to generate the analytical solutions as in Equations (6) and (7) to test DNN.

_Remark III.2_.: In practice, PGD is a parameterizable multi-step adaptive projection of the gradients. On \(\ell_{\infty}\)-transformations, the one-step gradient projection coincides with the Fast Gradient Sign Method (FGSM) [6].

### _Mixed score_

Recall that in Remark II.2, if \(\Delta x\) is big, the approximation quality from gradients is poor. Therefore, we can also define a \(\Delta x\)-dependent metric such that: when \(\Delta x\) is big, we use the forward score Equation (2); and when \(\Delta x\) is small, we consider the linear approximation. Thus we can define

\[MM(x^{\prime})=\left\{\begin{array}{ll}\max_{j\neq i}(g_{ji}(x^{\prime})), &||\Delta x||_{p}>\epsilon\\ \max_{j\neq i}(\bar{g}_{ji}(x^{\prime})),&||\Delta x||_{p}\leq\epsilon\end{array}\right. \tag{8}\]

We refer to Equation (8) as the _mixed margin score_. Similarly, for loss score, we have:

\[ML(x^{\prime})=\left\{\begin{array}{ll}\underline{CE}(f(x^{\prime}),i),&|| \Delta x||_{p}>\epsilon\\ \overline{CE}(f(x^{\prime}),i),&||\Delta x||_{p}\leq\epsilon\end{array}\right. \tag{9}\]

We refer to Equation (9) as the _mixed loss score_. Notice that the condition decision in Equations (8) and (9) is a cheap operation. We only need to compute \(\Delta x=x^{\prime}-x\) and find the \(\ell_{p}\)-norm of all \(\Delta x\). If we only consider small \(\ell_{p}\) transformations, mixed scores correspond to their gradient-score counterparts; and if all the transformation is large, mixed scores coincide with their forward-fitness counterparts.

_Remark III.3_.: The gradient-based scores are useful when the additive difference between the transformed input and the original input is small, and especially when the set of transformations forms a nice geometrical space. However, if the difference is large, then gradients provide imprecise estimations. In this case, following our directness argument, a better score is from the logits, especially when the number of inputs is tractable. However, gradient-based scores can still provide fast estimations, albeit the precision is questionable.

## IV evaluation

We empirically evaluate our claims made in Section III: 1) the directness metrics are effective in exposing bugs in functionality testing; 2) each metric is effective within their application scope and can have performance degradation outside the scope. Because the application scope is defined by whether the underlying transformation allows all small \(\ell_{p}\)-perturbations, we will evaluate the performance with different types of transformations separately. In particular, several recent SE testing works only studied non-\(\ell_{p}\) input transformation scenario [16, 21], and we use them as benchmarks. Our evaluation can also provide more understanding of their performance and application scope. More specifically, we want to answer the following research questions:

**RQ1:**: How similar are the metrics to rank the inputs?
**RQ2:**: Are the fitness scores induced from our framework better than the diversity metric to expose bugs in DNNs when not all small \(\ell_{p}\)-transformations are allowed?
**RQ3:**: Are the fitness scores induced from our framework better than the diversity metric to expose bugs in DNNs when all \(\ell_{p}\)-transformations are allowed?

Notice that RQ1 only asks for the correlations between different metric functions. This does not reflect what metrics are better for detecting the DNN bugs. However, for most testing frameworks, the ordinal relations of inputs are more important than the absolute values of scores. The correlation measurement helps us understand how similar the metrics are in different input transformation regimes.

### _Input transformations_

We consider two sets of transformations on the input.

1. The \(\ell_{p}\) transformations, also known as the adversarial attacks: the transformations are from \(\ell_{\infty}\)-balls or \(\ell_{2}\)-balls
with small specified radii \(\epsilon_{\infty}\) and \(\epsilon_{2}\). The radii of transformations are specified according to recent adversarial robustness works [23, 17, 24].
2. Common image transformations such as zooming, and blurring, which we refer to as _benign transformations_ as in [21].

Notice that the two transformations are not exclusive to each other: some benign transformations might also be \(\ell_{p}\)-transformations. The difference is that for \(\ell_{p}\)-transformations, we would allow all possible small additive \(\ell_{p}\)-perturbations. Notice that because the \(\ell_{p}\)-balls are continuous spaces, there are infinitely many transformations in theory. In practice, even if we discretize the space, the number of transformations is exponentially many with respect to the input dimension. As a result, if all \(\ell_{p}\)-transformations are allowed, for non-gradient methods, we can only sample the transformations within the \(\ell_{p}\)-balls; for gradient methods, we can project the gradient to the \(\ell_{p}\)-balls directly as described in Equations (6) and (7).

We also report the average \(\ell_{2}\)-norm of \(\Delta x\) for all the transformations on the test dataset. We choose \(\ell_{2}\)-norm instead of \(\ell_{\infty}\)-norm because the input space and inner product used in linear approximation Equation (1) are naturally equipped with the \(\ell_{2}\)-norm. Also, the \(\ell_{2}\)-norm result can reflect the difference in \(\ell_{\infty}\)-norm.

\(\ell_{p}\)**-transformations specifications.** We use radius \(0.08\) for the \(\ell_{2}\) transformations, radius \(0.01\) for the \(\ell_{\infty}\). Because we will test DNNs adaptively for \(5\) iterations, the choice of radius can ensure that the commonly practiced \(\ell_{p}\)-perturbations in adversarial robustness literature are reachable within \(5\) iterations.

**Benign-transformations specifications.** Our benign transformations include \(7\) types of transformations: shift, zoom, brightness, rotation, shearing, blur and contrast ratio as in [21]. The parameters are selected such that all the transformations in the evaluation of [21] are reachable within \(5\) iterations.

### _Experimental design_

**RQ1.** We fix randomly selected transformations for each input in the test set, and then rank the transformations based on various fitness measures. We consider three types of transformations separately: (i) one with only \(\ell_{p}\)-perturbations; (ii) one with only benign transformations; (iii) one with both adversarial and benign transformations.

For \(\ell_{p}\)-transformations, we randomly select \(14\)\(\ell_{\infty}\)-perturbations and \(14\)\(\ell_{2}\)-perturbations. For benign transformations, we generate \(4\) transformations for each of the seven benign transformations as in [21]. For mixed transformations, we combine all \(28\)\(\ell_{p}\)-perturbations and all \(28\) benign transformations together as the transformation candidates.

For every set of transformations, each fitness metric produces a ranked list of transformations, where the top ones are assumed to be more likely to expose bugs in the DNN. We then compute the rank similarities between the lists. For example, given a fixed set of inputs \((x_{1},x_{2},\ldots,x_{n})\), each metric prioritizes them to generate a ranked list. We compare the two ranked lists to study the correlation between pairs of metrics. We use a well-established metric, Rank Biased Overlap (RBO) [25], to measure this similarity between a pair of ranked lists. The RBO score between two lists is the weighted average of overlaps between all top sublists from both lists, in which the weight takes the rank into account. The higher the score is, the more similar the two ranked lists are. One can then use RBO to measure how similar the two ranked lists are to study the correlation of the fitness metrics.

**RQ2.** We apply the adaptive testing scenario to evaluate the metrics, i.e., a repeated loop involving mutation, feedback, and evaluation of fitness [10]. For each iteration, we consider all transformations over the inputs and then prioritize the transformed inputs according to various fitness scores. We retain the top input as the seed input for the next transformation round. We will record the accuracy induced from the retained input in each iteration, up to \(5\) iterations. This multi-iteration adaptive test is also similar to the PGD attack for \(\ell_{p}\)-perturbations [17].

In this experiment, we only allow benign transformations. This is similar to the evaluation in recent testing works [21, 21]. In particular, this can help us understand whether our directness metrics are effective in bug detection compared to existing works within their evaluation scope under the standard adaptive testing scenario.

**RQ3.** We will conduct the same experiments as RQ2's experiments, except that the underlying transformations are either \(\ell_{p}\) transformations or including both \(\ell_{p}\) and benign transformations. The rationale is that human judgment is invariant to both small perturbations and usual image transformations, so we should consider all of them, especially if an adversary can utilize all semantics-preserving transformations. Because the goal of the experiment is to evaluate the bug-finding capability of different fitness, for non-gradient methods over \(\ell_{p}\)-perturbations, their capability of bug-detection heavily relies on the number of sampled transformations. We, therefore, sample \(140\)\(\ell_{p}\)-transformations instead of \(28\) in the evaluation of RQ1.

### _Experimental specifications_

**Datasets.** We use three standard datasets: (1) SVHN [26] contains \(10\) classes representing digits, \(73257\) inputs for training, \(26032\) digits for testing. (2) CIFAR10 [27] contains \(60,000\)\(32\times 32\) colour images in \(10\) classes. There are \(50000\) training images and \(10000\) test images. (3) CIFAR100 [27] is similar to CIFAR10, but has \(100\) classes containing \(600\) images each. There are \(500\) training images and \(100\) testing images per class.

**Model architectures.** We use two classical architectures from the computer vision community for our experiments: VGG-16 [28], and ResNet-9 and ResNet-18 [29]. More experimental specifications can be found in the supplementary materials.

**Benchmarks.** We use three recent diversity-based testing works as benchmarks:

1. Boosting Diversity: [16] is our major benchmark. It is built upon the assumption that if the divergence between two executions is significant, then the inputs might be
classified differently. More specifically, for the pair of executions, one extracts neuron activations from any hidden layer in the DNN, and then compute the distributional difference of them between the original input and the transformed input. Transformed inputs with a great difference should be prioritized. As noted in [16], as one uses features extracted at the deeper layer of DNNs, the prioritization accuracy increases, but the execution time increases. To find the balance between these two, the standard practice is to choose the layer in the middle of the target DNN. We follow this guideline but also include a variant as a stronger benchmark; we use the logit layer as the representation layer, which, as the authors indicated, provided better prioritization estimation than the mid-intermediate layer.
2. NC: [20] monitors and gauges the intermediate neuron activities covered by current executions at various granularity levels. They profile the range of neuron outputs into \(k\)-multisections of the major function region and corner-cases regions during the training, then identify the ratio of sections covered under test-time executions. In [16], where NC is considered as one baseline of diversity measures, they adopt the original metric in [20] to compute the ratio of uncovered neurons given two executions; that is, the larger the value, the more diverse the two executions in terms of the NC patterns. We follow this modified NC measure as one of our baselines.
3. ATS: [21] is a recently proposed metric for input selection, i.e., choosing a small yet diverse subset of inputs from a large set. It computes a diversity metric from the normalized logit layer, similar to ours. It is not designed for our testing scenario, i.e., for every single input, choosing the best candidate among all transformed inputs. However, we adapt it to our scenario by feeding an input and its transformed mutants to the selection algorithm one at a time and then rank the inputs based on their metric. We include this variant as another benchmark.

We will use the following abbreviations for the methods considered in the evaluation: **FM** stands for forward margin score (Equation (2)); **FL** stands for forward loss score (Equation (3)); **BM** stands for backward margin score (Equation (4)); **BL** stands for backward margin score (Equation (5)); **MM** stands for mixed margin score (Equation (8)); **ML** stands for mixed loss score (Equation (9)); **BD** stands for the standard tool from [16], **BD-F** stands for a variant that uses the logit layer to compute the divergence. **NC** stands for NC method using the layer in the middle of DNN architecture, **ATS** stands for the adapted tool from [21].

To summarize, our metrics (FM, FL, BM, BL, MM, ML), ATS, and BD-F only utilize the logit layer, while BD and NC use the intermediate layer neuron activations. We choose the same intermediate layer for both BD and NC. We do not include evaluations of ATS on the CIFAR100 dataset since the fault pattern computation is cubic to the number of classes, which is too slow with 100 classes.

## V Results and discussion

In this section, we present selected experimental results due to space limits and discussions on the results. More results can be found in supplementary materials.

### _Rq1_

**RBO Benchmark.** The range of the RBO score is within \([0,1]\), and a high score indicates that two ranked lists are positively correlated and similar. We benchmark \(3\) RBO scores: (i) two same-ranked lists; (ii) two randomly ranked lists; (iii) two opposite-ranked lists. All lists are of length \(28\). With these benchmarks, we can interpret the results more precisely. The RBO benchmarks are:

1. Two same rank lists: \(1\);
2. Two randomly ranked lists: \(0.52\);
3. Two oppositely ranked lists: \(0.33\).

As a result, if the RBO score is less than \(0.52\), the two ranks are negatively correlated.

The RBO scores between different ranking methods are summarized in Table I.

**Observations.** 1. The forward loss and margin scores are always very positively correlated;

2. Forward scores and backward scores are strongly positively correlated when the underlying transformation is only \(\ell_{p}\)-transformations; but for benign transformations, the norms induced from \(\Delta x\) are fairly large, and the correlations between forward and backward scores are much weaker.

3. Most of the scores are above \(0.52\), except for BD(-F) & FL and NC & FL in some \(\ell_{p}\)-transformation-only regime.

**Implications.** 1. Both margin and loss scores can quantify how a DNN makes predictions, so their rankings of inputs to be classified incorrectly are very similar.

2. Gradient-based scores are excellent surrogates for forward scores only when the underlying transformations are small perturbations.

3. All the scores can be more efficient to expose functionality bugs than random prioritization. The only exception is that using BD with only \(\ell_{p}\) transformations. When applying small perturbations on the input, the difference measurement in the intermediate layer activation is not aligned with their loss change. This raises concerns about how accurate it is to use intermediate-layer divergence as a surrogate to measure the potential prediction change when the transformations are only small \(\ell_{p}\)-changes, which was not studied in [16].

### _Rq2_

Selected results are summarized in Table II.

**Observations.** 1. Our forward scores are the strongest testing metrics to expose functionality bugs in all experiments.

2. Recently proposed diversity methods, BD(-F) and ATS are also competitive metrics to test DNNs. Moreover, as suggested by [16], using the divergence in the final layers is a stronger metric than the intermediate layer. In fact, BD-F is the best metric without our forward scores.
3. The gradient-based scores are weaker than recent diversity-based methods, though they also consistently decrease the accuracy.

_Remark V.1_.: [16] is designed for metamorphic testing, where the ground truth labels are assumed unknown. In our metrics, we assumed the access to ground-truth labels as in the adversarial robustness literature [6, 17]. In the supplementary materials, we also include a study of the metamorphic variant of our metrics, and the result is consistent with the ground-truth label case. This indicates that the metamorphic testing considered in [16] can also be more effectively addressed with our directed testing approach.

**Implications.** 1. Because ATS and BD-F both use the logit layers as our scores, comparisons with them establish the main thesis that the functionality testing of DNNs is directed, and the bug potential can be measured by our scores.

2. Our theoretical analysis demonstrates that gradients are not a good estimation when the transformation induces a large difference to the inputs. This is consistent with our empirical observation, that gradient-based metrics are quite weak for benign transformations. As a result, when applying gradients to test networks, one should quantify the change introduced by the transformation. Indeed, gradient methods were applied to test neural networks beyond the \(\ell_{p}\)-transformations, and many of them are in the natural and programming languages domains [30, 31]. Researchers consider semantics preserving transformations to natural language or programs, and devised attacks and defenses using gradients against those vulnerabilities. These transformations are similar to benign transformations in our evaluation and there were no justifications that the changes induced by the transformations to the network input are small. The observation from our work raises concern about the effectiveness of these attacks and defenses. As a result,
We call for rigorous scrutiny of gradient-based tests on non-\(\ell_{p}\) types of input transformations, especially discrete tractable transformations.

### _Rq3_

Selected results are summarized in Table III.

**Observations.** 1. When only \(\ell_{p}\)-transformations are presented, the best metrics are gradient-based methods. In cases where both \(\ell_{p}\)-transformation and benign transformations are allowed, our mixed scores are the most effective metrics.

2. The BD(-F) scores are quite weak when only \(\ell_{p}\)-perturbations are allowed, even among all the non-gradient metrics.

**Implications.** 1. From RQ1's result, we can see that gradient surrogates are very positively correlated. Moreover, the linear approximation allows directly deriving the optimal transformation for \(\ell_{p}\)-perturbations. This observation is consistent with our theoretical analysis and validates the application of gradient methods for small \(\ell_{p}\)-transformations.

2. From RQ1's result, when only \(\ell_{p}\)-transformations are allowed, the correlation between our loss score FL and BD(-F) scores is in fact a little negative. In the experiment, BD(-F) induce a weak result for testing, even compared with our non-gradient forward loss score. These results show that the neuron divergence can be subtle when the input transformations are small \(\ell_{p}\)-perturbations. These scenarios were not evaluated in [16]. Our evaluation brings new knowledge on the application scope of [16].

3. In the mixed transformation setting, our mixed scores induce the strongest metric. Our results on various types of transformations indeed show that our metrics are superior to existing diversity-based metrics in detecting bugs within their application scopes.

## VI Related Work

In this section, we further discuss some related topics on DNN testing. In particular, we want to emphasize some discrepancies between SE practices and DL findings. This brings awareness of the gaps and hopefully can lead to bridging them.

**Neuron activation.** One of the first DNN testing works was inspired by the code coverage in traditional software testing practice to define NC [13]. However, the inherent difference between software and DNN challenges the direct application of the idea from one literature to the other. In software, execution trace usually has logical and qualitative semantics, i.e., the execution trace is decided by the logical conditions in the program. In contrast, the neuron activation in DNNs is quantitative, and its semantics is not as clear as in software. Therefore, maximizing NC does not provide clear interpretations as in traditional software.

Nonetheless, NC does provide rich information about the execution of the inputs. In DL society, neuron activation is referred to as _feature representation_, and the researchers have exploited the information from feature representations for various purposes. For instance, higher-level explanations about DNN behaviors can be abstracted out of feature representations, which are more user-friendly than simply inspecting neuron outputs [32, 33, 34]. Moreover, [35] illustrated that feature representations can provide an effective metric for human perceptual measurement. They defined a metric based on normalized neuron activation and found that it is strongly aligned with human judgment. This leads to the design of defense against more realistic and unseen threats [36].

[16] relies on the assumption that if the neuron representation of two metamorphic inputs on an intermediate layer is very divergent, then their predictions are also likely different. While the reversed argument always holds that different predictions must have divergent intermediate representations, the original assumption is not necessarily true. Especially how to measure the difference is unclear when the divergence is subtle, and this can lead to distinctive prioritization of inputs as in the \(\ell_{p}\)-transformation evaluations. However, similar ideas in DL communities are used to design OOD detectors. OOD tests are inputs that are far from the distribution the DNN has seen during the training (i.e., in-distribution), hence it is undefined behavior for DNN to predict given those tests. Assuming that in-distribution and OOD inputs have distinctive feature attribution patterns, there have been various approaches to exploiting statistical information from feature representations to distinguish in-distribution vs. OOD inputs [37, 38], and understand the characteristics of OOD inputs [34]. Consequently, bringing the OOD detection perspective to DNN testing, we suggest that the NC maximization would make the feature representations of generated inputs deviate from the training data, which are actually OOD. This is consistent with the finding in [14] that the inputs from maximizing the NC score are not natural. To this end, one needs to define the scope of bugs a priori to avoid unnecessary execution against OOD inputs.

**Gradient.** We carefully analyzed the application of gradients in DL testing. Gradients naturally exist for many mathematical functions, including neural networks. They quantify the local variability of a function and can provide the optimal linear approximation in a small neighborhood. Our gradient-based metrics are the applications of linear approximation and some of them coincide with the adversarial attack algorithms in DL literature, which indicates that these algorithms can be viewed as concrete instantiations of our framework for small additive transformations. Because traditional software is mostly not differentiable, gradients are less often used in software testing works. However, the gradient-based methods have been successful in many DL applications, encompassing \(\ell_{p}\)-adversarial attacks and gradient-based explanations [39, 40]. We believe that the SE practice can also utilize gradients as tools for testing DNNs, such as recent works on numerical defects in DL programs [41, 42].

**Bugs.** In this work, we restrict our scope to functionality bugs, i.e., inputs that are misclassified according to human perception. However, DL bugs can manifest in various forms. In [7],
the authors categorize model bugs into three types: model contamination, data contamination, and test-time contamination. Model contamination is caused by errors in the model parameters, for example, incorrectly initialized parameters; data contamination comes from defects in the training data; and test-time contamination is caused by shifts in test data. Functionality bugs can be viewed as test-time contamination: our data are transformed at test time. However, there are many more types of bugs in the DL system to be addressed and SE techniques can be applied. For example, model contamination is often caused by API misuses, which has been studied in many SE works [43, 44].

Diversity.We claim that merely increasing diversity is not the ideal framework for DNN functionality testing. However, we do not claim that diversity is useless for DL testing. It depends on our testing goal. A natural diversity issue of DL systems is fairness and bias [45, 46]: Between subgroups, DL systems contain performance discrepancies and make biased decisions towards a certain subgroup. This would also cause concerns for the deployment of DL systems because DNN classifications can discriminate certain subgroups. If the tests were only from a subgroup, then potential fairness and bias issues would be concealed. Therefore, designing meaningful diversity measures can help expose and alleviate the bias and fairness issues in DL systems.

With these DL lessons, a natural question arises:

What should be treated as bugs for DL systems?

We leave this as an open problem. We believe that if bugs are defined properly, then directed testing is more effective to detect the bugs.

## VII Threats to validity

Internal validity.Our subject of study is the DNN, whose training dynamics have many nuances. The selection of datasets, architectures, optimizers, and hyperparameters might not be optimal for all situations. However, we conduct the experiments in accordance with the best practice to our knowledge. Moreover, the numbers reported in this work are averaged over tens of thousands of inputs, which can help reduce randomness. Our benchmarks come from the most up-to-date works, and the scale of our experiments is also larger than theirs [16, 21]. For example, for each input, we consider up to over a hundred transformations, but the transformations considered in [16, 21] are significantly smaller.

External validity.We restrict the scope of our work to testing functionality bugs and claim that this task is directed. This does not imply that all DL testing tasks are directed and our work can be used to detect those bugs. One needs to first define what are bugs and then consider our work if these bugs fall in our scope.
## VIII Conclusion

In this paper, we argue that functionality testing should be addressed using the directed testing framework because they are well-defined and specific. Moreover, we propose several concrete metrics to quantify the potential faultiness of inputs and carefully characterize their scopes. We find that metrics perform drastically differently across input transformations and call for careful scrutiny of these methods.
