#  DeepKnowledge: Generalisation-Driven Deep Learning Testing

Sondess Missaoui<EMAIL_ADDRESS>Department of Computer Science,
University of York, UK ,  Simos Gerasimou<EMAIL_ADDRESS>Department of Computer Science, University of York, UK and  Nikolas Matragkas
<EMAIL_ADDRESS>CEA List, Université Paris-Saclay, France

###### Abstract.

Despite their unprecedented success, DNNs are notoriously fragile to small
shifts in data distribution, demanding effective testing techniques that can
assess their dependability. Despite recent advances in DNN testing, there is a
lack of systematic testing approaches that assess the DNN’s capability to
generalise and operate comparably beyond data in their training distribution.
We address this gap with DeepKnowledge, a systematic testing methodology for
DNN-based systems founded on the theory of knowledge generalisation, which
aims to enhance DNN robustness and reduce the residual risk of ‘black box’
models. Conforming to this theory, DeepKnowledge posits that core
computational DNN units, termed Transfer Knowledge neurons, can generalise
under domain shift. DeepKnowledge provides an objective confidence measurement
on testing activities of DNN given data distribution shifts and uses this
information to instrument a generalisation-informed test adequacy criterion to
check the transfer knowledge capacity of a test set. Our empirical evaluation
of several DNNs, across multiple datasets and state-of-the-art adversarial
generation techniques demonstrates the usefulness and effectiveness of
DeepKnowledge and its ability to support the engineering of more dependable
DNNs. We report improvements of up to 10 percentage points over state-of-the-
art coverage criteria for detecting adversarial attacks on several benchmarks,
including MNIST, SVHN, and CIFAR.

##  1\. Introduction

Deep Neural Networks (DNNs) exhibited enormous progress in achieving and in
some instances surpassing, human-level performance in many challenging tasks
(Sutskever et al., 2014; Hinton et al., 2012; Goodfellow and Papernot, 2017).
This success led to extensive DNN adoption in a wide spectrum of safety- and
security-critical applications ranging from drug discovery in healthcare
(Litjens et al., 2017) to flight control systems (Julian et al., 2016) and
autonomous driving (Bojarski et al., 2016).

Notwithstanding the remarkable advances, DNN models still present considerable
inconsistency in their performance (Zhang et al., 2019b), proven to be
unstable with respect to small perturbations in the input data. Several safety
incidents (e.g., Tesla’s Autopilot crash (Banks et al., 2018)), resulted in
high uncertainty and general mistrust in employing DNNs to undertake sensitive
tasks in applications where high dependability is non-negotiable. Industrial
studies have shown that DNN performance degrades considerably when data from
the operational environment shifts away from the distribution assumed during
training (Zhou and Sun, 2019), raising major concerns about the robustness of
the deployed model when facing unexpected data domain shift and/or adversarial
perturbations (Xu et al., 2020). Due to their black-box nature, testing DNNs
and detecting incorrect behaviours via conventional testing approaches is
inadequate to ensure high DNN trustworthiness (Burton et al., 2017).

Stemming from traditional software testing, systematic DNN testing has
emerged, aiming at understanding the capacity of DNNs to operate dependably
during deployment when encountering corner input data not seen during the
training phase (Riccio et al., 2020; Handelman et al., 2019). White-box DNN
testing informed by adequacy criteria has become a reference methodology
because of its significant advances over ad-hoc testing and its effectiveness
in detecting the diversity of test sets (Pei et al., 2017). Illustrative
examples from this line of research include testing techniques performing
coarse- and fine-grained analysis at neuron level (Pei et al., 2017; Ma et
al., 2018a), measuring the relative novelty of the input (Sun et al., 2018) or
leveraging explainability concepts (Gerasimou et al., 2020) (see Section 6).

Despite the innovations, existing data-centric approaches do not necessarily
help explain how the input features, computational units (neurons), and
optimization algorithms of a DNN jointly contribute to achieving its strong
generalisation power. Generalisation is crucial in a DNN, denoting its
capability to correctly recognize unseen input data (Wang et al., 2018), with
generalization bound signifying the DNN’s predictive performance (Reid, 2010).
From a safety assurance perspective, assessing the capability of DNNs to
respond appropriately when unexpected inputs and/ or corner cases appear in
real-world scenarios entails establishing a functional understanding of the
DNN components that learn to generalise (Burton et al., 2017). These DNN
‘learners’ are core computational units that can generalise knowledge
abstracted during training and apply it to a new domain without re-training.
These learners should be tested extensively to establish the capacity of the
DNN to operate dependably upon deployment.

Driven by this insight, we introduce DeepKnowledge, a systematic testing
methodology accompanied by a knowledge-driven test adequacy criterion for DNN
systems based on the out-of-distribution generalisation principle (Reid,
2010). The hypothesis underpinning our approach is that by studying the
generalisation capability of a DNN, we deepen our understanding of the model
decision making process. DeepKnowledge analyses the generalisation behaviour
of the DNN model at the neuron level to capture neurons that jointly
contribute to achieving the strong generalization power of the model not only
within the training distribution but also under a domain (i.e., data
distribution) shift. Therefore, we employ ZeroShot learning (Long et al.,
2017) to estimate the model’s generalisation capabilities under a new domain
distribution. ZeroShot learning enables the DNN model to make predictions for
classes that were not part of its training dataset. By analysing each neuron’s
ability to abstract knowledge from training inputs and generalising it to new
domain features, we establish a causal relationship between the neurons and
the DNN model’s overall predictive performance and identify _transfer
knowledge (TK) neurons_. The effective capacity of these transfer knowledge
neurons for learning (reuse–transfer features from training to a new domain)
positively influences the DNN generalisation behaviour and can explain which
high-level features influence its decision-making. Consequently, these neurons
are more critical for the correct DNN behaviour and should be allocated more
testing budget. The TK-based adequacy criterion instrumented by DeepKnowledge
quantifies the adequacy of an input set as the ratio of combinations of
transfer knowledge neuron neurons clusters covered by the set. We conduct a
large-scale evaluation with publicly available datasets (SVHN (Netzer et al.,
2011), GTSRB (Houben et al., 2013), CIFAR-10 and CIFAR-100 (Krizhevsky and
Hinton, 2010), MNIST (Deng, 2012)) and multiple DNN models for image
recognition tasks, demonstrating that our approach can develop a functional
understanding of the DNN generalisation capability and effectively evaluate
the testing adequacy of a test set. Moreover, the findings show a substantial
correlation between DeepKnowledge’s test adequacy criterion and the diversity
and capacity of a test suite to reveal DNN defects as indicated by the
coverage difference between the original test set and adversarial data inputs
(cf. Section 5.3). Our main contributions are:

  * •

The DeepKnowledge approach for identifying _transfer knowledge neurons_ that
have an influence on the DNN model’s decision confidence under domain shift
(Section 3.2);

  * •

The DeepKnowledge coverage criterion which can establish the adequacy of an
input set to trigger the DNN’s knowledge generalisation behaviour, thus
enabling to assess the semantic adequacy of a test set (Section 3.3);

  * •

An extensive experimental evaluation using state-of-the-art DNN models,
publicly-available datasets and recent test adequacy criteria (Section 5); and

  * •

A prototype open-source DeepKnowledge tool and repository of case studies,
both of which are publicly available from our project webpage at
https://anonymous.4open.science/r/DeepKnowledge-1FBC (Section 4).

Paper Structure. Section 2 overviews DNNs and DNN testing. Sections 3 and 4
introduce DeepKnowledge and its implementation. Section 5 describes the
evaluation of DeepKnowledge. Sections 6 and 7 discuss related work and
conclude the paper, respectively.

##  2\. Background

###  2.1. Deep Neural Network-based Systems

We consider any software system equipped with at least one DNN component as a
DNN-based system. These systems benefited from the great achievements DNNs
made across several application domains (Lecun et al., 2015). Unlike
traditional machine learning algorithms, DNNs do not require intensive feature
engineering (Lenz et al., 2015). Inspired by neurobiology, the DNN
architecture allows the network to extract features automatically from raw
data without manual intervention or support from domain experts. A DNN uses
mathematically complex computations to automatically extract high-level
patterns from within the inputs to make a prediction (Goodfellow et al.,
2016). Several DNN architectures have been introduced, including GANs
(Goodfellow et al., 2020) and transformers (Vaswani et al., 2017). Overall,
DNN can be classified into three main categories: multi-layer perceptrons
(MLP) (Kruse et al., 2022), convolutional neural networks (CNN) (Gu et al.,
2018), and recurrent neural networks (RNN) (Mikolov et al., 2010). As shown in
Figure 1, independently of its type, a DNN comprises multiple interconnected
neurons organized on hidden, input, and output layers. Neurons are
computational units that combine multiple inputs and produce a single output
by applying non-linear activation functions such as sigmoid, hyperbolic
tangent and rectified linear unit (ReLU). The main phases of DNN are training
(or learning), where the neural network abstracts knowledge (i.e., high-level
features) from training data, and inference (or prediction) when the network
applies the extracted knowledge to unforeseen inputs.

Figure 1. A four-layer fully-connected DNN system that receives inputs from
vehicle sensors (camera, LiDAR, infrared) and outputs a decision for speed,
steering angle and brake.

###  2.2. Software Testing DNNs

Software testing (Pezze and Young, 2008) is a well-defined software
engineering process, encompassing validation and verification, to establish
whether the system under test meets a set of specified requirements. More
informally, it is an in-depth investigation of the system’s behaviour to find
bugs, errors, or missing requirements in the developed software. A typical
example is unit testing (Runeson, 2006) that aims to tackle the smallest
testable parts of an application, called units, and assess their correctness
(Hunt and Thomas, 2004; Meszaros, 2007). An essential aspect of software
testing is the test oracle (Barr et al., 2014), which acts as a mechanism to
assess whether the software executed a test case correctly. Conceptually, this
involves comparing the oracle with the actual system output (Pezze and Young,
2008).

To ensure the safety and correctness of DNNs, research focuses on adapting
software testing methodologies such as test adequacy and code coverage to
DNNs. Software coverage is typically quantified by counting the statements or
paths executed by an input (Pezze and Young, 2008), aiming to maximize code
coverage, as higher coverage implies more comprehensive testing. However,
following a purely traditional software testing methodology to DNN testing is
not applicable due to: (i) the black-box nature of DNNs, which are data-driven
and typically non-interpretable predictive models for which their decision-
making process is unexplainable (Montavon et al., 2017); and (ii) volume-
related issue: DNN models are typically large structures whose parameters
(number of layers and neurons per layer) can reach billions in size. Thus, it
is inaccurate to compare neurons to software statements and layers to
execution paths (Zhang et al., 2019b). Instead, specialized testing criteria
have been proposed for DNNs. These criteria are designed to consider the DNN
structure and monitor the neurons and intrinsic network connectivity at
various levels of granularity (Zhang et al., 2019b). We review related work in
DNN testing and verification in Section 6.

Figure 2. Overview of DeepKnowledge

##  3\. DeepKnowledge

DeepKnowledge, whose high-level workflow is shown in Figure 2, is a systematic
testing approach for DNN systems. Using a trained DNN, an in-distribution (ID)
and an out-of-distribution (OOD) dataset (Yang et al., 2021), DeepKnowledge
analyses how the model learns and transfers knowledge abstractions under
domain shift. In this analysis, the learning problem is looked at through the
prism of the DNN generalisation theory (Urolagin et al., 2011; Kawaguchi et
al., 2017), based on which DeepKnowledge establishes a fundamental
understanding of out-of-distribution generalisation at the neuron level and
quantifies the individual contribution of each neuron to this process. Through
a filtering step, DeepKnowledge identifies a set of _Transfer Knowledge (TK)
neurons_ , which, given their contribution to the model’s generalisation
performance, are considered core DNN computational units. The rationale behind
using out-of-distribution generalisation is that it enables the analysis of
what part of the knowledge gained during DNN training can be employed in a new
domain without re-training or fine-tuning. Thus, it allows the simulation of
real-world corner cases the model could encounter during deployment. The next
DeepKnowledge step involves the execution of fine-grained clustering analysis
to determine activation value clusters that reflect the changes in neurons’
behaviour with respect to new inputs. The produced clusters of the transfer
knowledge neurons are then used to assess the coverage adequacy of the test
set, conforming to the combinatorial analysis method defined in DeepImportance
(Gerasimou et al., 2020).

The selection of the OOD dataset is crucial in DeepKnowledge. The OOD data is
regarded as a shift from the ID data, which by definition is the training
dataset in DNN engineering (Hendrycks et al., 2021). The distribution shift
can be either: (i) a synthetic distribution shift resulting from a computer-
generated perturbation, i.e., noise, adversarial attacks (Papernot et al.,
2016b); or (ii) a natural distribution shift due to unseen input data (Kurakin
et al., 2016). Since data distribution is critical in DNN testing, especially
for the generalisation behaviour, in DeepKnowledge, we carefully craft the OOD
data based on: (i) the semantic similarity with the ID dataset; and (ii) the
novelty compared to ID data samples. For instance, for a person detection DNN
model trained on data of pedestrians, images of people cycling would be ID
samples. In contrast, images of people with occlusion (i.e., hidden or
‘occluded’ by real-world objects such as a car) would be OOD samples.

###  3.1. Knowledge Abstraction Analysis

Identifying the Transfer Knowledge (TK) neurons within the DNN trainable
layers is a key principle of DeepKnowledge. Our approach begins by evaluating
the prominence of each neuron and its connections to the OOD generalisation
process (McCoy et al., 2019). This analysis aims to identify neurons that can
generalise knowledge abstracted during training and apply it to an OOD domain
without re-training or fine-tuning. These TK neurons are core components of
the DNN, collectively contributing to its generalisation behaviour, and,
consequently, have a positive effect on the DNN accuracy and robustness
(Kawaguchi et al., 2017), enabling them to reach their peak performance.

As shown in Figure 2, for this analysis, we employ two different datasets,
i.e., training samples and ZeroShot samples that correspond to in-distribution
and out-of-distribution data, respectively. These datasets are used to
quantify feature preference distributions over input features (e.g., image
pixels for CNN models) for both the training and ZeroShot stages. This
represents the knowledge abstraction analysis step that allows
quantifying/identifying how the DNN model computational units, i.e., neurons,
abstract/learn knowledge from the input feature space.

The identification of TK neurons is loosely inspired by recent research on
quantifying knowledge change in DNN under domain shift at the level of
individual neurons (Rethmeier et al., 2020). To this end, we employ the
Activation maximization explainability method (Erhan et al., 2009) to derive a
representation for features that neurons in the neural network have learned
during the training and ZeroShot stages. This method allows the identification
of input instances that produce the highest activation values for a neuron,
which are called _‘preferred’_ input features. Then, we turn these
representations into probability distributions to quantify the shift in
abstracted knowledge by utilizing a statistical metric on the defined space of
probability distributions at the individual level (i.e., per neuron). Neurons
able to achieve a certain threshold (that is empirically defined) of the
statistical measure are selected as candidate transfer knowledge neurons. We
leverage these insights and present the key contribution of this paper, namely
DeepKnowledge test adequacy criterion for testing DNN models by assessing test
set quality and semantic diversity.

Now, let’s formally explain this process. Given a trained DNN model $D$ with
$L$ trainable layers, each layer $L_{i},1\leq i\leq L$ has
$\left|L_{i}\right|$ neurons and the total number of neurons in $D$ is
$S=\sum_{i=1}^{L}\left|L_{i}\right|$. Let also $n_{ij}$ be the $j$-th neuron
in the $i$-th layer. When the context is clear, we use $n\in D$ to denote any
neuron that is a member of $D$ irrespective of its layer. Let $X$ denote the
input domain of $D$, representing either the in-distribution (ID) or out-of-
distribution (OOD) dataset, and $x_{k}\in X$ be the $k$-th concrete input from
$X$ (also termed input feature). Finally, we use the function
$\phi(x_{k},n_{ij})\in\mathbb{R}$ to signify the output of the activation
function of neuron $n_{ij}\in D$.

Given the input set $X$,
$D(x_{k})=\left(\phi\left(x_{k},n_{11}\right),\ldots,\phi\left(x_{k},n_{L_{L}|L_{L}|}\right)\right)$
enables calculating the activation trace of input $x_{k}\in X$. Using this
information, we signify with $\overline{n^{x_{k}}}=argmax_{n\in
D}\left(D(x_{k})\right)$ the maximally activated neuron and with
$\overline{\alpha^{x_{k}}}=max\left(D(x_{k})\right)$ the corresponding maximum
activation value. Applying these steps to the entire set $X$, we can construct
the matrix
$M=\left[\left(x_{k},\overline{n^{x_{k}}},\overline{\alpha^{x_{k}}}\right)\right]_{\forall
x_{k}\in X}$, where the $r$-th entry of $M$ captures the maximally activated
neuron $\overline{n^{x_{k}}}$ and its activation value
$\overline{\alpha^{x_{k}}}$ for input $x_{k}$. Using methods like Monte Carlo
Dropout (Srivastava et al., 2014; Gal and Ghahramani, 2016) can yield matrix
entries with different maximally activated neurons and/or values for the same
input $x_{k}$.

Next, DeepKnowledge establishes the input probability distribution per neuron
(Rethmeier et al., 2020) by aggregating the maximum activation values
$\overline{\alpha^{x_{k}}}$ for each neuron $n$ over its maximally activated
input features $x_{k}$. Accordingly,
$A(n_{ij})=\\{\left(x_{k},\mu_{x_{k}}\right)\;|\;\forall x_{k}\in
X:M_{n_{ij}}=\bigcup_{1\leq
r\leq\left|M\right|}(x_{k},n_{ij},\overline{\alpha^{x_{k}}})\;\bullet\;\mu_{x_{k}}=\frac{1}{\left|M_{n_{ij}}\right|}\sum_{\left(x,n,\alpha\right)\in
M_{n_{ij}}}\alpha\\}$. More informally, $A(n_{ij})$ provides the discrete
input activation distribution per neuron where $\mu_{x_{k}}$ is the mean
maximum activation of input $x_{k}$ for the $n_{ij}$ neuron. We can then use
the set $A(n_{ij})$ to calculate each neuron’s normalised probability
distribution per $x_{k}$ as:

(1) $P_{n_{ij}}=\left\\{\left(x_{k},p_{k}\right)\right\\}_{x_{k}\in
X},\qquad\quad p_{k}=\frac{\mu_{x_{k}}}{\sum_{(x_{l},\mu_{x_{l}})\in
A(n_{ij})}\mu_{x_{l}}}\vspace{-1mm}$

where $p_{k}$ is the activation probability of input feature $x_{k}$ given as
the mean activation $\mu_{x_{k}}$ over the sum of its activation means.

Finally, collecting the $P_{n_{ij}}$ enables assembling the DNN’s activation
distributions defined as
$P={P_{n_{ij}},\cdots,P_{n_{L_{L}\left|L_{L}\right|}}}$ over the dataset $X$.
Doing this both for the ID and OOD datasets enables assembling the
corresponding $P^{ID}$ and $P^{OOD}$ elements.

Algorithm 1 shows the high-level process DeepKnowledge employs for quantifying
knowledge change in model $D$. Starting with the empty matrix $M$ and
activation distribution $P$ (line 2), for any input $x_{k}\in X$, we perform
standard forward passes to compute the activation trace and identify the
maximally activated neuron $\overline{n^{x_{k,r}}}$ and the corresponding
maximum activation value $\overline{\alpha^{x_{k}}}$ (lines 3 – 8). Then, the
matrix $M$ of neurons input max activations is updated (line 7), which we
aggregate later to express each neuron as a probability distribution over
maximally activated inputs as defined in equation (1) (lines 10-17). Given the
input set $X$, each neuron $n$ is expressed as a distribution over preferred
inputs (line 16), which is returned when the algorithm terminates (line 19).
Executing Algorithm 1 for both $X^{ID}$ and $X^{OOD}$ datasets, yields the
distribution over preferred inputs $P^{ID}$ and $P^{OOD}$, respectively.

Algorithm 1 Abstracted Knowledge Quantification

1:function AbstractedKnowledgeQuantification($D$, $X$)

2: $M\leftarrow\emptyset$, $P\leftarrow\emptyset$

3: for $x_{k}\in X$ do $\triangleright$ $X$ $\in$ $\\{X^{ID},X^{OOD}\\}$

4: $\overline{n^{x_{k}}}=argmax(D(x_{k}))$

5: $\overline{\alpha^{x_{k}}}=max(D(x_{k}))$

6:
$v\leftarrow\left(x^{k},\overline{n^{x_{k}}},\overline{\alpha^{x_{k}}}\right)$
$\triangleright$ input’s activation row vector

7: $M=M\cup v$ $\triangleright$ neuron preferred inputs matrix

8: end for

9: for $n\in D$ do $\triangleright$ using $n$ (not $n_{ij}$) for
simplification

10: $A(n)\leftarrow\emptyset$

11: for $x_{k}\in X$ do

12: $M_{n_{ij}}=\textsc{ExtractFromM}(x_{k},n)$

13:
$\mu_{x_{k}}=\frac{1}{\left|M_{n_{ij}}\right|}\sum_{\left(x,n,\alpha\right)\in
M_{n_{ij}}}\alpha$

14: $A(n)=A(n)\cup\left(x_{k},\mu_{x_{k}}\right)$

15: end for

16: $P_{n}=\left\\{\left(x_{k},p_{k}\right)\right\\}_{x_{k}\in X}$
$\triangleright$ using (1)

17: $P=P\cup P_{n}$

18: end for

19: return $P$ $\triangleright$ $P^{ID}$ for $X^{ID}$, $P^{OOD}$ for $X^{OOD}$

20:end function

###  3.2. Transfer Knowledge Neurons Selection

Next, DeepKnowledge determines the _transfer knowledge (TK) neurons_ by
quantifying the _change_ and _diversity_ in the knowledge abstracted per
neuron $n_{ij}$ from the ID dataset to the OOD dataset using the activation
distribution probabilities $P^{ID}$ and $P^{OOD}$. Our approach estimates
_knowledge change_ by employing statistical difference measures between the
preference features distributions calculated per dataset. Statistical distance
measures are commonly used to measure distribution shifts (Hendrycks et al.,
2021). DeepKnowledge leverages the Hellinger Distance (HD), a symmetric
divergence measure that quantifies the deviation between two distributions,
i.e., it measures how far apart the two features preferred distributions are
in the space of probability distributions common to them. Concretely,
DeepKnowledge uses HD to measure the per-neuron knowledge change between the
activation distribution probabilities over the ID and OOD datasets, i.e.,
$p=P_{n_{ij}}^{ID}$ and $q=P_{n_{ij}}^{OOD}$ given by

(2)
$HD(p,q)=\frac{1}{\sqrt{2}}\sqrt{\sum_{j=1}^{X}(\sqrt{p_{j}}-\sqrt{q_{j}})^{2}}\;\in[0,1]\vspace{-2mm}$

HD is a bounded metric, taking values between 0 and 1; when $HD\\!=\\!0$, the
two distributions are identical, and when $HD\\!=\\!1$, they are the furthest
apart. Thus, a high distance for the $ij$-th neuron’s features preferred
distributions under domain shift ($P_{n_{ij}}^{ID},P_{n_{ij}}^{OOD}$),
illustrates that the neuron did not generalize its abstracted knowledge to the
new domain features well. The lower the HD value, the more the neuron can
transfer learned knowledge to the OOD data space. An empty distribution over
$P_{n_{ij}}^{ID}=0$ or $P_{n_{ij}}^{OOD}=0$ results in an invalid HD value,
which is assigned the smallest value possible (0).

Note that DeepKnowledge can employ other divergence metrics, including
Wasserstein distance (Vallender, 1974) and Jensen–Shannon divergence (Menéndez
et al., 1997). We emphasise the importance of using symmetric distance
measures as they have the advantage of providing a unique interpretation of
the knowledge shift, i.e.,
$distance(P_{n_{ij}}^{ID}||P_{n_{ij}}^{OOD})=distance(P_{n_{ij}}^{OOD}||P_{n_{ij}}^{ID})$,
while also requiring less computation time than asymmetric divergence measures
such as Kullback-Leiber (Joyce, 2011).

In addition to the _knowledge change_ given by the HD value per neuron,
DeepKnowledge uses _knowledge diversity_ between the ID and OOD datasets to
extract the TK neurons. Similarly to (Rethmeier et al., 2020), DeepKnowledge
leverages the feature length $l$ as a knowledge diversity indicator. To this
end, for each neuron $n_{ij}$, we quantify the total number of unique
preferred inputs denoted by $l(\overline{X_{n_{ij}}})$, where
$\overline{X_{n_{ij}}}=\\{x_{k}\;|\;\forall\left(x_{k},p_{k}\right)\in{P}_{n_{ij}}\bullet
p_{k}>0\\}$. As before, the set and its length can be constructed both for the
ID and OOD datasets. The rationale for this knowledge diversity analysis is
that input features that have maximally activated the same neuron in the ID
and OOD datasets can be considered semantically similar and can be deployed to
quantify the generalization capabilities of the neuron. The analysis enables
further the elimination of neurons whose HD value is low but do not have many
preferred inputs, and thus their $l(\overline{X_{n_{ij}}})$ is small either
for the ID or OOD datasets.

We define three neuron types based on $l(\overline{X_{n_{ij}}})$ (Rethmeier et
al., 2020): (1) ‘Gained’ neurons, where
$l(\overline{X^{ID}_{n_{ij}}})<l(\overline{X^{OOD}_{n_{ij}}})$, indicating
that these neurons have gained knowledge in ZeroShot learning and can
specialize in the new domain; (2) Avoided neurons, where
$l(\overline{X^{ID}_{n_{ij}}})>l(\overline{X^{OOD}_{n_{ij}}})$, meaning that
they lost part of the learned knowledge during the ZeroShot stage; and (3)
‘Stable’ neurons, where
$l(\overline{X^{ID}_{n_{ij}}})=l(\overline{X^{OOD}_{n_{ij}}})$.

Employing the _knowledge change_ and _knowledge diversity_ mechanisms
presented above, DeepKnowledge selects the set of transfer knowledge TK
neurons. The HD threshold and the neuron type (gained, avoided, preferred) are
hyperparameters of DeepKnowledge for knowledge change and diversity
estimation, respectively. We illustrate in our empirical evaluation (Section
5) what constitutes appropriate parameters ($HD\in(0.01-0.05]$ and gained).

###  3.3. Knowledge Coverage Estimation

To increase our confidence in the robustness of the DNN model’s behaviour, we
assess the test set adequacy, i.e., how well the test set exercises the set of
identified TK neurons. First, DeepKnowledge determines regions within the TK
activation value domain central to the DNN execution and clusters them into
separate combinations. Each combination of clusters reflects different
knowledge abstracted during the training phase. We can assess the knowledge
diversity of the test set by assessing if it adequately covers these
combinations, similar to combinatorial interaction testing in conventional
software testing (Petke et al., 2015). We posit that semantically similar
inputs, i.e., inputs with similar features, generate activation values with a
similar statistical distribution at the neuron level (Gerasimou et al., 2020;
Ma et al., 2018b).

More specifically, we adopt the combinatorial neuron coverage method
introduced in (Gerasimou et al., 2020) to first iteratively cluster the vector
of activation values of each TK neuron from the training set (cf. Section
3.1). Then, given inputs from the testing set, we measure the number of
combinations this set covers over the combinations of activation value
clusters of the TK neurons. Since the cluster combinations of activation
values generated correspond to semantically different features of the input
set, this combinatorial analysis could assess the knowledge diversity of the
test set.

Formally, we first identify regions within the TK value domain that are key to
the DNN, i.e., clusters of activation values using iterative unsupervised
learning via the combinations of k-means clustering (Likas et al., 2003) and
the Silhouette index (Rousseeuw, 1987). Clustering enables dimensionality
reduction of neuron activation values, with the Silhouette index employed to
reinforce the clustering output. Silhouette is an internal clustering
validation index that computes the goodness of a clustering structure without
external information, facilitating the selection of the optimal number of
clusters. The Silhouette score for $c\in\mathbb{N}^{+}$ clusters for neuron
$n\in D$ is given by:

(3)
$S_{n}^{c}=\frac{1}{\left|X\right|}\sum_{x=1}^{X}\frac{E\left(x\right)-I\left(x\right)}{max\left(E\left(x\right),I\left(x\right)\right)}$

where $I\left(x\right)$ is the intracluster cohesion, i.e., the average
distance of activation value $\phi\left(x,n\right)$ to all other values in the
same cluster, and $E\left(x\right)$ is the inter-cluster separation, i.e., the
distance between $\phi\left(x,n\right)$ and activation values in its nearest
neighbour cluster.

Next, we measure the degree to which a test set $X$ covers the clusters of TK
neurons, by evaluating the combinations of activation value clusters triggered
by inputs $x\in X$. Thus, having the set of clusters extracted and optimized,
we identify the vector of TK neurons cluster combinations (TCC) as follows:

(4) $TCC=\prod_{n\;\in\;TK}\left\\{Centroid\left(\Psi_{n}^{i}|\>\forall 1\leq
i\leq\left|\Psi_{n}\right|\right)\right\\}$

where the set $\Psi_{n}^{i}$ contains the activation values of cluster $i$ for
the $n$-th TK neuron and $Centroid(\Psi_{i}^{n})$ gives its centre of mass.
For more details on cluster extraction and $\Psi_{n}^{i}$ estimation, please
see (Gerasimou et al., 2020).

Finally, given the input set $X$, the DeepKnowledge  _Transfer Knowledge
Coverage (TKC)_ score is computed as the ratio of TCC covered by all $x\in X$
over the size of the TCC set as follows:

(5) $\mathrm{TKC}=\frac{|\\{TCC{\left(i\right)}|\exists x\in X:\forall
V_{n}^{j}\in TCC{\left(i\right)}\bullet
min\;d(\phi(x,n),V_{n}^{j})\\}|}{\left|TCC\right|}$

TCC(i) is covered if there is an input $x$ for which the Euclidean distance
$d(\phi(x,n),V_{n}^{j})$ between the activation values of all TK neurons and
the $i$-th neuron’s cluster centroids is minimised. Overall, the TKC score
quantifies if the combinations of activation value clusters of TK neurons have
been covered adequately by the test set. A higher TKC value entails a
knowledge-diverse test set executing different TK neuron cluster combinations,
helping to assess the test set’s knowledge diversity. TKC is sound by
construction; adding another input $x^{\prime}$ in $X$ will either preserve or
increase the TKC score.

##  4\. Implementation and hardware

All experiments were conducted on a high-performance computer running a
cluster GPU with NVIDIA 510.39. We implement DeepKnowledge using the state-of-
the-art framework, open-source machine learning framework Keras (v2.2.2)
(Gulli and Pal, 2017) with Tensorflow (v2.6) backend. We report the full
experimental results next. To allow for reproducibility, our open-source
implementation and evaluation subjects are available on DeepKnowledge’s Github
repository at https://anonymous.4open.science/r/DeepKnowledge-1FBC.

##  5\. Experimental Evaluation

###  5.1. Research Questions

RQ1 (Knowledge Generalisation): Is DeepKnowledge capable of capturing the most
influential neurons of a DNN system?

RQ2 (Hyperparameter Sensitivity): How do DeepKnowledge’s hyperparameters that
encode knowledge change and knowledge diversity affect the execution of the
approach?

RQ3 (Effectiveness): Can the TKC criterion of DeepKnowledge be effectively
used as a test adequacy criterion?

RQ4 (Correlation): How does TKC perform against state-of-the-art DNN testing
criteria (Pei et al., 2017; Ma et al., 2018a; Gerasimou et al., 2020; Kim et
al., 2019)?

RQ5 (OOD Selection ): What is the impact of the out-of-distribution dataset on
the selection of TK neurons and the coverage results?

###  5.2. Experimental Setup

Datasets and DNN-based Systems. We extensively evaluate DeepKnowledge on four
different DNN-based systems using: (1) the original test sets; and (2)
adversarial examples generated by four attack strategies. Table 1 lists the
widely-adopted datasets employed to evaluate DeepKnowledge. In particular, we
used MNIST (28x28 grayscale images of handwritten digits), SVHN (32x32
coloured images of real-world house numbers), and CIFAR-10 (32x32 coloured
images from 10 classes) as in-distribution (ID) data, and as out-of-
distribution datasets (OOD): Fashion MNIST (a dataset of Zalando’s article –
28x28 grayscale images), EMNIST (MNIST extension with both handwritten digits
and letters – 28x28 pixel images), CIFAR-100 (dataset of 32x32 colour images
from 100 different classes). The GTSRB dataset (Houben et al., 2013) (from the
German Traffic Sign Recognition Benchmark) with over 50,000 images classified
into 40 classes was also used for RQ5. We should highlight that the selected
OOD dataset should share features with the ID dataset (e.g., image size,
colour) (Hendrycks et al., 2021). We performed a comprehensive assessment of
DeepKnowledge using DNN-based systems from recent research (Zhang and Li,
2020). The majority of employed DNN models are convolutional DNNs with
different architectures, trained with different datasets, making them suitable
for the empirical analysis of DeepKnowledge and TKC as an adequacy criterion.
We study LeNet-1 and LeNet-5 from the LeNet family (LeCun, 1998), and the
ResNet18 model (Targ et al., 2016).

Table 1. Datasets and DNN models used in our experiments ID Dataset |  OOD Dataset |  DNN Model |  # Layers
---|---|---|---
MNIST (Deng, 2012) |  Fashion MNIST (Xiao et al., 2017) |  LeNet1 & 4 (LeCun, 1998) |  5
SVHN (Netzer et al., 2011) |  EMNIST (Cohen et al., 2017) |  LeNet5 (LeCun, 1998) |  7
CIFAR-10 (Krizhevsky and Hinton, 2010) |  CIFAR-100 (Krizhevsky et al., 2009) |  ResNet18 (Zhang et al., 2019a) |  18

Adversarial Examples. We use the following techniques to evaluate
DeepKnowledge through adversarial examples: Fast Gradient Sign Method (FGSM)
(Liu et al., 2019), Basic Iterative Method (BIM) (Goodfellow et al., 2014),
Momentum Iterative Method (MIM) (Dong et al., 2018), and Projected Gradient
Descent (PGD) (Deng and Karam, 2020). Adversarial inputs are crafted using the
Cleverhans Python library (Papernot et al., 2016a) to generate perturbed
examples based on given benign inputs. For each of the original test sets
(SVHN, MNIST and CIFAR-10), we generate an adversarial set of equal size.

Coverage Criteria Configurations. We used the following DeepKnowledge
parameter ranges in the experiments: $HD\in[0.01,0.25]$,
$|TK|\in\left\\{20\%,30\%,40\%\right\\}$ of the total candidate TK neurons and
all three knowledge diversity types, i.e., Gained, Avoided, and Stable
(Section 3.2). When comparing DeepKnowledge against state-of-the-art coverage
criteria for DNN systems, for each criterion we use the hyperparameters
recommended in its original research paper. To this end, for all research
questions, we set the threshold for neuron coverage (NC) (Pei et al., 2017) to
0.7. For k-multisection neuron coverage (KMNC) and top-k neuron coverage
(TKNC) (Ma et al., 2018a), the number of multisections $k=10$ and $k=3$,
respectively. For neuron boundary coverage (NBC) and strong neuron activation
coverage (SNAC) (Ma et al., 2018a), we set as the lower/upper bound the
minimum/maximum activation values encountered in the training set. For
likelihood-based surprise coverage (LSC) (Kim et al., 2019), we use the last
convolutional layer, with the upper bound set to 2000 and using 1000 buckets.

###  5.3. Results and Discussion

RQ1 (Knowledge Generalisation).

In this RQ, We experiment with our test coverage criterion on state-of-the-art
DNNs that have a broad range of sizes to demonstrate the utility of our
criteria with respect to the Knowlwdge generalisation efficiency of testing
input generation, In this research question, we investigate whether the
selected TK neurons can be used to instrument the TKC test adequacy criterion.
To answer this research question, we employ a DeepKnowledge-based model
retraining strategy where we augment the training set with a set of input
images selected by the DeepKnowledge. The input images that maximally activate
the TK neurons are identified and used to retrain the DNN. If these inputs are
effective in improving the model accuracy, then, DeepKnowledge is able to
select semantically meaningful data that improve the model generalisation
capabilities.

Table 2. Effect of TKC-based retraining on DNN accuracy

DNN |  Dataset |  Initial |  $T2_{TKC}$ |  $T2_{R}$ |  $T2_{A}$
---|---|---|---|---|---
LeNet1 |  MNIST |  97.75% |  98.84% |  91.69% |  9.74%
LeNet5 |  MNIST |  98.10% |  99.04% |  98.89% |  9.32%
SVHN |  84.51% |  85.12% |  84.18% |  19.58%
CIFAR-10 |  93.05% |  99.88% |  89.94% |  10.97%
ResNet18 |  SVHN |  95.77% |  95.90% |  95.56% |  6.69%

  * •

*Initial, $T2_{TKC}$, $T2_{R}$ and $T2_{A}$ denote the initial DNN accuracy, and after DeepKnowledge-, Random- and Adversarial-based retraining, respectively.

We split our ID test set into separate subsets $T_{1}$ and $T_{2}$ using a 60%
- 40% ratio. Then, we identified input images that improve DeepKnowledge
coverage (i.e., maximally activated the TK neurons) from within the $T_{2}$
subset, producing DeepKnowledge-guided input sets $T2_{TKC}$. We highlight
that only inputs that produce the highest activation for all TK neurons are
kept. This resulted in augmented sets limited in size, i.e., 6K, 3K, and 5K,
for MNIST, SVHN, and CIFAR-10, respectively. Similarly, we generated augmented
sets $T2_{R}$ using randomly selected neurons, ensuring that none of the
$T2_{R}$ inputs is included in $T2_{TKC}$. The third augmented set $T2_{A}$ is
generated using the PGD adversarial technique. Finally, we retrain the DNN
models separately using the three different augmented datasets: $T2_{TKC}$,
$T2_{R}$ and $T2_{A}$. Based on empirical observations that DNN retraining on
small datasets shows a performance decline after the 10th epoch, we stop
retraining after that point.

Table 2 reports the DNN prediction accuracy on the $T_{1}$ testing sets after
retraining. For each model, we report the accuracy of the original model
without retraining (‘Initial’), and the prediction accuracy after the
DeepKnowledge-based ($T2_{TKC}$), random-based ($T2_{R}$) and PGD-based
($T2_{A}$) retraining strategies. The accuracy of each model after retraining
with a separate test set shows that DeepKnowledge-based augmentation strategy
using $T2_{TKC}$ achieved on average $1\%$ higher accuracy improvement over
the initial accuracy, e.g., from $97.75\%$ to $98.84\%$ for LeNet1-MNIST and
from $98.10\%$ to $99.04\%$ for LeNet5-MNIST. Unsurprisingly, the adversarial-
based augmentation strategy ($T2_{A}$) produced a set that caused a
significant drop in accuracy, from $97.75\%$ to $9.74\%$ for LeNet1-MNIST.
Clearly, the limited size of the $T2_{TKC}$, e.g., only 5K instances have been
generated for the CIFAR-10 dataset, contributes to the small rise in the DNN
accuracy after DeepKnowledge-based retraining. Although the improvement made
by $T2_{TKC}$ was limited (in some cases similar in comparison with the
$T2_{R}$ set), the consistency in this behaviour over the 3 models and 3
different datasets lead to the conclusion that DeepKnowledge-based
augmentation strategy is efficient in generating data instances that improve
the DNN accuracy. The $T2_{TKC}$ set improved the predictive accuracy and
general performance of the different tested models. It added new variances of
data through samples that had a direct impact on the learning process. Thus,
these inputs are semantically significant to the generalisation process of a
DNN. These observations provide insights into the sensitivity of TK neurons to
semantically relevant input features, which entails that TK neurons are key
contributors to the model generalization capability.

Based on these findings, we conclude that DeepKnowledge can identify neurons
that significantly contribute to the generalization process and, as a result,
influence the DNN decision-making process. Thus, the TK neurons can be
efficiently utilized as a test adequacy criterion to evaluate the testing
adequacy of a test set. Further, DeepKnowledge can be successfully deployed to
adjust the DNN training phase by augmenting the training dataset to improve
the predictive accuracy and general performance, thus reducing its erroneous
behaviour.

RQ2 (Hyperparameter Sensitivity). We examined the impact of DeepKnowledge’s
hyperparameters used for encoding knowledge change and knowledge diversity. To
this end, we varied (i) the percentage of selected TK neurons, i.e.,
$|TK|\in\left\\{20\%,30\%,40\%\right\\}$ of the total candidate TK neurons
resulting from Step 2 of DeepKnowledge; (ii) the HD threshold, i.e.,
$HD\in[0.01,0.25]$; and (iii) the neuron knowledge diversity types, i.e.,
Gained and Stable. The results show that the TK set size varies between
different DNNs. In some cases, this variation is extensive: 920 for LeNet1,
883 for LeNet5 and 1289 for ResNet18. The comparison of the HD bound for
different DNN and dataset combinations showed that the $HD$ distribution of
candidate neurons has an upper bound of $0.25\pm 0.01$.

An effective testing criterion should provide fine-grained insight into the
DNN behaviour and also be hard, but not impossible, to cover (Zhang et al.,
2019b). Since DeepKnowledge is underpinned by the knowledge generalisation
principle (Neyshabur et al., 2017), we aim at selecting a subset of the TK
neurons that satisfy the main generalisation criterion of _optimal capacity_ ,
which provides a balance between overfitting and learning new knowledge. To
evaluate the validity of a candidate TK neuron $n$ against this criterion, we
empirically analyse the following combinations of knowledge change
$HD\in\\{(0.01-0.05],(0.10-0.15],(0.15-0.20],(0.20-0.25]\\}$ and knowledge
diversity: Gained, $l(\overline{X^{ID}_{n}})<l(\overline{X^{OOD}_{n}})$, or
Stable, $l(\overline{X^{ID}_{n}})=l(\overline{X^{OOD}_{n}})$. Then, given the
set of candidate TK neurons, we select the top $p$ percentage with
$p\in\left\\{10\%,20\%,30\%,40\%,50\%\right\\}$, forming the transfer
knowledge neuron sets $TK_{10},TK_{20}$ etc.

As shown in Figure 3, overall the average DeepKnowledge coverage values
decrease when the analysis is performed on a bigger percentage of TK neurons
(e.g., $TK_{40}$). By analyzing the effect of knowledge diversity (i.e.,
Gained vs Stable) on the average results of DeepKnowledge, we observe Stable
TK neurons perform poorly as a test adequacy criterion as the coverage becomes
saturated and stagnates for most of the tested datasets (see the left column
in Figure 3), i.e., the coverage reaches its peak already with only $TK_{10}$
of the TK candidate neurons set. For instance, for MNIST and SVHN, the
DeepKnowledge coverage value stagnated at 100% even with the increase of the
percentage of used TK neurons from $TK_{10}$ to $TK_{50}$. We observe the same
behaviour irrespective of the selected HD values. Accordingly, this can be
interpreted as a failure of these neurons to generate diverse cluster
combinations (TCC) and thus to abstract diverse semantic features from the
dataset.

We notice that the selected TK neurons featured as Gained, result in coverage
values (right column in Figure 3) that moderately decrease as the percentage
increases (from $TK_{10}$ to $TK_{50}$). This criterion is also shown to be
more difficult to cover by the test sets. Moreover, DeepKnowledge coverage
with $HD\in(0.01-0.05]$ entails the ability to assign degraded coverage value
according to the size of TK neurons set, with an adequacy criterion gradually
harder to cover compared to the other neurons belonging to $HD$ ranges
$(0.10-0.15]$, $(0.15-0.20]$ or $(0.20-0.25]$.

The results show that Gained TK neurons with $HD\in(0.01-0.05]$ are more
difficult to cover, leading to a coverage metric that gradually decreases as
the percentage $p$ increases. The activation values of those TK neurons are
concentrated in specific regions within their inputs domain. Informally, those
regions capture features shared between ID and OOD datasets and are the most
influential for learning. Thus, the combinations of TK neuron clusters are
less likely to be covered if testing inputs do not introduce new semantically
diverse features (compared to those from the ID dataset).

Figure 3. DeepKnowledge coverage with different hyperparameters $HD$ and $l$
settings for different datasets.

Table 3 illustrates these results in more detail, showing the average TKC
value for different DNN models and percentages of Gained TK neurons with
$HD\in(0.01-0.05]$. To reduce randomisation bias, we report average results
over five independent runs. As shown, the TKC adequacy criterion with the
hyperparameters Gained, $l(\overline{X^{ID}_{n}})<l(\overline{X^{OOD}_{n}})$,
and $HD\in(0.01-0.05]$ results in a coverage value that becomes lower as the
number of TK neurons increases. Considering ResNet18, for instance, coverage
decreases from $82.69\%$ to $0.62\%$ for $TK_{10}$ ($\left\lvert
TK_{10}\right\rvert=14$), and $TK_{50}$ ($\left\lvert
TK_{50}\right\rvert=73$), respectively. These results are not surprising, as
the number of clusters of TK neurons extracted from equation (4) is between
two and five. In fact, the decrease in DeepKnowledge coverage reflects the
rise in the combinations of TK neurons clusters (TCC) that increases
exponentially as the percentage $p$ increases (e.g., $[86,104]$ for
$\left\lvert TK_{10}\right\rvert=14$ and $[11994,1934664]$ for $\left\lvert
TK_{50}\right\rvert=73$).

Table 3. Average TKC results for different percentages of TK neurons
$p\in\\{10\%,20\%,30\%,40\%,50\%\\}$ using Gained TK neurons and
$HD\in(0.01-0.05]$

$p\%$ 10 20 30 40 50 MNIST LeNet4 75.00%  $\left\lvert n\right\rvert=15$
37.39% 10.08% 6.43% 0.43% LeNet1 45.27%  $\left\lvert n\right\rvert=17$ 1.64%
0.35% 0.06% 0.00% SVHN LeNet5 70.12%  $\left\lvert n\right\rvert=35$ 17.88%
0.01% 0.00% 0.00% ResNet18 38.69%  $\left\lvert n\right\rvert=35$ 11.55% 0.23%
0.00% 0.00% CIFAR-10 LeNet5 49.59%  $\left\lvert n\right\rvert=15$ 41.67%
19.82% 10.79% 0.13% ResNet18 82.69%  $\left\lvert n\right\rvert=14$ 38.76%
25.57% 13.12% 0.62%

The results above provide evidence that TK neurons showing a steady increase
in their features length $l$ during the ZeroShot stage, using the OOD dataset,
can specialize their knowledge to the new domain, achieving a balance between
overfitting and learning new knowledge in line with the generalisation
principle (Neyshabur et al., 2017). Hence, we conclude that DeepKnowledge is
sensitive to its hyperparameters, and is affected by the selected HD values,
knowledge diversity, and the percentage of candidate TK neurons. Further,
DeepKnowledge benefits from choosing Gained TK neurons with
$HD\in(0.01-0.05]$.

Table 4. DeepKnowledge coverage (TKC) vs other metrics with adversarial inputs
added to the original test set $S_{0}$

LeNet1 (MNIST) LeNet4 (MNIST) LeNet5 (SVHN) ResNet18 (CIFAR-10) LeNet5
(CIFAR-10) S0 MIM BIM FGSM PGD S0 MIM BIM FGSM PGD S0 MIM BIM FGSM PGD S0 MIM
BIM FGSM PGD S0 MIM BIM FGSM PGD TKC20 1.64% 3.49% 3.35% 3.43% 3.72% 37.39%
50.69% 51.11% 49.66% 50.80% 17.88% 19.10 18.43% 19.21% 19.08% 38.76% 45.07%
44.99% 45.14% 44.99% 41.67% 50.07% 49.80% 49.93% 49.80% TKC30 0.35% 0.73%
0.71% 0.72% 0.75% 10.08% 17.39% 17.30% 17.20% 16.88% 0.01% 0.04% 0.04% 0.07%
0.06% 25.57% 25.63% 25.57% 25.58% 25.57% 19.80% 19.95% 19.88% 19.85% 19.95%
TKC40 0.06% 0.14% 0.11% 0.13% 0.13% 6.43% 12.17% 11.92% 12.13% 11.51% 0.00%
0.02% 0.01% 0.02% 0.02% 13.12% 13.15% 13.14% 13.14% 13.15% 10.79% 10.82%
10.84% 10.80% 10.82% LSC 0.2% 0.3% 0.36% 4.0% 4.0% 0.20% 0.30% 0.40% 0.40%
0.40% 0.20% 0.30% 0.30% 0.30% 0.30% 0.3% 0.20% 0.20% 0.30% 0.30% 0.1% 0.26%
0.2% 0.3% 0.3% KMNC 69.20% 71.92% 81.82% 83.60% 80.95% 62.71% 82.70% 79.60%
82.35% 84.70% 54.33% 71.60% 64.9% 62.7% 60.97% 43.60% 76.30% 64.40% 82.19%
69.28% 44.73% 59.7% 61.50% 66.11% 64.30% NC 20.05% 20.05% 20.05% 20.05% 20.05%
23.80% 23.80% 23.80% 23.80% 23.80% 73.21% 73.57% 74.28% 77.85% 73.57% 71.78%
75.94% 67.21% 74.49% 73.76% 74.28% 75.03% 75.03% 75.03% 75.03% NBC 7.97%
38.04% 40.94% 46.01% 41.66% 15.47% 46.42% 40.70% 47.61% 42.40% 22.67% 46.50%
45.30% 31.13% 23.57% 12.50% 28.10% 33.40% 41.30% 38.80% 9.10% 12.8% 20.3%
46.2% 40.7% SNAC 12.31% 62.315 79.71% 73.18% 72.46% 19.04% 61.90% 53.40%
54.76% 54.76% 39.28% 13.80% 71.30% 83.20% 40.71% 21.42% 31.40% 37.20% 53.40%
45.10% 16.78% 21.1% 19.8% 41.3% 46.2% TKNC 91.30% 91.30% 91.30% 91.30% 91.30%
100.00% 100.0% 100.00% 100.00% 100.00% 87.14% 88.21% 87.50% 87.50% 87.44%
87.85% 95.65% 94.92% 93.47% 95.65% 89.64% 89.64% 89.64% 98.8% 98.8%

RQ3 (Effectiveness): The effectiveness of DeepKnowledge as an adequacy
criterion is evaluated by analysing its sensitivity to adversarial inputs and
whether it can detect misbehaviours in datasets with inputs semantically
different from the original test inputs. Thus, we control input diversity by
cumulatively adding inputs generated by different adversarial techniques. By
exploiting the state-of-the-art adversarial approaches FGSM (Liu et al.,
2019), BIM (Kurakin et al., 2016), MIM (Dong et al., 2018), and PGD (Deng and
Karam, 2020), multiple sets of adversarial inputs are devised with a size
equal to 30% of the original test set $S_{0}$. Then, we execute the studied
DNN models with these inputs and compare the observed changes of various
coverage criteria, including DeepKnowledge. Similarly to state-of-the-art DNN
testing criteria NC (Pei et al., 2017), IDC (Gerasimou et al., 2020), KMNC,
NBC, SNAC, TKNC (Ma et al., 2018a), and LSC (Kim et al., 2019), we expect an
increase in the computed DeepKnowledge coverage.

Table 4 (rows $TKC_{20}$, $TKC_{30}$, $TKC_{40}$) reports the average
DeepKnowledge coverage results respectively for
$p\in\left\\{20\%,30\%,40\%\right\\}$ across the three augmented test sets
MNIST, SVHN and CIFAR-10 with the four different adversarial attacks. The
DeepKnowledge coverage has a notable increase with LeNet1 and LeNet4 models
trained with MNIST once adversarial examples are introduced. As shown in row
$TKC_{20}$, DeepKnowledge-based coverage increases for LeNet4 from $37.39\%$
for the original MNIST test inputs to $51.11\%$ when BIM examples were
introduced. Considering the CIFAR-10 dataset, the DeepKnowledge values still
increase when augmenting the test set with adversarial examples but with
different margins for different DNNs. For ResNet18, as shown in row $TKC_{20}$
(Table 4), DeepKnowledgeincreases by $\pm 7\%$ when introducing FGSM
adversarial inputs compared to its original value with $S_{0}$.

Compared to the original test set $S_{0}$, there is a considerable increase in
the DeepKnowledge result for the augmented test sets for all DNNs models. As
expected, the DeepKnowledge result for $p=20\%$ ($TKC_{20}$) is higher than
that for $p=40\%$ ($TKC_{40}$) as the number of TCC grows exponentially with
the number of TK neurons. In general, the increase is more significant in test
sets involving adversarial inputs generated using MIM attacks. This study
demonstrates the effectiveness of TK as a test adequacy criterion in detecting
misbehaviours in test sets, revealing its sensitivity even when exposed to a
small set of adversarial inputs. Further, DeepKnowledge is consistent with
existing coverage criteria in measuring shifts in data distributions due to
adversarial inputs (Kim et al., 2019; Ma et al., 2018a).

Based on these findings, we conclude that DeepKnowledge is sensitive to
adversarial inputs and is effective in helping software engineers detect
misbehaviours in data and adversarial attacks in general.

RQ4 (Correlation): In this research question, we assess whether DeepKnowledge
is correlated to existing test adequacy criteria for DNNs. We evaluate this
correlation in response to semantically diverse input sets. In fact, we aim at
a coverage metric that is consistent with existing coverage criteria based on
counting aggregation. We control the input diversity by cumulatively adding
inputs from the test set (i.e., $25\%S_{0}$, $50\%S_{0}$, $75\%S_{0}$ and
$100\%S_{0}$). By gradually adding inputs, we generate sets that comprise
diverse inputs with semantically different features. For instance, with over
26K inputs, the $25\%S_{0}$ of SVHN test set comprises less diverse features
than $50\%S_{0}$. We report results on how state-of-the-art coverage criteria
perform across the three test sets SVHN, MNIST, and CIFAR-10 versus
DeepKnowledge in Table 5.

Table 5. DeepKnowledge coverage (TKC) vs other metrics for different test set
sizes (in % of the original test set $S_{0}$)

LeNet 4 (MNIST) LeNet5 (SVHN) LeNet5 (CIFAR-10) ResNet18 (CIFAR-10) Test set
size 1/4 S0 1/2 S0 3/4 S0 S0 1/4 S0 1/2 S0 3/4 S0 S0 1/4 S0 1/2 S0 3/4 S0 S0
1/4 S0 1/2 S0 3/4 S0 S0 TKC20 25.75% 31.23% 35.33% 37.39% 9.88% 13.46% 15.94%
17.88% 40.62% 40.62% 41.67% 41.67% 28.00% 34.64% 36.87% 38.76% TKC30 5.29%
7.58% 9.09% 10.08% 0.00% 0.00% 0.01% 0.01% 11.04% 15.26% 17.79% 19.82% 13.42%
19.12% 22.94% 25.57% TKC40 2.88% 4.34% 5.55% 6.43% 0.00% 0.00% 0.00% 0.00%
5.22% 7.71% 9.45% 10.79% 6.68% 9.67% 11.65% 13.12% IDC4 12.50% 12.50% 12.50%
12.50% 12.50% 18.75% 18.75% 18.75% 25% 25% 25% 25% 27.67% 27.67% 27.67% 32.12%
IDC6 3.12% 3.12% 3.12% 3.12% 3.12% 4.69% 4.69% 4.69% 10.94% 10.94% 10.94%
10.94% 20.94% 20.94% 24.04% 26.14% LSC 0.2% 0.2% 0.2% 0.2% 0.2% 0.2% 0.2% 0.2%
0.1% 0.1% 0.1% 0.1% 0.2% 0.2% 0.3% 0.3% KMNC 48.9% 56.26% 60.26% 62.71% 40.64%
47.49% 51.57% 54.33% 30.81% 38.04% 42.01% 44.73% 30.15% 37.09% 40.95% 43.60%
NC 23.80% 23.80% 23.80% 23.80% 69.64% 71.42% 72.5% 73.21% 68.57% 71.42% 72.85%
74.28% 68.21% 70.0% 71.42% 71.78% NBC 4.76% 11.90% 14.28% 15.47% 8.57% 12.67%
18.92% 22.67% 1.78% 4.82% 7.14% 9.10% 2.32% 6.60% 10.35% 12.5% SNAC 9.52%
14.28% 16.66% 19.04% 14.64% 22.85% 33.21% 39.28% 3.57% 9.28% 13.21% 16.78%
3.92% 10.0% 17.14% 21.42% TKNC 100.00% 100.00% 100.00% 100.00% 84.64% 87.14%
87.14% 87.14% 84.64% 88.92% 89.64% 89.64% 83.92% 86.42% 87.14% 87.85%

First, we found that TCC as a test adequacy criterion is consistent with the
state-of-the-art coverage criteria based on neuron activation/property values
such as NC (Pei et al., 2017), KMNC (Ma et al., 2018a) and NBC (Ma et al.,
2018a). DeepKnowledge’s values increase with the increase of test set size,
i.e., with MNIST, $p=20\%$ ($TKC_{20}$), the TKC score increases from
$25.75\%$ to $31.23\%$ when cumulatively adding 25% (column $1/2S_{0}$) of the
test set and reaches its maximum $37.39\%$ for the MNIST test set as soon as
we reach 100% of its size, i.e., $\left|S_{0}\right|=10K$.

An in-depth exploration of the obtained results reveals that DeepKnowledge
(rows $TKC_{20}$, $TKC_{30}$ and $TKC_{40}$ in Table 5) was able to detect
these diversities through an increasing coverage value. In contrast, NC was
stable with different sizes of $S_{0}$ for MNIST and exhibited a small
increase with the other datasets. On the other hand, coverage criteria based
on input surprise, i.e., LSC, were stable when cumulatively increasing the
size of the test set, e.g., 0.1% for LeNet5 - CIFAR 10 and 0.2% for
LeNe4-MNIST. Since LSC (Kim et al., 2019) aims to measure the relative
novelty/surprise of input with respect to the training set, it is legitimate
to assume that the surprise level for these subsets of $S_{0}$ is
insignificant and was not translated into an increase in the LSC score as the
test set size increases. This is an interesting finding that requires further
investigation. While the original tests $100\%S_{0}$ achieves 62.71% KMNC (k =
10) on LeNet4, it only obtains 48.9% on $25\%$ of $S_{0}$. In line with
DeepKnowledge and KMNC results, the $100\%$ of $S_{0}$ MNIST test set achieves
a higher value for NBC with $15.47\%$ while $25\%$ of $S_{0}$ achieves only
4.76%. We can notice the same behaviour with all the DNN and dataset
combinations we have tested. The observed trend supports our claim that
DeepKnowledge assesses an input based on the knowledge (i.e., features) that
can be abstracted from it by the TK neurons. Hence, the TK neurons learn
unique features from within a dataset based on how well the input data can
cover various TCC combinations (4), which ensures the semantic diversity of
the tested inputs.

Overall, DeepKnowledge coverage has a good performance in detecting the
diversity of the test set. TKC increases with the increase of test set size,
proving that the DeepKnowledge adequacy criterion is sound (Pezze and Young,
2008) and sensitive to semantically different inputs. Further, DeepKnowledge
coverage shows similar behaviour to state-of-the-art coverage criteria for DNN
systems, and is consistent with coverage criteria based on aggregating neuron
property values (e.g., NBC, IDC).

Based on these findings, we confirm the positive correlation between our
approach and the studied neuron coverage criteria. We also conclude that the
TKC test adequacy criterion can support software engineers to evaluate the
diversity of a test set that comprises semantically different test inputs.

RQ5 (OOD Selection). DeepKnowledge is mainly based on suitable ID and OOD
dataset selection. Thus, we examine the robustness of our approach by
evaluating the effect of ID/OOD dataset combinations on the TK neurons
selection stage of DeepKnowledge as well as the final coverage estimation. For
each model, we replicate the experiments with the same settings and
hyperparameters, changing only the OOD dataset. We monitor the OOD selection
based on the rules discussed earlier in Section 3, so that the ID and OOD data
have common features, i.e., color, resolution, etc.

Table 6 illustrates the results where we measure the consistency of the
coverage against the changes in the OOD dataset. The results reveal that on
average the number of TK neurons correlates strongly with the type of OOD
data. Upon closer examination, we observe that the variations occur not only
because of increasing the number of TK, but also due to changing most of them
when the OOD dataset changes. Thus, the alteration in TK neurons has impacted
the coverage estimation. For instance, the coverage for LeNet1-MNIST with
adequacy criterion 30% of TK neurons ($TKC_{30}$) increased from $30.23\%$ to
$71.43\%$ when using EMNIST as the OOD dataset instead of Fashion MNIST,
although the size of TK set has been reduced from 29 to 12. We observe a
similar pattern with LeNet4-MNIST when we deploy EMNIST as the OOD dataset.
With LeNet5-SVHN the change in OOD data displays unexpected behaviours. We
observe that both the number of TK neurons and the coverage decreased when the
EMNIST OOD data was replaced by the GTSRB dataset. Independently of the number
of TK neurons, the coverage increases based on the covered TCC combinations.
In fact, when equipped with a suitable OOD dataset, we can effectively
identify TK neurons that capture the most essential semantic features present
in the dataset, i.e, coverage combinations TCC. This, in turn, leads to a more
equitable and accurate coverage estimation when testing the model on the test
set.

Therefore, to ensure a comprehensive testing process, it is essential to limit
the selection of ID/OOD combinations to datasets where all images belong to
the same “macro” category. This restriction enables inputs to share crucial
semantic features, facilitating the identification of the appropriate TK
neurons. Based on these findings, we conclude that the selection of OOD
dataset does affect the whole DeepKnowledge workflow and the coverage
estimation in particular.

LeNet1 (MNIST) LeNet4 (MNIST) LeNet5 (SVHN) OOD F.MNIST EMNIST F.MNIST EMNIST
EMNIST GTSRB #TK 29 12 24 12 87 75 TKC30 30.23% 71.43% 28.89% 66.67% 39.71%
14.01% TKC100 38.32% 49.44% 34.30% 50.13% 8.71% 1.37% • *OOD dataset size: 800
samples for all model/data combinations. F.MNIST: Fashion MNIST dataset; #TK:
total number of TK neurons.

Table 6. Comparison of different OOD datasets impact on the number of TK
neurons and TKC score

###  5.4. Threats to Validity

Construct Validity. The primary threat to the construct validity of
DeepKnowledge is the correctness of the studied DNN models and the datasets
used. We mitigate these construct validity threats using widely-studied
datasets, and also employing publicly-available architectures and pre-trained
DNN models that achieved competitive performance; see Table 1. Further, we
mitigate threats to identifying core units responsible for the DNN prediction
and knowledge abstraction by adapting the state-of-the-art approach for the
analysis of knowledge generalisation under domain shift (Kawaguchi et al.,
2017; Rethmeier et al., 2020).

Internal Validity. The main internal validity threat that could introduce bias
to DeepKnowledge is the computation of the TKC score. To mitigate this threat,
we design a study based on a specific set of research questions. We first
illustrate the effectiveness of the DeepKnowledge selection methodology of TK
neurons and their effectiveness as a test adequacy criterion in RQ1 and RQ4.
For RQ4, we illustrate DeepKnowledge’s effectiveness by augmenting the
original test sets with numerically diverse inputs. RQ3 was also designed to
illustrate the effectiveness of DeepKnowledge to detect adversarial examples
and confirmed its positive correlation with the state-of-the-art test adequacy
criteria NC (Pei et al., 2017), KMNC, NBC, SNAC, TKNC (Ma et al., 2018a), LSC
(Kim et al., 2019) and IDC (Gerasimou et al., 2020). The performance of our
approach is also demonstrated through RQ2 for different values of TK neurons
$p\in\left\\{10\%,20\%,30\%,40\%,50\%\right\\}$, Hellinger Distance
$HD\in(0.01-0.05]$ and neuron types (‘Gained’ or ‘Stable’), signifying
knowledge change and knowledge diversity, respectively (Rethmeier et al.,
2020).

External Validity. Threats to external validity have been mostly mitigated by
developing DeepKnowledge on top of the open-source frameworks Keras and
Tensorflow. We also validated DeepKnowledge using popular DNNs trained on
publicly-available datasets (e.g., MNIST, SVHN, CIFAR-10). However, more
experiments are needed to validate DeepKnowledge for TK neurons’ extraction
using Supervised transfer learning rather than ZeroShot transfer learning.
This will enable us to assess if TK neurons can generalise knowledge in more
challenging domain shift scenarios.

##  6\. Related Work

Testing and assurance of DNN-based systems is a growing research field (Burton
et al., 2017; Singh et al., 2019). Recent research efforts have resulted in a
broad range of approaches for DNN testing; see (Zhang and Li, 2020; Huang et
al., 2020) for recent surveys. In fact, DNN behaviour heavily relies on the
training data and learned parameters such as weights and biases, which makes
it impractical to apply conventional software testing techniques to ensure the
quality and safety of DNNs. Thus, recent advances introduced techniques for
DNN adequacy criteria (Pei et al., 2017; Ma et al., 2018a; Gerasimou et al.,
2020; Kim et al., 2019), input generation and selection (Tian et al., 2018; Ma
et al., 2021; Sun et al., 2020), fault localisation (Eniser et al., 2019) and
repair (Sohn et al., 2023; Kim et al., 2023).

Test adequacy, in particular, provides a quantitative assessment of the
effectiveness of a testing set, where achieving high coverage indicates
comprehensive testing of the model (Zhang et al., 2020). DeepXplore (Pei et
al., 2017) pioneered research by proposing neuron coverage (NC) as a white-box
testing criterion to identify the parts of DNN logic exercised by a test set.
Simply put, NC measures the ratio of neurons whose activation values are above
a predefined threshold. Eventually, it helps software testers to select
properties of a program to focus on during testing. More fine-grained
approaches were introduced which provide improvements over NC, including
DeepGauge (Ma et al., 2018a), DeepGini (Feng et al., 2020), DeepTest (Tian et
al., 2018) and DeepImportance (Gerasimou et al., 2020). For instance,
DeepGauge (Ma et al., 2018a) has introduced multi-granularity testing criteria
based on a more detailed analysis of neuron activation values. It defines an
activation interval that is divided into $k$ equal sections, and the coverage
of the test set is estimated based on its ability to cover these sections.
Similarly, recent studies explored the computational behaviour of DNNs by
looking at each neuron’s contribution to the model’s decision-making. For
instance, DeepImportance (Gerasimou et al., 2020) leverages an explanation
method (Bach et al., 2015) to evaluate how prominent each neuron and its
connections are in the model’s final prediction through relevance propagation
techniques.

Concerns about the DNN stability to adversarial attacks were also investigated
(Dong et al., 2018; Gopinath et al., 2017) with approaches proposing solutions
to optimize the DNN performance in response to malicious data perturbations.
DeepHunter (Xie et al., 2019) proposes a data augmentation strategy as a
possible solution. Another line of research investigated more general
correctness properties related to data distributions. A general framework for
automated verification of safety DNN properties is proposed in (Huang et al.,
2017). The authors assess the safety at an individual decision level in terms
of invariance of the classification within a specific region of a data point,
enabling the propagation of this analysis layer by layer, thus allowing
greater scalability. Similarly, SafeML (Aslansefat et al., 2020), like LSC
(Kim et al., 2019), addresses the distribution shift problem by exploiting
statistical distance measures. The technique compares training datasets versus
the inference data input to assess the DNN robustness independently of the
training approach itself.

Most developed testing techniques extend existing software testing techniques
via simple adaptations to DNN. However, none of them investigated how
knowledge transfer and knowledge generalisation (Urolagin et al., 2011) can be
used to inform test adequacy analysis. In contrast, DeepKnowledge drives the
identification of transfer knowledge neurons based on knowledge change and
diversity.

##  7\. Conclusions and Future Work

DeepKnowledge targets DNN testing by quantifying the knowledge diversity of a
test set. Our approach leverages the generalisation property of DNNs to
analyze the knowledge transfer capability of a DNN under domain shift. Thus,
it identifies a finite set of neurons responsible for knowledge generalisation
and uses this information to define the Transfer Knowledge Coverage (TKC)
adequacy criterion for assessing how well the test set exercises different DNN
behaviours. Our empirical evaluation shows the effectiveness of DeepKnowledge
in improving DNN’s accuracy when used to guide the selection of inputs to
augment the training dataset. Our future work involves (i) extending
DeepKnowledge and the TKC test adequacy criterion to support object detection
models (e.g., YOLO (Redmon et al., 2016)); (ii) automating data augmentation
using DeepKnowledge to reduce the cost of data creation and labelling; and
(iii) adapting DeepKnowledge to support model pruning to improve the DNN
robustness and accuracy.

## References

  * Aslansefat et al. (2020) Aslansefat, K., Sorokos, I., Whiting, D., Tavakoli Kolagari, R., Papadopoulos, Y., 2020.  Safeml: safety monitoring of machine learning classifiers through statistical difference measures, in: International Symposium on Model-Based Safety and Assessment, Springer. pp. 197–211.
  * Bach et al. (2015) Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W., 2015\.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one 10, e0130140.
  * Banks et al. (2018) Banks, V.A., Plant, K.L., Stanton, N.A., 2018.  Driver error or designer error: Using the perceptual cycle model to explore the circumstances surrounding the fatal tesla crash on 7th may 2016.  Safety science 108, 278–285.
  * Barr et al. (2014) Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S., 2014.  The oracle problem in software testing: A survey.  IEEE transactions on software engineering 41, 507–525.
  * Bojarski et al. (2016) Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., et al., 2016\.  End to end learning for self-driving cars.
  * Burton et al. (2017) Burton, S., Gauerhof, L., Heinzemann, C., 2017.  Making the case for safety of machine learning in highly automated driving, in: International Conference on Computer Safety, Reliability, and Security, Springer. pp. 5–16.
  * Cohen et al. (2017) Cohen, G., Afshar, S., Tapson, J., Van Schaik, A., 2017\.  Emnist: Extending mnist to handwritten letters, in: 2017 international joint conference on neural networks (IJCNN), IEEE. pp. 2921–2926.
  * Deng (2012) Deng, L., 2012.  The mnist database of handwritten digit images for machine learning research.  IEEE Signal Processing Magazine 29, 141–142.
  * Deng and Karam (2020) Deng, Y., Karam, L.J., 2020\.  Universal adversarial attack via enhanced projected gradient descent, in: 2020 IEEE International Conference on Image Processing (ICIP), IEEE. pp. 1241–1245.
  * Dong et al. (2018) Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., Li, J., 2018.  Boosting adversarial attacks with momentum, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185–9193.
  * Eniser et al. (2019) Eniser, H.F., Gerasimou, S., Sen, A., 2019.  Deepfault: Fault localization for deep neural networks, in: International Conference on Fundamental Approaches to Software Engineering, Springer. pp. 171–191.
  * Erhan et al. (2009) Erhan, D., Bengio, Y., Courville, A., Vincent, P., 2009\.  Visualizing higher-layer features of a deep network.  University of Montreal 1341, 1\.
  * Feng et al. (2020) Feng, Y., Shi, Q., Gao, X., Wan, J., Fang, C., Chen, Z., 2020.  Deepgini: prioritizing massive tests to enhance the robustness of deep neural networks, in: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 177–188.
  * Gal and Ghahramani (2016) Gal, Y., Ghahramani, Z., 2016\.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: international conference on machine learning, PMLR. pp. 1050–1059.
  * Gerasimou et al. (2020) Gerasimou, S., Eniser, H.F., Sen, A., Cakan, A., 2020.  Importance-driven deep learning system testing, in: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), IEEE. pp. 702–713.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., Courville, A., 2016.  Deep Learning.  MIT Press.  http://www.deeplearningbook.org.
  * Goodfellow and Papernot (2017) Goodfellow, I., Papernot, N., 2017\.  The challenge of verification and testing of machine learning.
  * Goodfellow et al. (2020) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2020\.  Generative adversarial networks.  Communications of the ACM 63, 139–144.
  * Goodfellow et al. (2014) Goodfellow, I.J., Shlens, J., Szegedy, C., 2014.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572 .
  * Gopinath et al. (2017) Gopinath, D., Katz, G., Pasareanu, C.S., Barrett, C., 2017\.  DeepSafe: A data-driven approach for checking adversarial robustness in neural networks.  arXiv preprint arXiv:1710.00486 .
  * Gu et al. (2018) Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al., 2018.  Recent advances in convolutional neural networks.  Pattern Recognition 77, 354–377.
  * Gulli and Pal (2017) Gulli, A., Pal, S., 2017.  Deep learning with Keras.  Packt Publishing Ltd.
  * Handelman et al. (2019) Handelman, G.S., Kok, H.K., Chandra, R.V., Razavi, A.H., Huang, S., Brooks, M., Lee, M.J., Asadi, H., 2019\.  Peering into the black box of artificial intelligence: evaluation metrics of machine learning methods.  American Journal of Roentgenology 212, 38–43.
  * Hendrycks et al. (2021) Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al., 2021\.  The many faces of robustness: A critical analysis of out-of-distribution generalization, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340–8349.
  * Hinton et al. (2012) Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., et al., 2012.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  IEEE Signal Processing Magazine 29, 82–97.
  * Houben et al. (2013) Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., Igel, C., 2013.  Detection of traffic signs in real-world images: The german traffic sign detection benchmark, in: The 2013 international joint conference on neural networks (IJCNN), Ieee. pp. 1–8.
  * Huang et al. (2020) Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., Yi, X., 2020.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.  Computer Science Review 37, 100270\.
  * Huang et al. (2017) Huang, X., Kwiatkowska, M., Wang, S., Wu, M., 2017.  Safety verification of deep neural networks, in: Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, Springer. pp. 3–29.
  * Hunt and Thomas (2004) Hunt, A., Thomas, D., 2004.  Pragmatic unit testing in c# with nunit.  The Pragmatic Programmers.
  * Joyce (2011) Joyce, J.M., 2011.  Kullback-leibler divergence, in: International encyclopedia of statistical science. Springer, pp. 720–722.
  * Julian et al. (2016) Julian, K.D., Lopez, J., Brush, J.S., Owen, M.P., Kochenderfer, M.J., 2016.  Policy compression for aircraft collision avoidance systems, in: IEEE Digital Avionics Systems Conference (DASC), pp. 1–10.
  * Kawaguchi et al. (2017) Kawaguchi, K., Kaelbling, L.P., Bengio, Y., 2017.  Generalization in deep learning.  arXiv preprint arXiv:1710.05468 .
  * Kim et al. (2019) Kim, J., Feldt, R., Yoo, S., 2019.  Guiding deep learning system testing using surprise adequacy, in: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE. pp. 1039–1049.
  * Kim et al. (2023) Kim, J., Humbatova, N., Jahangirova, G., Tonella, P., Yoo, S., 2023.  Repairing dnn architecture: Are we there yet?, in: 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), IEEE. pp. 234–245.
  * Krizhevsky and Hinton (2010) Krizhevsky, A., Hinton, G., 2010\.  Convolutional deep belief networks on cifar-10.  Unpublished manuscript 40, 1–9.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al., 2009\.  Learning multiple layers of features from tiny images .
  * Kruse et al. (2022) Kruse, R., Mostaghim, S., Borgelt, C., Braune, C., Steinbrecher, M., 2022.  Multi-layer perceptrons, in: Computational Intelligence. Springer, pp. 53–124.
  * Kurakin et al. (2016) Kurakin, A., Goodfellow, I., Bengio, S., 2016.  Adversarial examples in the physical world.  arXiv preprint arXiv:1607.02533 .
  * LeCun (1998) LeCun, Y., 1998.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/ .
  * Lecun et al. (2015) Lecun, Y., Bengio, Y., Hinton, G., 2015.  Deep learning.  Nature 521, 436–444.
  * Lenz et al. (2015) Lenz, I., Lee, H., Saxena, A., 2015.  Deep learning for detecting robotic grasps.  The International Journal of Robotics Research 34, 705–724.
  * Likas et al. (2003) Likas, A., Vlassis, N., Verbeek, J.J., 2003.  The global k-means clustering algorithm.  Pattern recognition 36, 451–461.
  * Litjens et al. (2017) Litjens, G., Kooi, T., Bejnordi, B.E., et al., 2017.  A survey on deep learning in medical image analysis.  Medical Image Analysis 42, 60–88.
  * Liu et al. (2019) Liu, Y., Mao, S., Mei, X., Yang, T., Zhao, X., 2019\.  Sensitivity of adversarial perturbation in fast gradient sign method, in: 2019 IEEE symposium series on computational intelligence (SSCI), IEEE. pp. 433–436.
  * Long et al. (2017) Long, Y., Liu, L., Shao, L., Shen, F., Ding, G., Han, J., 2017.  From zero-shot learning to conventional supervised classification: Unseen visual data synthesis, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1627–1636.
  * Ma et al. (2018a) Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al., 2018a.  Deepgauge: Multi-granularity testing criteria for deep learning systems, in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 120–131.
  * Ma et al. (2018b) Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al., 2018b.  DeepGauge: Multi-granularity testing criteria for deep learning systems, in: IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 120–131.
  * Ma et al. (2021) Ma, W., Papadakis, M., Tsakmalis, A., Cordy, M., Traon, Y.L., 2021.  Test selection for deep learning systems.  ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 1–22.
  * McCoy et al. (2019) McCoy, R.T., Min, J., Linzen, T., 2019.  Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance.  arXiv preprint arXiv:1911.02969 .
  * Menéndez et al. (1997) Menéndez, M., Pardo, J., Pardo, L., Pardo, M., 1997\.  The jensen-shannon divergence.  Journal of the Franklin Institute 334, 307–318.
  * Meszaros (2007) Meszaros, G., 2007.  xUnit test patterns: Refactoring test code.  Pearson Education.
  * Mikolov et al. (2010) Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., Khudanpur, S., 2010.  Recurrent neural network based language model., in: Interspeech, Makuhari. pp. 1045–1048.
  * Montavon et al. (2017) Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R., 2017.  Explaining nonlinear classification decisions with deep taylor decomposition.  Pattern Recognition 65, 211–222.
  * Netzer et al. (2011) Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y., 2011.  Reading digits in natural images with unsupervised feature learning .
  * Neyshabur et al. (2017) Neyshabur, B., Bhojanapalli, S., McAllester, D., Srebro, N., 2017\.  Exploring generalization in deep learning.  Advances in neural information processing systems 30\.
  * Papernot et al. (2016a) Papernot, N., Goodfellow, I., Sheatsley, R., Feinman, R., McDaniel, P., et al., 2016a.  cleverhans v2. 0.0: an adversarial machine learning library.  arXiv preprint arXiv:1610.00768 10\.
  * Papernot et al. (2016b) Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., et al., 2016b.  The limitations of deep learning in adversarial settings, in: International Symposium on Security and Privacy (S&P), pp. 372–387.
  * Pei et al. (2017) Pei, K., Cao, Y., Yang, J., Jana, S., 2017.  Deepxplore: Automated whitebox testing of deep learning systems, in: proceedings of the 26th Symposium on Operating Systems Principles, pp. 1–18.
  * Petke et al. (2015) Petke, J., Cohen, M.B., Harman, M., Yoo, S., 2015\.  Practical combinatorial interaction testing: Empirical findings on efficiency and early fault detection.  IEEE Transactions on Software Engineering 41, 901–924.
  * Pezze and Young (2008) Pezze, M., Young, M., 2008.  Software testing and analysis: process, principles, and techniques.  John Wiley & Sons.
  * Redmon et al. (2016) Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016\.  You only look once: Unified, real-time object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788.
  * Reid (2010) Reid, M., 2010.  Generalization Bounds. Springer US, Boston, MA.  pp. 447–454.  URL: https://doi.org/10.1007/978-0-387-30164-8_328, doi:10.1007/978-0-387-30164-8_328.
  * Rethmeier et al. (2020) Rethmeier, N., Saxena, V.K., Augenstein, I., 2020.  Tx-ray: Quantifying and explaining model-knowledge transfer in (un-) supervised nlp, in: Conference on Uncertainty in Artificial Intelligence, PMLR. pp. 440–449.
  * Riccio et al. (2020) Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., Tonella, P., 2020\.  Testing machine learning based systems: a systematic mapping.  Empirical Software Engineering 25, 5193–5254.
  * Rousseeuw (1987) Rousseeuw, P.J., 1987.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.  Journal of computational and applied mathematics 20, 53–65.
  * Runeson (2006) Runeson, P., 2006.  A survey of unit testing practices.  IEEE software 23, 22–29.
  * Singh et al. (2019) Singh, G., Gehr, T., Püschel, M., Vechev, M., 2019\.  An abstract domain for certifying neural networks.  Proceedings of the ACM on Programming Languages 3, 41.
  * Sohn et al. (2023) Sohn, J., Kang, S., Yoo, S., 2023.  Arachne: Search-based repair of deep neural networks.  ACM Transactions on Software Engineering and Methodology 32, 1–26.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014.  Dropout: A simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research 15, 1929–1958.
  * Sun et al. (2018) Sun, Y., Huang, X., Kroening, D., Sharp, J., Hill, M., Ashmore, R., 2018\.  Testing deep neural networks.  arXiv preprint arXiv:1803.04792 .
  * Sun et al. (2020) Sun, Z., Zhang, J.M., Harman, M., Papadakis, M., Zhang, L., 2020.  Automatic testing and improvement of machine translation, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 974–985.
  * Sutskever et al. (2014) Sutskever, I., Vinyals, O., Le, Q.V., 2014.  Sequence to sequence learning with neural networks, in: International Conference on Neural Information Processing Systems, pp. 3104–3112.
  * Targ et al. (2016) Targ, S., Almeida, D., Lyman, K., 2016.  Resnet in resnet: Generalizing residual architectures.  arXiv preprint arXiv:1603.08029 .
  * Tian et al. (2018) Tian, Y., Pei, K., Jana, S., Ray, B., 2018.  Deeptest: Automated testing of deep-neural-network-driven autonomous cars, in: Proceedings of the 40th international conference on software engineering, pp. 303–314.
  * Urolagin et al. (2011) Urolagin, S., KV, P., Reddy, N., 2011.  Generalization capability of artificial neural network incorporated with pruning method, in: International Conference on Advanced Computing, Networking and Security, Springer. pp. 171–178.
  * Vallender (1974) Vallender, S., 1974.  Calculation of the wasserstein distance between probability distributions on the line.  Theory of Probability & Its Applications 18, 784–786.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017\.  Attention is all you need.  Advances in neural information processing systems 30\.
  * Wang et al. (2018) Wang, H., Keskar, N.S., Xiong, C., Socher, R., 2018\.  Identifying generalization properties in neural networks.  arXiv preprint arXiv:1809.07402 .
  * Xiao et al. (2017) Xiao, H., Rasul, K., Vollgraf, R., 2017.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747 .
  * Xie et al. (2019) Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., See, S., 2019.  Deephunter: a coverage-guided fuzz testing framework for deep neural networks, in: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 146–157.
  * Xu et al. (2020) Xu, H., Ma, Y., Liu, H.C., Deb, D., Liu, H., Tang, J.L., Jain, A.K., 2020\.  Adversarial attacks and defenses in images, graphs and text: A review.  International Journal of Automation and Computing 17, 151–178.
  * Yang et al. (2021) Yang, J., Zhou, K., Li, Y., Liu, Z., 2021.  Generalized out-of-distribution detection: A survey.  arXiv preprint arXiv:2110.11334 .
  * Zhang et al. (2019a) Zhang, C.W., Yang, M.Y., Zeng, H.J., Wen, J.P., 2019a.  Pedestrian detection based on improved lenet-5 convolutional neural network.  Journal of Algorithms & Computational Technology 13, 1748302619873601.
  * Zhang and Li (2020) Zhang, J., Li, J., 2020.  Testing and verification of neural-network-based safety-critical control software: A systematic literature review.  Information and Software Technology 123, 106296.
  * Zhang et al. (2019b) Zhang, J.M., Harman, M., Ma, L., Liu, Y., 2019b.  Machine learning testing: Survey, landscapes and horizons.  arXiv preprint arXiv:1906.10742 .
  * Zhang et al. (2020) Zhang, J.M., Harman, M., Ma, L., Liu, Y., 2020.  Machine learning testing: Survey, landscapes and horizons.  IEEE Transactions on Software Engineering .
  * Zhou and Sun (2019) Zhou, Z.Q., Sun, L., 2019.  Metamorphic testing of driverless cars.  Communications of the ACM 62, 61–67.