TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep
  Neural Networks
Successful deployment of Deep Neural Networks (DNNs) requires their
validation with an adequate test set to ensure a sufficient degree of
confidence in test outcomes. Although well-established test adequacy assessment
techniques have been proposed for DNNs, we still need to investigate their
application within a comprehensive methodology for accurately predicting the
fault detection ability of test sets and thus assessing their adequacy. In this
paper, we propose and evaluate TEASMA, a comprehensive and practical
methodology designed to accurately assess the adequacy of test sets for DNNs.
In practice, TEASMA allows engineers to decide whether they can trust
high-accuracy test results and thus validate the DNN before its deployment.
Based on a DNN model's training set, TEASMA provides a procedure to build
accurate DNN-specific prediction models of the Fault Detection Rate (FDR) of a
test set using an existing adequacy metric, thus enabling its assessment. We
evaluated TEASMA with four state-of-the-art test adequacy metrics:
Distance-based Surprise Coverage (DSC), Likelihood-based Surprise Coverage
(LSC), Input Distribution Coverage (IDC), and Mutation Score (MS). Our
extensive empirical evaluation across multiple DNN models and input sets such
as ImageNet, reveals a strong linear correlation between the predicted and
actual FDR values derived from MS, DSC, and IDC, with minimum R^2 values of
0.94 for MS and 0.90 for DSC and IDC. Furthermore, a low average Root Mean
Square Error (RMSE) of 9% between actual and predicted FDR values across all
subjects, when relying on regression analysis and MS, demonstrates the latter's
superior accuracy when compared to DSC and IDC, with RMSE values of 0.17 and
0.18, respectively. Overall, these results suggest that TEASMA provides a
reliable basis for confidently deciding whether to trust test results for DNN
models.
Amin Abbasishahkoo, Mahboubeh Dadkhah, Lionel Briand, Dayi Lin
TEASMA: A Practical Approach for the Test Assessment of Deep Neural Networks using Mutation Analysis

###### Abstract.

Successful deployment of Deep Neural Networks (DNNs), particularly in safety-critical systems, requires their validation with an adequate test set to ensure a sufficient degree of confidence in test outcomes. Mutation analysis, one of the main techniques for measuring test adequacy in traditional software, has been adapted to DNNs in recent years. This technique is based on generating mutants that aim to be representative of actual faults and thus can be used for test adequacy assessment. In this paper, we investigate for the first time whether mutation operators that directly modify the trained DNN model (i.e., post-training) can be used for reliably assessing the test inputs of DNNs. We propose and evaluate _TEASMA_, an approach based on post-training mutation for assessing the adequacy of DNN's test sets. In practice, _TEASMA_ allows engineers to decide whether they will be able to trust test results and thus validate the DNN before its deployment. Based on a DNN model's training set, _TEASMA_ provides a methodology to build accurate prediction models of the Fault Detection Rate (FDR) of a test set from its mutation score, thus enabling its assessment. Our large empirical evaluation, across multiple DNN models, shows that predicted FDR values have a strong linear correlation (\(R^{2}\geq 0.94\)) with actual values. Consequently, empirical evidence suggests that _TEASMA_ provides a reliable basis for confidently deciding whether to trust test results or improve the test set.

Deep Neural Network, Test Assessment, Mutation Analysis +
Footnote †: journal: T

## 1. Introduction

Effective testing of Deep Neural Networks (DNN) is essential to ensure they can be trusted, particularly as they are increasingly deployed in safety-critical systems. However, to rely on testing outcomes, it is crucial to assess the adequacy of the test set with which the DNN is to be validated. In the context of DNN testing, several techniques have been proposed for test adequacy assessment and are mostly relying on coverage criteria (Louis et al., 2017; Li et al., 2018; Li et al., 2019; Li et al., 2019; Li et al., 2019), inspired by practice in testing traditional software. However, these coverage criteria have shown no significant correlation with the number of mispredicted inputs or detected faults in a test set (Li et al., 2019; Li et al., 2019; Li et al., 2019), thus putting into question whether they are reliable criteria for test assessment for DNN models.

Mutation analysis, one of the most used techniques for test assessment in traditional software (Dadkhah and Kast, 2018; Li et al., 2019), has been adapted to DNN models in recent years. DNN-specific Mutation Operators (MO) have been proposed (Li et al., 2019; Li et al., 2019; Li et al., 2019; Li et al., 2019) since traditional MOs are not directly applicable to DNNs (Li et al., 2019). Indeed, developing a DNN model entails applying a training process relying on both a training program and a training set. Therefore, researchers have proposed MOs that fall into two primary categories: post-training MOs (Li et al., 2019; Li et al., 2019), which mutate the DNN model obtained after the training process, and pre-training MOs (Li et al., 2019; Li et al., 2019), which mutate either the training program or the training set and use these mutated versions in the training process to generate mutated models.

Post-training MOs were introduced by mutation testing frameworks for DNN models: MuNN (Li et al., 2019) and DeepMutation (Li et al., 2019). Additionally, DeepMutation introduced a set of pre-training MOs. However, DeepMutation's authors later introduced DeepMutation++ (Li et al., 2019), a mutation testing tool exclusively implementing the post-training MOs. Hembatova _et al._(Hembatova et al., 2019) further expanded the concept of pre-training MOs by investigating real faults in DNN models (Li et al., 2019) and proposing corresponding, more realistic pre-training MOs. They also introduced a statistical approach to determine whether a mutant generated by pre-training MOs is killed (Li et al., 2019). However, this approach requires generating many instances of the original DNN model and each mutated model by repeating the training process many times for each model. Such an expensive procedure makes the cost of mutation testing impractical in many situations.

The use of mutation analysis in traditional software is motivated by the strong association between mutants and faults reported in the literature (Li et al., 2019; Li et al., 2019; Li et al., 2019). Regarding DNNs, the ability of MOs to generate killable and non-trivial mutants has been thoroughly investigated (Li et al., 2019; Li et al., 2019). However, the association between the mutants and DNN faults, and thus the ability to assess a test set based on its Mutation Score (MS), has never been studied. In this paper, we address this gap by investigating such an association for mutants generated from post-training MOs, motivated by their much lower cost compared to pre-training MOs. For this purpose we investigate, across a number of models and input subsets from training sets, the
relationship between MS and Fault Detection Rates (FDR) considering various MS definitions proposed in the literature (Beng et al., 2017; Chen et al., 2018). We perform our experiments using post-training MOs implemented by DeepMutation++, a State-of-The-Art (SoTA) mutation testing tool for Feed-Forward Neural Networks (FNNs). We show that there is a strong correlation between FDR and MS indicating that mutants generated by post-training MOs are highly associated with faults.

Building on correlation results, in this paper, we propose _TEASMA_, a practical approach to assess the adequacy of test sets for DNN models using mutation analysis based on post-training MOs. Following _TEASMA_, engineers can develop a specific FDR prediction model, along with Prediction Intervals (PI), for each DNN model based on its training set. This prediction model can be used by engineers to predict a test set's FDR based on its MS and decide, also considering the prediction's PI, whether a test set is adequate for validation in their context. Developing _TEASMA_ is further motivated by the costly and time-consuming process of preparing large test sets, whose labeling usually requires domain experts. Using _TEASMA_, a test set is assessed before labeling and execution, solely based on the training set used for training the DNN model. Test inputs can then be labeled and executed once the test set is deemed adequate, thus preventing the waste of testing resources. Further, if sufficiently fast and when MS is shown to accurately predict FDR, _TEASMA_ could be used to automatically guide test selection.

We evaluated _TEASMA_ on five widely used DNN models and four image recognition input sets. We compared, for a large number of test subsets, their predicted FDR with the actual one and we could observe, in all cases, a very high linear correlation (\(R^{2}\geq 0.94\) ) between the two, with a slope close to 1 for the regression line. We also show that our prediction models have narrow PIs, implying that the predicted FDR can confidently be used as a basis for deciding about the adequacy of a test set. The key contributions of our paper are summarized as follows:

* An empirical analysis of SoTA post-training MOs showing that the MS calculated based on these mutants is strongly correlated with FDR.
* _TEASMA_, a practical approach utilizing post-training MOs for assessing a test set before validation and deployment of a DNN, which is based on regression analysis using the training set with the objective to predict the FDR of test sets as a basis for assessing their adequacy.
* An empirical evaluation of _TEASMA_ on five widely used DNN models and four image input sets, demonstrating its ability to accurately predict a test set's FDR based on its MS, relying on a prediction model built during training.

The remainder of this paper is structured as follows: Section 2 provides background and discussions of the related work directly relevant to our research questions. Section 3 presents our proposed approach for test adequacy assessment. Section 4 describes the experiments we performed to assess our approach. Section 5 presents and discusses the results for each research question, and Section 6 concludes the paper.

## 2. Background and Related Work

This section is divided into four parts. In the first part, we briefly describe the proposed mutant generation processes for DNNs. The second part summarizes different ways of identifying killed mutants and calculating MS. In the third part, we discuss the association between mutants and faults and point out the challenge of investigating such association in DNNs. The last part describes a SoTA approach that can be leveraged to overcome this challenge and perform our experiments.

### Mutant Generation

Applying mutation testing to DNN-based systems requires defining specific operators for DNN models (Krizhevsky et al., 2014). The proposed MOs can be broadly classified into two primary categories, namely pre-training (Krizhevsky et al., 2014; Chen et al., 2018; Chen et al., 2018) and post-training (Beng et al., 2017; Chen et al., 2018; Chen et al., 2018) operators, as illustrated in Figure 1. Pre-training MOs, also called source-level MOs, slightly modify the original version of either the program or the input set that has been used to train the original DNN model under test. These modified versions are then used by the training process to generate mutated models (i.e., mutants). Mutation testing using pre-training MOs is computationally expensive since it entails repeating the training process for generating each mutant. On the other hand, post-training MOs, also referred to as model-level MOs, modify the already trained original model, eliminating the need for additional training to generate mutants.

Ma _et al._(Ma et al., 2018) proposed DeepMutation, the first mutation testing framework for DNNs that includes both pre- and post-training MOs. Pre-training operators are classified into data-level and program-level operators whereas post-training operators are classified into weight-level, neuron-level, and layer-level operators, depending on the specific component of the DNN model that they modify. The DeepMutation++ tool (Beng et al., 2017) was later introduced by the same authors implementing only the post-training operators. Shen _et al._(Shen et al., 2018) proposed the MuNN framework, including five post-training operators that modify neurons, activation function, bias values, and weight values of a DNN model. Since post-training MOs directly modify the DNN model, not all of them are applicable to any DNN model. For example, applying the Layer Deactivation (\(LD\)) operator of DeepMutation (Ma et al., 2018) on some layers may break the model's structure. Therefore, DeepMutation restricts the application of this operator to only those layers whose input and output shapes are consistent, if the model has such layers.

Humbatova _et al._(Humbatova et al., 2018) proposed DeepCrime, a mutation testing tool that includes 24 pre-training mutation operators inspired by real faults in deep learning systems (Krizhevsky et al., 2014; Chen et al., 2018; Chen et al., 2018). The main goal was to obtain more realistic mutation operators. They also proposed a statistical approach (Krizhevsky et al., 2014) for identifying killed mutants that requires multiple instances of each mutant. Therefore, DeepCrime repeats the training process on the original and modified versions of the

Figure 1. Pre-training and post-training mutation operators
training set and program many times, resulting in many instances of the original model and each mutant. This further increases the cost of mutation testing, requiring not only to repeat the training process and generating more mutants but also executing each test set on all mutant instances.

Most of the MOs proposed for DNNs (Humbs, 2017; Zhang et al., 2018; Zhang et al., 2018) have parameters that define the scale of modifications by that MO and need to be configured before their application. For example, operators that modify training sets could be configured by defining the percentage of inputs to be modified. Therefore, each MO can be applied to the DNN with multiple configurations, some of them resulting in generating mutants with undesirable characteristics: equivalent, trivial, and redundant. It is essential to identify and filter such mutants since they not only increase the overall execution time of mutation analysis but may also affect the mutation score. Different approaches have been proposed to address this issue (Humbs, 2017; Zhang et al., 2018; Zhang et al., 2018). In the DeepMutation++ tool (Humbs, 2017), mutants with prediction accuracy lower than a specified threshold are eliminated. Then, for each input set, mutants with high error rates (i.e., killed by most of the inputs), are filtered out to prevent inflating the MS with easy-to-kill mutants. The DeepMutation++ tool (Humbs, 2017) sets the accuracy threshold to be 90% of the original model's accuracy, and the error rate threshold to be 20% of the input set size.

The high computational cost of mutation testing is its main drawback in practice. This issue is even more critical in the case of DNNs considering large input sets, which is usual in practice. In this context, the substantial computational overhead of mutant generation using pre-training MOs, due to the need to perform training on each mutant, is a significant drawback when compared with post-training MOs. This is particularly true for DeepCrime which involves generating many instances of each mutant, even if we only consider the most effective configurations (Humbs, 2017) for each MO to create non-equivalent and non-trivial mutants 1. Therefore, it is essential to investigate if the more practical post-training MOs are also able to generate mutants associated with faults that can support our objective, assess the adequacy of test sets. In this paper, we perform such an investigation and propose _TEASMA_, a practical approach for test set adequacy assessment in DNNs based on mutation analysis.

Footnote 1: To give a concrete idea, in our experiments, DeepCrime took about 20 to 30 times the amount of computation time of DeepMutation++. This ratio would of course increase with larger models and input sets.

### Mutation Score Calculation

Similar to traditional software, killed DNN mutants can be identified by comparing the output of a mutant \(M_{i}\) with the original model \(N\) and the corresponding MS can be calculated as:

\[StandardMS(T,M)=\frac{\sum_{M_{i}\in M}killed(T,M_{i})}{|M|} \tag{1}\]

where \(M\) is the set of mutants, \(M_{i}\in M\) is \(killed\) by test set \(T\) (\(killed(T,M_{i})=1\)) if there is at least one input \(t\in T\) that is correctly classified by \(N\) and is not correctly classified by mutant \(M_{i}\). DeepMutation (Zhang et al., 2018), however, calculates MS in a different way. For a classification problem with \(k\) classes \(C=\{C_{1},...,C_{k}\}\), a test input \(t\in T\) kills class \(C_{j}\in C\) of mutant \(M_{i}\in M\) if \(t\) is correctly classified as \(C_{j}\) by the original model \(N\) and is not classified as \(C_{j}\) by mutant \(M_{i}\). Accordingly, MS is calculated for a test set \(T\) as:

\[DeepMutationMS(T,M)=\frac{\sum_{M_{i}\in M}killedClasses(T,M_{i})}{|M|\times|C|} \tag{2}\]

where \(killedClasses(T,M_{i})\) is the number of classes of mutant \(M_{i}\) killed by inputs in \(T\)(Zhang et al., 2018). The difference when calculating MS for the same test set using \(StandardMS\) based on Equation (1) and \(DeepMutationMS\) based on Equation (2) can be very large. For instance, consider at test \(T\)s from the input set of handwritten digits (MNIST). Each input \(t\in TS\) is an image belonging to a class \(C_{j}\). Let the test set \(TS\) be represented by its correct input labels \(label(TS)=\{3,5,3,7\}\), including two different images from the same class representing digit 3 but also two images from classes 5 and 7. Further, consider that all images are correctly predicted by \(N\). If we have a set of mutants \(M=\{M_{1},M_{2}\}\), where the first two test inputs of \(TS\) are mispredicted by \(M_{1}\) and all test inputs are mispredicted by \(M_{2}\), then \(DeepMutationMS(TS,M)=\frac{2+3}{2\times 10}=0.25\) while \(StandardMS(TS,M)=1\). The score calculated by \(StandardMS\) for a test set \(T\) is higher since killing a mutant \(M_{i}\) with \(T\) requires only one test input \(t\in T\) to be mispredicted by \(M_{i}\).

Hu _et al._(Humbs, 2017) defined a killing score metric for each test input \(t\) as the proportion of mutants in \(M\) whose output is different from \(N\).

\[KillingScore(t,M,N)=\frac{|M_{i}|M_{i}\in M\wedge N(t)\neq M_{i}(t)|}{|M|} \tag{3}\]

Humbatova _et al._(Humbs, 2017) used this metric to calculate the MS of a test set \(T\) on mutants generated by DeepMutation++ (Humbs, 2017):

\[KillingScoreBasedMS(T,M)=\frac{\sum_{t\in T}KillingScore(t,M,N)}{|T|} \tag{4}\]

However, calculating MS as the average \(KillingScore\) of test inputs in a test set may not indicate the ability of the entire test set to kill mutants. A test set cannot achieve such a high MS unless many of its test inputs are able to kill most of the mutants. This is visible in results reported by Humbatova _et al._(Humbs, 2017) where the MS calculated by \(KillingScoreBasedMS\) for entire test sets ranges from 0.059 to 0.33. Humbatova _et al._(Humbs, 2017) further proposed a statistical approach for identifying killed mutants when applying pre-training MOs in DeepCrime where the training process is repeated \(n\) times, leading to \(n\) instances of the original model and mutants being generated (Section 2.1). This further increases the cost of mutant generation using pre-training MOs and makes them computationally challenging for practical use in our context. Note that this statistical approach for identifying killed mutants is not applicable to post-training MOs, since they directly modify an already trained DNN model.

Mutation analysis involves identifying test sets with high MS based on the key assumption that MS captures a test set's capability to detect faults. In DNNs, however, various MS formulas lead to different results and which test sets lead to high MS varies according to which formula is used. It is evident that more test sets can achieve a high MS using \(StandardMS\) (Equation (1)) than when using \(DeepMutationMS\) (Equation (2)). We also expect that not many test sets can achieve a high MS using \(KillingScoreBasedMS\)
(Equation (4)). Therefore, because we cannot a priori determine which MS formula satisfies our assumption to the largest extent, it is essential to consider all of them when experimenting with mutation analysis for DNNs.

### Mutants and Faults

In traditional software, MOs are defined to introduce simple syntactical changes representing real mistakes often made by programmers into the source code of a program. Similarly, DeepCrime defined a set of pre-training MOs based on an analysis of real faults (Deng et al., 2015; Wang et al., 2016; Wang et al., 2017) related to the training set or program used to train a DNN model. However, the high cost of generating mutants using pre-training MOs hinders their application in practice. Post-training MOs, in contrast, apply modifications directly to the original model but have been criticized for performing modifications that are not realistic (Wang et al., 2016). Nevertheless, for mutation analysis to be effective in assessing the adequacy of test sets, regardless of how mutation operators are defined, it is important to generate mutants whose corresponding MS is, for a test set, a good predictor of fault detection. Therefore, considering the strong practical advantages of post-training MOs, despite their presumed lack of realism, they should be carefully investigated to build predictors of fault detection before being dismissed as inadequate.

The use of mutation analysis for assessing the quality of test sets in traditional software is supported by the observed association between mutants and real faults, as reported in the literature (Wang et al., 2016; Wang et al., 2017; Wang et al., 2017). This association has been investigated by performing correlation analysis (Wang et al., 2016; Wang et al., 2017; Wang et al., 2017) between the MS of a test set and its capability to detect faults. Regression analysis has also been applied for modeling this relationship to build a prediction model for FDR (Wang et al., 2016; Wang et al., 2017). In DNNs, however, the association between mutants and faults has never been investigated. The main challenge in such investigation is that, in contrast to traditional software where it is possible to identify real faults by isolating the faulty statement that caused a failure, identifying faults in DNNs is not a straightforward task due to the complexity and non-linear nature of the DNNs. However, a recent approach proposed by Aghababaeyan _et al._(Aghababaeyan et al., 2016) for estimating faults is a possible, practical, and automated solution. We describe next how we rely on this approach to investigate the relationship between mutants and faults, and how MS can help predict FDR.

### Fault Estimation

Although the number of mispredicted test inputs is a well-known metric in evaluating DNN testing approaches (Wang et al., 2016; Wang et al., 2017; Wang et al., 2017), it is not adequate in our experiments. Indeed, a single fault within a DNN model can make the model mispredict multiple test inputs. Subsequently, a test set that identifies a considerable number of mispredictions may actually detect only a few underlying faults, whereas another test set with the same number of mispredictions could detect a higher number of distinct faults. Therefore, assessing test sets based on mispredictions and thus investigating the relationship between the number of killed mutants and mispredictions can be misleading. Consequently, we rely on a recent approach introduced by Aghababaeyan _et al._(Aghababaeyan et al., 2016) for estimating faults in a DNN model, that are defined as distinct root causes of mispredictions (Bahdan et al., 2016). They cluster mispredicted inputs that exhibit similar features in three key steps: feature extraction, dimensionality reduction, and density-based clustering. Initially, VGG-16 (Vogtani et al., 2016) is employed to extract the feature matrix from the mispredicted inputs, serving as the foundation for clustering. Subsequently, dimensionality reduction is applied to enhance clustering performance within the inherently high dimensional feature space. The HDBSCAN algorithm (Bahdan et al., 2016) is then utilized to cluster the mispredicted inputs based on the extracted features. In their empirical analysis of the resulting clusters, Aghababaeyan _et al._(Aghababaeyan et al., 2016) have shown that inputs in each cluster are mispredicted due to the same fault in the model and inputs in different clusters are mispredicted due to distinct faults. Therefore, if a test set contains only one of the test inputs in a cluster, then it is able to detect the underlying fault of that cluster.

Suppose we identify a set of \(k\) Clusters of Mispredicted Inputs \(CMI=\{CM_{\text{I}},...,CM_{\text{I}}\}\), we can then assume that each cluster \(CMI_{\text{I}}\) contains inputs that are mispredicted due to the same root cause (i.e., the underlying fault \(F_{i}\)). Therefore, we say that test set \(T\)_detects_ a fault \(F_{i}\in F\) if there is at least one test input \(t\in T\) such that \(t\in CMI_{\text{I}}\). Finally, we calculate the FDR for a test set \(T\) as:

\[FDR(T,F)=\frac{\sum_{F_{i}\in F}detect(T,F_{i})}{|F|} \tag{5}\]

In the next section, we propose a practical solution for test set adequacy assessment using mutation analysis. The objective is to predict the expected FDR for a given test set and DNN, based on mutation and regression analysis performed during training.

## 3. Practical Approach

In this section, we present _TEASMA_, a practical solution for assessing test sets for DNNs using mutation analysis. As mentioned earlier, the basic idea is to rely on mutation and regression analysis based on the training set to accurately predict the FDR of test sets, which can then be used for evaluating their adequacy before proceeding with labeling and execution. In subsequent sections, we will empirically evaluate the accuracy of _TEASMA_ in predicting FDR and its applicability. The process to be followed by engineers in practice is illustrated in Figure 2 and includes three steps:

**a) Calculating MS and FDR for subsets of the training set.** In this step, only post-training MOs are used to generate mutants as described in Section 2.1, since in our application context pre-training MOs are not a practical option. The training set is executed against the original DNN model and mispredicted inputs are clustered, as described in Section 2.4, to identify faults. Then, a large number of subsets of varying sizes are randomly sampled from the training set, and their MS and FDR are calculated to be used in the next step.

**b) Performing regression analysis.** In this step, regression analysis between FDR and MS is performed to obtain the best model for predicting FDR based on MS. The best shape for the regression model is determined empirically, prioritizing simpler linear models when they fare well. But because we can expect non-linear relationships, for reasons explained later, we also consider quadratic and exponential regression models, as well as regression trees when no regression function fits well. Regression trees enable the optimal partition of the MS range into optimal sub-ranges with similar FDR values. Note that the best model shapes may differ across DNNs
because of differences in their architecture and training sets. Regardless of its shape, the regression model must be sufficiently accurate to enable practically useful FDR predictions. This prediction model is then used in the next step to assess the adequacy of test sets.

**c) Assessing test set adequacy.** In this step, given a test set, its MS is calculated using the original DNN model and previously generated mutants. Then, the prediction model obtained from the previous step is used to predict the test set's FDR (\(\widehat{FDR}\)), including PIs. Given a certain percentage of confidence (e.g., 95% in our experiments), such intervals specify the range of potential actual FDR values for test sets achieving a specific MS value and thus provide engineers with valuable insights into the uncertainty of FDR predictions. Depending on the context, one can then decide how conservative should decision-making be by considering the lower or higher bounds of the PI. The \(\widehat{FDR}\) of a test set, if deemed sufficiently accurate based on its PI, can then be used to assess its capability to detect faults and therefore whether, in a given context, it is adequate to validate the DNN model.

We believe that our proposed solution--assuming an accurate FDR prediction model can be built using the training set--is practical. Indeed, we use post-training MOs which are directly applied to the DNN model and do not entail any re-training, thus avoiding the main challenge with pre-training MOs. In practice, such assessment helps guide the selection of test inputs and significantly reduces test costs by only labeling a test set once it is considered adequate. However, since we build the FDR prediction model using the training set and use it to determine the adequacy of a test set, we implicitly assume that the two sets have similar distributions. This is, however, a common assumption for the training of DNNs to be considered adequate.

## 4. Experimental Procedure

Our main goal is to evaluate the accuracy of _TEASMA_ in predicting a test set's FDR using post-training MOs and a regression model built entirely based on the training set. To accomplish this, for each subject, we performed the following high-level steps:

1. Select a large number of input subsets of diverse sizes from the training set (Section 4.3).
2. Generate and select mutants using SoTA mutation operators and mutation testing tools (Section 4.4).
3. Identify distinct faults in the DNN models based on the models' mispredicted inputs in the training set (Section 4.5).
4. Conduct experiments using the mutants, faults, and training input subsets to answer the research questions presented next (Section 4.6).

### Research Questions

Our experiments are designed to answer the following research questions:

**RQ1: Can an accurate regression model be built to explain FDR as a function of MS?**

In this question, we investigate the output of the second step of _TEASMA_ in Section 3. We select a large number of input subsets from the training set, calculate their MS and FDR using the faults estimated in the original model as described in Section 4.5, and investigate, for mutants derived from post-training MOs, if the MS of an input subset is significantly correlated with its FDR. In that case, we resort to regression analysis to model the relationship between MS and FDR and investigate if an accurate regression model can be built to predict a subset's FDR from its MS. If that is the case, the predicted FDR can serve as a good basis on which to decide whether a test set provides sufficient confidence about the validation of the DNN before its deployment. Though such questions have been investigated for traditional software (Brock et al., 2018; Chen et al., 2019), that has not been the case for DNNs.

**RQ2: Can a test set's FDR be accurately predicted from its MS using the regression model built on the training set?**

In this question, we evaluate the last step of _TEASMA_ in Section 3 and investigate if we can accurately predict a test set's FDR using the regression model built in the previous step on the training set. For this purpose, we select a large number of input subsets from the test set and calculate their MS and FDR using the faults identified in the original model. Then, we use the regression model to predict FDR for each subset and compare it with the actual one, to assess whether, as expected, they have a linear relationship with a high correlation and a slope close to 1.

Figure 2. The process of test adequacy assessment with _TEASMA_
### Subjects

We performed our study on different combinations of input sets and models. Since we rely on a SoTA fault estimation approach for DNNs that is tailored for image inputs, we selected a set of widely-used publicly available image input sets including MNIST (Huang et al., 2017), Fashion-MNIST (Huang et al., 2017), Cifar-10 (Chen et al., 2017), and SVHN (Zhuang et al., 2017). Some of these input sets have also been used for evaluating SoTA mutation testing tools and operators (Fashion-MNIST, 2017; Dosovitskiy et al., 2018; Dosovitskiy et al., 2018; Dosovitskiy et al., 2018).

MNIST is a set of images of handwritten digits and Fashion-MNIST contains images of clothes that are associated with fashion and clothing items. Each one of MNIST and Fashion-MNIST represents a 10-class classification problem and consists of 70,000 grayscale images of a standardized 28x28 size. Cifar-10 is another widely-used input set that includes a collection of images from 10 different classes (e.g., cats, dogs, airplanes, cars). SVHN (Zhuang et al., 2017), a set of real-world house numbers collected by Google Street View, can be considered similar to MNIST since each image represents a sequence of digits. The Cifar-10 and SVHN input sets contain 32x32 cropped colored images.

In our experiments, we trained five SoTA DNN models using the above input sets: LeNet5, LeNet4, ResNet20, VGG16, and an 8-layer Convolutional Neural Network (Conv-8). Since not all of the post-training MOs are applicable to all DNN models, as described in Section 2.1, we selected DNN models with different internal architectures. All the feasible combinations of DNN models and input sets we experiment with are referred to as subjects. Overall, we perform our experiment on six subjects including a wide range of diverse image inputs and DNN architectures. Table 1 lists the details of each subject including the size of test and training datasets, the number of epochs used to train the model, the accuracy of the DNN model on the training and test input sets, and the number of identified faults based on the training set. Note that we reused the training and test sets as defined in the original sources (Chen et al., 2017; Huang et al., 2017; Zhuang et al., 2017).

### Sampling Input Subsets

To evaluate _TEASMA_, we select input subsets from the training set, calculate the MS and FDR of each subset, and build a regression model. For this purpose, we require a large number of input subsets that possess diverse MS and FDR values. Therefore, we start sampling input subsets with sizes 100, 300, and 500 and continue with larger or smaller sizes where needed, until the entire spectrum of FDR with minimum and maximum values (\(0\leq FDR\leq 1\)) is covered.For each subset size, we sample 300 subsets with replacements. Since one of the MS formulas we use (Equation (2)) considers the number of images from each class \(C_{i}\) in an input subset, we complement the conventional approach of creating randomly sampled subsets by creating a set of uniformly sampled subsets. Random subsets may contain any number of input images from each class \(C_{i}\) but uniform subsets contain an equal number of images from each class \(C_{i}\). Uniform subsets may achieve higher MS values than random subsets based on Equation (2). Therefore, we perform our experiments on both types of input subsets and empirically determine which one serves our purpose better.

### Generating and filtering Mutants

We use DeepMutation++ (Fashion-MNIST, 2017), the most recent SoTA mutation testing tool for DNNs that implements eight post-training MOs suitable for FNNs. Some of the implemented MOs need to be configured by setting values for their parameters. For example, the Neuron Effect Blocking (NEB) operator is configured by determining the number of neurons it affects. By default, the DeepMutation++ tool configures this operator to sample a predefined ratio of 1% of the total neurons in a DNN model and limits the total number of generated mutants to 50. In our experiments, we used the default configuration for generating mutants. Furthermore, we perform the same procedure proposed by the tool and use its default thresholds to filter out some of the generated mutants. For example, we exclude mutants that are either too similar to the original model or too easy to kill. Specifically, we exclude mutants when they yield more than 90% of the original model's accuracy or mispredict more than 20% of the inputs correctly predicted by the original model, as reported in the original papers (Fashion-MNIST, 2017; Dosovitskiy et al., 2018). Further, in our experiments, we perform an additional step to filter out equivalent mutants since DeepMutation++ does not have a mechanism to do so. To calculate MS precisely, we execute the entire training set against the mutants, and if a mutant cannot be killed by any training input, it is considered equivalent and hence excluded.

### Estimating Faults

In order to evaluate _TEASMA_, we estimate faults in the DNN model using the approach described in Section 2.4 based on the mispredicted inputs of the training set. We execute the entire training set against the original DNN model, extract the mispredicted inputs, and apply the fault identification approach (Chen et al., 2017) on these mispredicted inputs. Some of the subjects used in our experiment include original models that achieve very high accuracy (e.g., 98%). As a result, the number of inputs mispredicted by the original model is a very small percentage of the entire training set and the number of estimated faults is very low. In such cases, we need more faults to be able to investigate the ability of a test set to detect faults. For this purpose, we use snapshot models (Fashion-MNIST, 2017), the intermediate models obtained during the training process of the original DNN model. Once the original model training is completed, we execute the entire training set against all snapshot models and identify mispredicted inputs. The clustering approach (Chen et al., 2017) is then applied to mispredicted inputs to identify all the faults that were observed during the training process.

### Experiments

This section explains how we conduct our experiments to evaluate _TEASMA_ and answer the research questions described in Section 4.1. To answer RQ1, we estimate faults of the DNN model based on the training set as described in Section 4.5, generate mutants using DeepMutation++, and calculate FDR and MS for a large number of diverse input subsets sampled from the training set. Subsequently, we measure the correlation between FDR and MS using the non-parametric Spearman coefficient based on all subsets, since we cannot assume linearity. We also measure the correlation for mutants generated by each MO separately and then attempt to find
subsets of operators leading to higher correlation. Before proceeding to the next step, we compare the highest correlation achieved by subsets of MOs with that obtained with all MOs. If the correlation of the former is higher than the latter, then we only consider mutants generated by the optimal subset with the highest correlation.

Next, using all the training subsets, we perform regression analysis, prioritizing simpler linear models but relying on non-linear models where needed to model the relationships between MS and FDR, and inspect the resulting coefficient of determination (\(R^{2}\)) to measure the goodness of fit. Furthermore, we report two common measures for evaluating the accuracy of predictions, namely the Mean Magnitude of Relative Error (MMRE) and the Root Mean Square Error (RMSE) to assess how accurately we can predict FDR. To obtain more realistic results, we perform a \(K\)-fold cross-validation procedure with \(K=5\) and report the average of the \(R^{2}\), MMRE, and RMSE across all folds. Based on these metrics, we determine for each subject, the best regression model for predicting FDR based on MS. To summarize, the experiment that we conducted using the training set to answer RQ1 included the following steps:

1. Calculate MS and FDR for each input subset
2. Measure the Spearman correlation between FDR and MS
3. Perform regression analysis between FDR and MS using linear and non-linear models (quadratic, exponential, and regression trees)
4. Evaluate regression models by performing \(K\)-fold cross-validation and calculating \(R^{2}\), MMRE, and RMSE to identify the best regression model

To address RQ2, we sample a large number of input subsets from the test set, calculate their MS using previously generated mutants, and leverage the previously built and selected regression model to predict FDR (\(\widehat{FDR}(T)\)). We should note that we determine killed mutants by comparing their output with that of the original model and thus the actual labels of test inputs are not required. To estimate the actual FDR (\(ActualFDR\)) of each test subset, we use the same fault clusters identified based on the training set as described in Section 4.5 and we assign each mispredicted test input to one of the clusters. Since the fault estimation approach (Section 2.4) employs the HDBSCAN algorithm (Brandt, 2002), a density-based hierarchical clustering method where each cluster is characterized by a number of core points, we assign each mispredicted test input to the cluster featuring a core point that is closest to the input. To ensure an accurate calculation of \(ActualFDR\), we only consider in the denominator of the Equation (5) the number of fault clusters that are detectable by the test set i.e., clusters that have at least one mispredicted test input assigned to them. Subsequently, we are able to compare for each test subset, the predicted FDR with the actual one and investigate if the former is closely aligned with the latter and can thus be trusted to make decisions about the test set. Therefore, to answer RQ2, we went through the following steps:

1. Randomly sample a large number of input subsets of diverse sizes from the test set
2. Calculate MS and FDR (\(ActualFDR\)) for each test subset
3. Use previously selected prediction model to compute \(\widehat{FDR}\) based on MS for each test subset
4. Based on all test subsets, measure the linear correlation (\(R^{2}\)) between \(\widehat{FDR}\) and \(ActualFDR\) and check the slope of the regression line

As described in Section 2.2, different methods have been proposed to determine whether a mutant is killed and therefore to calculate MS. To fully address our research questions, we ran a comprehensive analysis by considering all the MS calculation variants described in Section 2.2. As a result, we ran three variations of our experiments that only differ in the way they calculate MS; (E1) \(Deep\)\(Mutation\)\(MS\), (E2) \(Standard\)\(MS\), and (E3) \(Killing\)\(Score\)\(Based\)\(MS\).

Experiments on all subjects took about a week using a cloud computing environment provided by the Digital Research Alliance of Canada (Cowell et al., 2017), on the Cedar cluster with a total of 1352 _NVIDIA P100 Pascal_ GPUs with \(8\) to _16 GB_ of memory.

## 5. Results

In this section, we report results pertaining to our research questions and discuss their practical implications.

### RQ1: Can an accurate regression model be built to explain FDR as a function of MS?

To answer this question, we first investigate the extent of the correlation between MS and FDR. We perform our experiment on a large number of randomly and uniformly sampled input subsets from the training set, also considering different MS calculation variants.The correlation analysis results using Spearman correlation

\begin{table}
\begin{tabular}{|c|c c c|c c|c c|c c|} \hline ID & Input & Training & Test & Model & Epochs & Train & Test & Number of mispredictions & Number of faults \\  & set & set & set & & accuracy & accuracy & (training) & (training) \\ \hline S1 & MNIST & 60,000 & 10,000 & LeNet5 & 12 & 99\% & 98\% & 2462 & 351 \\ \hline S2 & \multirow{2}{*}{Cifar-10} & \multirow{2}{*}{50,000} & \multirow{2}{*}{10,000} & Conv-8 & 20 & 94\% & 85\% & 2507 & 354 \\ \cline{5-10}
coefficients for experiment variants E1 and E2 are shown in Table 2. Given space constraints, we only report results for random subsets as they yielded slightly higher correlations. The best correlation achieved for each subject across experiment variants, given in each column of Table 2, is highlighted in bold font. We did not report the results for experiment variant E3 since all of the correlation values for both random and uniform subsets across all subjects are less than 0.1, indicating a complete absence of correlation. This can be explained by the fact that MS is determined by computing the average \(KillingScore\) (Equation (3)) of all inputs within a subset. Consequently, for subsets of different sizes, MS calculated based on \(KillingScoreBasedMS\) (Equation (4)) falls within a similar range (0 to 0.1). As discussed in Section 2.2, this confirms that calculating the MS of a subset as the average capability of its inputs to kill mutants cannot be used as an indicator of the capability of the entire subset to kill mutants.

E1 shows the strongest correlation across all subjects between MS and FDR (above 0.98 across all subjects). In this experiment, we calculate MS as proposed originally by DeepMutation (Equation (2)). In contrast, for experiment E2, \(R^{2}\) ranges from 0.51 to 0.92, which is still a strong correlation. The method for identifying killed mutants and calculating MS therefore has a substantial impact on the results of mutation analysis. Given the strong correlations achieved in E1 and E2 across all subjects, we conclude that mutants generated by DeepMutation's post-training MOs are highly associated with faults. Given the above results, we focus solely on \(DeepMutationMS\) and random subsets from now on. Before proceeding to the next steps, we calculated the correlation between FDR and MS individually for each MO as well as for different subsets of operators. However, none of them achieved a significantly higher correlation than the correlation obtained with all MOs. Consequently, we proceeded to the next step considering, for each subject, all of the applicable MOs and performed regression analysis to model the relationship

Figure 3. Selected regression models across subjects based on the training set

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline Accuracy & \multicolumn{6}{c|}{Subjects} \\ \cline{2-7} metric & S1 & S2 & S3 & S4 & S5 & S6 \\ \hline \(R^{2}\) & 0.98 & 0.99 & 0.98 & 0.98 & 0.99 & 0.98 \\ \hline MMRE & 0.19 & 0.13 & 0.15 & 0.14 & 0.1 & 0.15 \\ \hline RMSE & 0.05 & 0.04 & 0.04 & 0.05 & 0.03 & 0.04 \\ \hline Shape* & RT & RT & RT & E & RT & RT \\ \hline \multicolumn{7}{l}{*Selected regression model shapes include Exponential (E) and Regression Tree (RT)} \\ \end{tabular}
\end{table}
Table 3. Accuracy and shape of the selected regression models when predicting FDR with MS, based on the training set and using a 5-fold cross-validation procedure
between MS and FDR on the training set. For each subject, we examined multiple regression models starting with a simple linear regression model and then using more complex non-linear models (quadratic, exponential, and regression trees) when needed. We then performed a 5-fold cross-validation procedure to calculate \(R^{2}\), MME, and RMSE, and selected the most accurate model for each subject. Due to space limitation, we only report the results for the selected regression model for each subject in Table 3. These results show a minimum \(R^{2}\) value of 0.98 across all subjects, indicating that MS explains most of the variance in FDR. The average RMSE across all \(K\)-folds for all subjects ranges from 0.03 to 0.05, highlighting the high prediction accuracy. Additionally, the average MME varies between 0.1 and 0.19 indicating a low relative error. This suggests that MS can be used to very accurately predict FDR in the training set.

The final row of Table 3 reports the shape of the selected regression model on each subject. Regression trees are the best for all subjects but S4. This suggests that standard regression functions are often not a good fit and regression trees offer more flexibility by optimally partitioning the MS range and computing averages within partitions. To prevent overfitting of the regression trees, which would yield poor results on test sets, we limited their maximum depth to 5.

To visualize our results, we present for each subject in Figure 3, the scatterplots and fitted regression curves between FDR and MS along with 95% prediction intervals (PI), based on all input subsets sampled from the training set. We compute PIs for regression tree models based on a bootstrapping method for non-parametric regression models (K
sampled input subsets. To address this, we selected diverse input subsets, employing both random and uniform sampling techniques and varying sample sizes. Another internal threat to validity stems from selected parameters used throughout the experiments. This encompasses parameters for clustering mispredicted inputs to estimate faults and for configuring MOs. To mitigate this concern, regarding clustering parameters, we explored different settings for each subject and opted for the best configuration based on the evaluation of the Silhouette index (Srivastava et al., 2017) and DBCV scores (Srivastava et al., 2017), as suggested by Aghababaeyan _et al._(Ag
## Acknowledgements

This work was supported by a research grant from Huawei Technologies Canada Co., Ltd, as well as by the Mitacs Accelerate Program. The experiments conducted in this work were enabled in part by support provided by the Digital Research Alliance of Canada 2.

Footnote 2: [https://alliancecan.ca](https://alliancecan.ca)
