Path Analysis for Effective Fault Localization in Deep Neural Networks
Despite deep learning's transformative impact on various domains, the
reliability of Deep Neural Networks (DNNs) is still a pressing concern due to
their complexity and data dependency. Traditional software fault localization
techniques, such as Spectrum-based Fault Localization (SBFL), have been adapted
to DNNs with limited success. Existing methods like DeepFault utilize SBFL
measures but fail to account for fault propagation across neural pathways,
leading to suboptimal fault detection. Addressing this gap, we propose the
NP-SBFL method, leveraging Layer-wise Relevance Propagation (LRP) to identify
and verify critical neural pathways. Our innovative multi-stage gradient ascent
(MGA) technique, an extension of gradient ascent (GA), activates neurons
sequentially, enhancing fault detection efficacy. We evaluated the
effectiveness of our method, i.e. NP-SBFL-MGA, on two commonly used datasets,
MNIST and CIFAR-10, two baselines DeepFault and NP- SBFL-GA, and three
suspicious neuron measures, Tarantula, Ochiai, and Barinel. The empirical
results showed that NP-SBFL-MGA is statistically more effective than the
baselines at identifying suspicious paths and synthesizing adversarial inputs.
Particularly, Tarantula on NP-SBFL-MGA had the highest fault detection rate at
96.75%, surpassing DeepFault on Ochiai (89.90%) and NP-SBFL-GA on Ochiai
(60.61%). Our approach also yielded results comparable to those of the
baselines in synthesizing naturalness inputs, and we found a positive
correlation between the coverage of critical paths and the number of failed
tests in DNN fault localization.
Soroush Hashemifar, Saeed Parsa, Akram Kalaee
**Path Analysis for Effective Fault Localization in Deep Neural Networks**

###### Abstract

Deep learning has revolutionized various real-world applications, but the quality of Deep Neural Networks (DNNs) remains a concern. DNNs are complex and have millions of parameters, making it difficult to determine their contributions to fulfilling a task. Moreover, the behavior of a DNN is highly influenced by the data used during training, making it challenging to collect enough data to exercise all potential DNN behavior under all possible scenarios. This paper proposes NP-SBFL method to locate faulty neural pathways (NP) using spectrum-based fault localization (SBFL). Our method identifies critical neurons using the layer-wise relevance propagation (LRP) technique and determines which critical neurons are faulty. Moreover, we propose a multi-stage gradient ascent (MGA), an extension of gradient ascent (GA), to effectively activate a sequence of neurons one at a time while maintaining the activation of previous neurons, so we are able to test the reported faulty pathways. We evaluated the effectiveness of our method, i.e. NP-SBFL-MGA, on two commonly used datasets, MNIST and CIFAR-10, two baselines DeepFault and NP-SBFL-GA, and three suspicious neuron measures, Tarantula, Ochiai, and Barinel. The empirical results showed that NP-SBFL-MGA is statistically more effective than the baselines at identifying suspicious paths and synthesizing adversarial inputs. Particularly, Tarantula on NP-SBFL-MGA had the highest fault detection rate at 96.75%, surpassing DeepFault on Ochiai (89.90%) and NP-SBFL-GA on Ochiai (60.61%). Our approach also yielded comparable results to the baselines in synthesizing naturalness inputs, and we found a positive correlation between the coverage of critical paths and the number of failed tests in DNN fault localization.

**Keywords**: Fault Localization; Neural Pathway; Deep Neural Networks; Neuron Relevancy; Statistical Analysis

## 1 Introduction

Deep Learning (DL) has been successful in addressing many real-world problems, such as classifying images [1], recognizing human speech [2], natural language processing [3], and software engineering tasks [4]. However, Deep Neural Networks (DNNs) have demonstrated quality issues, such as their inability to classify samples correctly in real-world applications despite achieving high accuracy. Real-world cases, such as Tesla/Uber accidents and incorrect diagnoses in healthcare, signify the need for assessment approaches in DNNs to assure their quality, particularly in safety- and security-critical systems [5]. Obviously, only DNN-based systems with high levels of trustworthiness should be approved to work in the public domain [6]. However, the complexity of DNN architectures makes it difficult to determine the contribution of each parameter to fulfilling a task. This complexity, combined with the fact that a DNN's behavior is produced according to the quality of data used to train it, makes it challenging to collect enough data to exercise all potential DNN behavior under various scenarios [7]. Therefore, effective testing approaches are needed to evaluate the quality of decisions made by DNN-based systems [8].

Although extensive research has shown that directly applying software testing methods to DNNs is not feasible, it is possible to map advanced principles behind to test DNNs [9, 10]. However, black-box DNN testing is limited in
providing insights into activation pattern of intermediate neurons and identifying inputs that result in exposing unexpected network behavior. To address this, researchers have turned to white-box testing techniques from software engineering. For example, DeepXplore [11] and DeepGauge [12] use differential algorithms and multi-granularity coverage criteria for conducting effective tests. Other research has proposed testing criteria and techniques inspired by metamorphic testing [13], combinatorial testing [14], mutation testing [15], MC/DC [16], symbolic execution [17], and Concolic testing [18].

Software fault localization aims to identify a system's parts responsible for incorrect behavior [19]. Spectrum-based fault localization (SBFL) is a promising technique for locating faults in traditional software programs, as explained in Section 2. Only one technique, DeepFault [20], currently employs traditional software fault localization to find suspicious neurons [20]. It utilizes measures such as Tarantula and Ochiai from the SBFL domain. Along with DeepFault, several approaches have been recently proposed to locate the most suspicious neurons within the network in Deep Neural Networks (DNNs) using statistical techniques [21-24].

However, none of the existing fault localization techniques considers the propagation of faults through the neural connections in different layers of the DNN. Identifying the faulty path can be achieved through pathways of sequential neurons most responsible for incorrect behavior. However, there are challenges in considering pathways in DNNs. Firstly, it is not easy to define a pathway. Additionally, the tremendous number of pathways in a DNN makes it impossible to calculate pathways for large DNNs, which can quickly become infeasible.

This paper proposes a novel approach for DNN fault localization that adapts Spectrum-based Fault Localization (SBFL) to locate faulty neural pathways. The NP-SBFL-MGA approach identifies critical neurons using the Layer-wise Relevance Propagation (LRP) technique and determines which critical neurons are faulty. A Multi-stage Gradient Ascent (MGA) technique, an extension of gradient ascent, is used to effectively activate a sequence of neurons one at a time while maintaining the activation of previous neurons. The method's effectiveness is evaluated on two commonly used datasets, MNIST and CIFAR-10, and compared with two baselines, DeepFault and NP-SBFL-GA, using three suspicious neuron measures, Tarantula, Ochiai, and Barinel. The approach outperforms the baselines in identifying faulty neural pathways and synthesizing adversarial inputs.

Specifically, Tarantula on NP-SBFL-MGA had the highest fault detection rate at 96.75%, surpassing DeepFault on Ochiai (89.90%) and NP-SBFL-GA on Ochiai (60.61%). It is concluded that this approach provides a systematic and effective framework for DNN fault localization and can help improve the quality and reliability of DNN-based software.

The effectiveness of the proposed method is evaluated on two commonly used datasets, MNIST [27] and CIFAR-10 [28]. The method is compared to two baselines, DeepFault and NP-SBFL-GA (a version of NP-SBFL that uses gradient ascent to activate neurons during input synthesis). Results show that NP-SBFL is statistically more effective than the baselines at identifying suspicious paths and synthesizing adversarial inputs. Specifically, the instance NP-SBFL-MGA using Tarantula had a 96.75% average fault detection rate compared to 89.90% for DeepFault using Ochiai and 60.61% for NP-SBFL-GA using Ochiai across all datasets and models. The proposed method also identifies unique suspicious neurons, especially in complex models, whereas the other methods rely heavily on a common neuron, making them less reliable. In addition to the fault detection rate, the naturalness of the synthesized inputs is evaluated using popular distance metrics such as Manhattan, Euclidean, Chebyshev, inception score, and Fre'chet Inception Distance. The approach yields comparable results to DeepFault. Furthermore, a positive correlation is found between the coverage of critical paths and the number of failed tests in DNN fault localization.

Overall, the contributions of this paper to the field of DNN fault localization are as follows: First, spectrum-based suspiciousness measures from software engineering are used to locate faulty pathways of neurons in DNNs. Second, verifying identified faulty pathways through multi-stage gradient ascent instead of just gradient ascent. Third, the proposal of an algorithm that guides the synthesis of inputs to activate potentially suspicious neural pathways. Fourth, the thorough evaluation of NP-SBFL on two publicly available datasets (MNIST and CIFAR-10) with different models demonstrates its feasibility and effectiveness.

The evaluation results show that the NP-SBFL method effectively identifies faulty neural pathways and synthesizes adversarial inputs. Compared to existing baselines, the method has demonstrated superiority in identifying unique suspicious neurons, especially in complex models. Furthermore, evaluating the synthesized inputs' naturalness shows
comparable results to existing methods. The correlation analysis also suggests a positive relationship between the coverage of critical paths and the number of failed tests in DNN fault localization.

The proposed NP-SBFL method offers a systematic and practical framework for DNN fault localization that can potentially improve the quality and reliability of DNN-based software. The contributions of this paper provide a foundation for future research on DNN fault localization, which could guide the development of safer and more trustworthy DNN-based systems.

The subsequent sections of this paper are structured as follows. Section 2 provides a comprehensive overview, covering the background of traditional software fault localization, the fault taxonomy in DNNs, and the relevance propagation concept. Section 3 summarizes the existing literature on fault localization in DNNs. In Section 4, we introduce our proposed approach, NP-SBFL. Section 5 contains a meticulous account of the experimental setup, the evaluation performed, and the potential threats to validity. Finally, Section 6 concludes the paper.

## 2 Background

The following section provides an overview of fault localization in traditional software, followed by an introduction to fault taxonomy in DNN models. Additionally, a detailed explanation of the relevancy concept is also included.

### 2.1 Fault Localization in Traditional Software

Fault localization (FL) can be categorized as white box testing that aims to identify source code elements more prone to produce faults [19]. An FL technique is applied to test a program P against a test oracle T, which reports tests with passed and failed status. Subsequently, an FL measurement technique is employed to determine how likely each program element is to cause fault during the execution time, i.e. suspiciousness of the program element.

Spectrum-based FL (SBFL) [29] belongs to a family of debugging techniques aiming to identify potentially faulty code by analyzing both passing and failing executions of a faulty program. It infers statistical properties from these executions and provides developers with a ranked list of potentially faulty statements to investigate. When given a faulty program and a set of test cases, an automated debugging tool based on SBFL generates a ranked list of potentially faulty statements. Developers can then inspect these statements one by one until they find the bug.

### 2.2 Faults Taxonomy in Deep Neural Networks

There are fundamental differences in definition of fault and detecting faults between regular software programs and DNN models [30]. Regular programs express their logic through control flow, while weights among neurons and various activation functions play crucial role in the logic of DNN programs. In software programs, bugs are often identified by comparing ground-truth outputs with expected outputs. As long as there is a difference between the ground-truth output and the expected one, a bug is considered to be present. In contrast, DNN-based program learns from a training dataset and once the DNN misclassifies during inference, the input samples are termed as failure cases. However, it is essential to note that since a DNN model cannot guarantee complete accurate classifications, such failures do not necessarily imply the presence of a bug. To address this, DeepDiagnosis [30] has discussed eight types of failure symptoms and their underlying causes. Additionally, Neural Trojans represent another type of symptom that can lead to misbehavior in DNNs [31]. These failure roots and symptoms are summarized in Table 1.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline \# & **Symptom** & **Description** & **Root causes** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Typical failure symptoms and their root causes in a DNN.
### Layer-Wise Relevance Propagation

Layer-wise relevance propagation (LRP) [32][33] is an explanation technique applicable to neural network models that handle inputs like images, videos, or text [34][35]. LRP operates by propagating the prediction f(x) backward in the neural network using specifically designed local propagation rules.

The propagation process employed by LRP adheres to a conservation property, where the relevance received by a neuron must be equally redistributed to the neurons in the lower layer. This behavior is akin to Kirchoff's conservation laws observed in electrical circuits. Let j and k represent neurons at two consecutive layers in the neural network. The relevance scores are propagated from a given layer onto the neurons of the lower layer using the rule depicted in Eq. 1.

\[R_{j}=\sum_{k}\frac{z_{jk}}{\sum_{j}z_{jk}}R_{k} \tag{1}\]
The variable z\({}_{jk}\) represents the contribution of neuron j to the relevance of neuron k. The denominator ensures the conservation property is maintained. The propagation process concludes once the input features are reached. By applying this rule to all neurons in the network, the layer-wise conservation property \(\sum_{j}R_{j}=\sum_{k}R_{k}\) can be easily verified and extended to a global conservation property \(\sum_{i}R_{i}=f(x)\). The overall LRP procedure is illustrated in Figure 1.

## 3 Related work

The literature discusses two distinct approaches for fault localization in DNN models. Analytical methods employ analytical techniques to identify the most suspicious neurons within the network. On the other hand, statistical methods use statistics, such as spectrum-based techniques, to identify faulty paths in the network. In the following review, we will briefly explore previous works in these two categories and investigate how FL techniques have been applied to DNNs.

### 3.1 Localizing faults in parameters

DeepFault [20] is an SBFL-based method designed for DNNs, which requires can be categorized as white-box approaches. DeepFault starts by identifying neurons which are more likely to cause misclassifications. To achieve this, DeepFault analyzes the behavior of intermediate neurons at inference, establishing a hit spectrum for each neuron under a specific test set. By analyzing these hit spectrums, DeepFault identifies highly defective neurons and synthesize new input samples that activate error prone neurons. The identification of suspicious neurons is based on calculating a suspiciousness score for each neuron using various measurements of suspiciousness: Tarantula [36], Ochiai [37], and DStar [38]. These measures have been used for fault localization in the field of software engineering for the first time [19]. Neurons with higher suspiciousness scores contribute more to incorrect DNN decisions, due to their lack of adequately training budget. Consequently, the weights of these neurons need to be further adjusted [39]. In the synthesis phase, correctly classified inputs are used to generate input samples that aim to maximize the activation values of neurons identified as suspicious. By doing so, the information produced by these neurons propagates through the network, causing shifts in the decision boundaries learned by the network.

NNrepair [22] proposes separate repair techniques for both the intermediate and last layers, aiming to find potentially faulty network weights. Repairing all the output classes at once can be challenging. So, NNrepair prepare a set of expert networks, each specializing in one specific label class, which is computationally more feasible. These experts are combined to create a final repaired classifier. In the fault localization step, network activation patterns are utilized as an oracle for correct behavior. An activation pattern \(\sigma\) specifies an activation state ("on" or "off") for a subset of neurons at a layer in the network. These patterns guide the identification of potentially faulty neurons. Highly

Figure 1: Illustration of the LRP procedure. Each neuron redistributes to the lower layer as much as it has received from a higher layer.
supported correct-label patterns corresponding to each output class at an intermediate layer are extracted. A highly supported correct-label pattern suggests that the network would likely classify any input satisfying the pattern to the appropriate label, while misclassified inputs would not meet the correct label pattern. For each incorrect input, the activations of the neurons corresponding to the correct-label pattern are compared, and neurons with different activations are considered potentially faulty. The repair process aims to modify the outputs of these neurons for each failing input to match the correct label pattern for their respective labels.

### 3.2 Localizing faults in architecture

DeepLocalize [21] presents a fault localization approach for DNNs, which requires access to the source code of the DNN model. DeepLocalize aims at fixing bugs in the architecture of the DL program. They propose two mechanisms to collect dynamic traces: (1) the first technique works by mapping the code into an intermediate representation of the DNN. This representation allows the internal states of the DNN to be monitored, enabling the insertion of probes for dynamic analysis during the training process. (2) The second technique investigates the internal state of the DNN source code by injecting a callback function. Through this dynamic analysis, DeepLocalize identifies the faulty hyper-parameters, including layers, responsible for the error. DeepLocalize aims to achieve three main objectives: (1) determine if there exist a bug inside the DL program, (2) localize the fault, and (3) provide failure information. They test DeepLocalize on buggy models collected from commits of GitHub and posts on Stack Overflow platforms, demonstrating its ability to identify bugs in 23 out of 29 buggy programs using the first technique and 34 out of 40 using the callback function.

DeepDiagnosis [30], as an analytical approach, offers bug fixes in the DL program. During training, the approach monitors essential indicators like weights and gradients, analyzing the recorded values to detect symptoms and potential training issues. When a symptom is identified, a Decision Tree is utilized to diagnose it based on pre-defined rules. The process begins by taking the initial model architecture and a training dataset as input and passing a callback method to the _fit(.)_ method. This callback method captures and records key values (e.g., weights and gradients) during feed-forward and backward propagation. A dynamic detector reports various symptoms at different stages during training, based on error conditions. If a symptom is detected, the recorded key values are analyzed to determine potential locations in the input model that need correction. DeepDiagnosis then reports the type of symptom, the layers and stages where it was detected, and suggests a location for fixing the issue.

Finally, Cao et al. [23] propose DeepFD, a learning-based fault diagnosis and localization framework that treats the fault localization task as a learning problem, in a statistical manner. It infers suspicious fault types by monitoring runtime features extracted during DNN model training and then locates the diagnosed faults in the architecture of the DL programs. DeepFD overcomes limitations by identifying the root causes of faults in DL programs instead of neurons and diagnoses them using a learning approach instead of a set of hard-coded rules.

Table 2 summarizes the benefits of existing and our proposed NP-SBFL fault localization methods.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **Approach** & **Repair model** & **Phase** & **Detection region** & **Focus** \\ \hline DeepFault [20] & Statistical & & Post-train & whole network & Neuron \\ \hline DeepLocalize [21] & Statistical & & During train & - & Hyper-parameters \\ \hline NNrepair [22] & Statistical & & Post-train & fully connected layers & Weights \\ \hline DeepDiagnosis [30] & Analytical & & During train & - & Hyper-parameters \\ \hline \hline \end{tabular}
\end{table}
Table 2: An overview of the explained Fault-Localization methods.
## 4 Proposed Approach

This section introduces our NP-SBFL white-box approach, which presents a systematic testing method for DNNs. The approach aims to identify and locate highly erroneous neural pathways within a DNN. By employing a sequence of analysis, identification, and synthesis steps on a pre-trained DNN, NP-SBFL can effectively pinpoint these problematic pathways and create new inputs that activate them. A comprehensive overview of NP-SBFL is given in Section 4.1, while Sections 4.2 to 4.4 delve into the specific details of the NP-SBFL steps.

### Overview

In this section, we provide an overview of NP-SBFL. Initially, we extract neurons that are critical from each layer of the neural network. This extraction process involves using LRP, as explained in Section 2.3, to unhide the internal decision process for input data. LRP measures how relevant each neuron is, starting from the output layer and tracing back to the first layer, in terms of their impact on prediction results. This relevance signifies the influence of each neuron on the way the network make decisions. Once we have the relevant neurons and their activation values for each layer, we employ fault localization methods based on spectrum, to identify suspicious neurons. Finally, we generate a synthesized dataset to activate these suspicious paths and verify whether they lead to incorrect classification results. To achieve this, we propose the use of a multi-stage gradient ascent technique to activate these faulty paths effectively.

### Neural Pathways Analysis

In this section, we present the method for identifying the essential decision path for a given input. As mentioned earlier, the relevance of each neuron \(R_{i}^{l}\) signifies its contribution to the final decision. A higher relevance value indicates that neuron i at layer l has a more significant impact on the decision-making process. Since a DNN's prediction relies on the extracting high quality features at each layer, the critical neurons with high relevance reveal the logic behind the decision process of the DNN [40].

Inspired by [40], for an input x, we formulate the condition of critical neurons as shown in Eq. 2 at each layer \(l_{i}\):

\[\sum_{i\in l_{i}}R_{i}^{l}\ >\ \alpha\ \cdot\ g_{f}(x) \tag{2}\]

where \(\alpha\) is a hyper-parameter controlling the volume of neurons that are considered as critical at each layer. By definition, a critical neuron positively contributes to the decision, meaning it has a positive relevance value. To determine the set of critical neurons, we define this set as the minimum subset of neurons with a cumulative relevance exceeding a predefined threshold. This threshold is a fraction of \(g_{f}(x)\) denoting the cumulative relevance of the input values. By adjusting \(\alpha\), we can control the number of critical neurons located at each layer, where smaller \(\alpha\) values result in fewer critical neurons being chosen.
### Suspicious Neural Pathways Identification

The next stage of NP-SBFL involves examining neural activity to pinpoint faulty neurons in each layer individually. To achieve this, NP-SBFL detects suspicious neurons by defining characteristics of a neuron's execution procedure, drawing inspiration from DeepFault [20]. However, unlike DeepFault, which considers all neurons collectively, our focus is on critical neurons in each layer, as they hold greater relevance to the DNN's decision-making process. The quantities \(A^{c}_{p}\) and \(A^{c}_{f}\) represent the number of times a neuron was covered by the input sample and resulted in a passed or failed decision, respectively. Similarly, quantities \(A^{n}_{p}\) and \(A^{n}_{f}\) indicate cases where the neuron remained uncovered by the input sample. The abstract procedure for determining the suspicious scores are presented in Algorithm 1.

```
1:Input a neural network with L layers; test set T; activation threshold \(\beta\)
2:Output suspiciousness scores of all neurons in the network
3:\(A^{c}_{p}=\) matrix of active-passed tests with initial value of \(0\)
4:\(A^{n}_{p}=\) matrix of inactive-passed tests with initial value of \(0\)
5:\(A^{n}_{f}=\) matrix of active-failed tests with initial value of \(0\)
6:\(A^{n}_{f}=\) matrix of inactive-failed tests with initial value of \(0\)
7:\(supiciousness Scores=\) matrix of suspiciousness scores associated with each neuron
8:for\(input_{sample}\in test\ set\ T\)do
9: predict \(input_{sample}\) by the network
10: get relevancy and activation values of all layers of network
11:\(neurons_{critical}=[]\)
12:for\(layer\ l=1,2,\ldots,L\)do
13: find critical neurons according to equation (2) and append them to \(neuron_{critical}\)
14:endfor
15:for\(layer\ l=1,2,\ldots,L\)do
16:for\(neuron\ n\in layer\ l\)do
17:if\(input_{sample}\) is correctly classified then
18:if\(n_{activation}\geq\beta\) and \(n\in neurons_{critical}[l]\)then
19:\(increment\ A^{n}_{p}[l][n]\)
20:endif
21:if\(n_{activation}<\beta\) and \(n\in neurons_{critical}[l]\)then
22:\(increment\ A^{n}_{p}[l][n]\)
23:endif
24:else
25:if\(n_{activation}\geq\beta\) and \(n\in neurons_{critical}[l]\)then
26:\(increment\ A^{n}_{f}[l][n]\)
27:endif
28:if\(n_{activation}<\beta\) and \(n\in neurons_{critical}[l]\)then
29:\(increment\ A^{n}_{f}[l][n]\)
30:endif
31:endif
32:endfor
33:endfor
34:for\(layer\ l=1,2,\ldots,L\)do
35:for\(neuron\ n\in l\)do
36:\( suspiciousness Scores[l][n]=SBFL(A^{c}_{p}[l][n],A^{n}_{p}[l][n],A^{c}_{f}[l][n],A^{n}_{f}[l][n])\)
37:endfor
38:return\(suspiciousness Scores\)
```

**Algorithm 1** Determine Suspiciousness Scores

Under a test set T, NP-SBFL analyzes the behavior of relevant neurons in the DNN to create a hit spectrum for each neuron (lines 6-32), which describes its dynamic behavior. This involves utilizing the activation patterns of relevant neurons in each layer for analysis (lines 16-29). After obtaining the set of hit spectrums through DNN
analysis, NP-SBFL proceeds to individually identify suspicious relevant neurons in each layer. These suspicious neurons are grouped from the first to the last layer, forming a sequence of neurons. Finally, NP-SBFL employs a suspiciousness measure based on spectrum, which calculates a suspiciousness score for each neuron based on spectrum-related information (lines 33-37). The higher the suspiciousness score, the more likely the neuron was inadequately trained, contributing to incorrect DNN decisions. In this paper, we implement NP-SBFL using three distinct suspiciousness measures: Tarantula [36], Ochiai [41], and Barinel [29]. Their algebraic formulas are presented in Table 3.

These suspiciousness measures operate on the principle that a neuron is deemed more suspicious if it is frequently covered by test inputs resulting in incorrect DNN decisions, and less frequently covered by test inputs leading to correct decisions. Upon analyzing the suspiciousness of neurons in each layer of a DNN, the neurons are arranged in descending order of suspiciousness. The k most likely defective neurons are then selected in each layer to form a pattern of interconnected neurons.

### Suspiciousness-Guided Input Synthesis

To assess the faulty paths identified in the previous steps, it is necessary to activate these suspicious paths and check whether they lead to misclassifications by the DNN. To achieve this, we aim to maximize the activation of the set of suspicious neurons determined by NP-SBFL, using a technique called Activation Maximization. Activation Maximization is a visualization method for neural networks that seeks to maximize the activation of specific neurons. During standard training, the weights and biases of the neural network are iteratively adjusted to minimize the error (or loss) across training examples in the dataset. Activation Maximization, on the other hand, works inversely. After the classifier has been trained, we want to iteratively find the parts of the data that the model associates with a specific class.

One approach used in Activation Maximization is Gradient Ascent (GA). Gradient Ascent involves using the derivative of a differentiable function, f(x): \(R^{d}\rightarrow\mathbb{R}\), to determine how much f(x) will change if we increase x slightly. This information is valuable for optimization problems, such as finding a minimum. In Gradient Descent algorithm, we start from any point and move in the direction that decreases f(x) the most, repeatedly calculating the gradient and taking steps until we reach the minimum. In the context of our model, the gradient of the function (neural network) provides information about each input dimension's effect on increasing the function's value [43].

For activating target neurons, we apply the GA to each neuron, adding the gradient of that neuron with respect to the input data to the input image pixels. However, since NP-SBFL operates based on a sequence of neurons, the

\begin{table}
\begin{tabular}{l c} \hline \hline
**Suspiciousness Measure** & **Algebraic Formula** \\ \hline Tarantula & \(\dfrac{A_{f}^{c}}{A_{f}^{c}\ +\ A_{f}^{n}}\) \\ \(\dfrac{A_{f}^{c}}{A_{f}^{c}\ +\ A_{f}^{n}}\ +\dfrac{A_{p}^{c}}{A_{p}^{c}\ +\ A_{p}^{n}}\) \\ \hline Ochiai & \(\dfrac{A_{f}^{c}}{\sqrt{\left(A_{f}^{c}\ +\ A_{f}^{n}\right)\ *\ (A_{f}^{c}\ +\ A_{p}^{c})}}\) \\ \hline Barinel & \(1-\dfrac{A_{p}^{c}}{A_{f}^{c}\ +\ A_{p}^{c}}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Suspiciousness measures used in NP-SBFL.
naive GA may not be effective in activating these paths. Activating a neuron in later layers could inadvertently deactivate previous neurons in earlier layers along the path, or vice-versa.

To address this issue, we developed a method called multi-stage Gradient Ascent (MGA), an extension of GA. MGA activates a sequence of neurons one at a time while maintaining the activation of previous neurons. To achieve this, we minimize a loss function for all layers sequentially. The intended loss function is represented by Eq. 3.

\[\mathit{loss}_{L}=\mathit{-activation}_{target}^{L}+\sum_{l=1}^{L-1}(- \mathit{activation}_{target}^{l}+\mathit{\left\lfloor activation_{target}^ {L}\right\rfloor}\mathit{-activation}_{target}^{l}) \tag{3}\]

The loss function is minimized layer-by-layer. For each layer L, the target activations in the current layer (first term) and all previous target activations are kept activated (second term). As described in Algorithm 2, the gradient of this loss function is then added to the input image to synthesize it and activate the suspicious paths (line 11). Finally, the domain constraints are applied to the synthesized image (line 12), which consists of normalizing the image's new pixel values in range of [0, 1]. The effectiveness of MGA is explored in later sections.

```
1:Input a neural network with L layers; test set T; list of suspicious neurons in each layer \(\mathit{suspicious_{neurons}}\); learning rate \(lr\)
2:Output a set of synthesized images to activate faulty paths
3:\(synthesizedImages\) = []
4:for\(input_{sample}\in tests\,set\,\,T\)do
5:for\(iteration=1,2,\,\ldots,\,I\)do
6: predict \(input_{sample}\) by the network
7:if\(input_{sample}\) is correctly classified then
8:for\(layer\,l=1,2,\ldots,L\)do
9:for\(neuron\,n\in\mathit{suspicious_{neurons}}[l]\)do
10: calculate loss objective according to equation (3)
11:endfor
12:\(gradients=\frac{\partial loss}{\partial input_{sample}}\)
13:\(input_{sample}=input_{sample}-lr*gradients\)
14: apply domain constraints on \(input_{sample}\)
15:endfor
16: append synthesized \(input_{sample}\) to \(synthesizedImages\)
17:endif
18:endfor
19:endfor
20:return\(synthesizedImages\)
```

**Algorithm 2** Suspiciousness-Guided Input Synthesis

## 5 Implementation

The prototype tool we have implemented on top of PyTorch v2.0 has been designed to streamline the evaluation and adoption of the NP-SBFL approach. By providing intuitive functionality, we aim to make it easier for researchers and practitioners to incorporate NP-SBFL into their work. Additionally, we have made the full experimental results available on the NP-SBFL project page at [https://github.com/soroushhashemifar/NP-SBFL](https://github.com/soroushhashemifar/NP-SBFL), allowing for easy access and reference.

## 6 Evaluation

In this section, we describe our experiments to evaluate the effectiveness of our fault localization technique, NP-SBFL, in localizing faulty sequences of neurons within a neural network. Firstly, we describe our experimental setup, including the dataset used, the neural network architecture, and the evaluation metrics employed. Next, we outline the research questions we aim to answer through our experimental evaluation. These questions include comparing the performance of different fault localization approaches, evaluating our technique's effectiveness using different
suspiciousness measures, and examining the correlation between critical path coverage and the number of failed tests. Subsequently, we present the results of our experiments and discuss their implications. We provide a detailed analysis of the performance of different fault localization approaches and evaluate our technique's effectiveness using different suspiciousness measures. Finally, we discuss potential threats to the validity of our experimental evaluation and outline steps taken to mitigate these threats.

### Experimental Setup

We evaluate NP-SBFL using two widely-used datasets. The first dataset, MNIST [27], contains handwritten digit samples comprising 60,000 training and 10,000 testing samples. Each sample comprises a 28x28 pixel image with a class label ranging from 0 to 9. The second dataset, CIFAR-10 [28], is an image dataset containing 50,000 training samples and 10,000 testing samples. This dataset includes 32x32 images of ten distinct classes, such as dogs, birds, and cars.

In our analysis of both datasets, we examine six DNNs implemented through the PyTorch framework, with the specific configurations outlined in Table 4. Each DNN has distinct architectural features and varying numbers of trainable parameters. To obtain a minimum accuracy of 95% on the MNIST dataset, we employ three deep Fully-connected NNs. Conversely, for the CIFAR-10 dataset, we experiment with three convolutional NNs enhanced by max-pooling and ReLU activation functions to achieve a minimum accuracy of 70%.

We have implemented NP-SBFL using the suspiciousness measures Tarantula, Ochiai, and Barinel, as shown in Table 5. To assess the effectiveness of NP-SBFL, we conducted experiments using different numbers of suspicious neurons, specifically k@{1,5,10} for MNIST models and k@{10,30,50} for CIFAR models. In addition, we conducted extensive experiments for models of Table 4 to optimize the hyper-parameters of Algorithm 2 and facilitate the reproducibility of our results. Through empirical analysis, we determined that the values listed in Table 5 are suitable for perturbing inputs in MNIST and CIFAR-10 models. The NP-SBFL method employs two gradient ascent synthesizers - NP-SBFL-GA, a simple one, and NP-SBFL-MGA, a multi-stage synthesizer. For DeepFault, we utilized the step size values recommended by the original paper.

Furthermore, we determine an upper-bound for allowed distance (d) to ensure it does not exceed the distances specified in Table 5, considering the range of values provided in each dimension of input sample and also its highest pixel value. For DeepFault, we again employed the distance values suggested by the original paper. Exploring alternative step and parameter d is an area of future investigation for our research.

The evaluations were conducted on an Ubuntu desktop with a memory capacity of 16 GB and an Intel(r) Core(tm) i7-4790K CPU running at 4.00GHz with eight cores. Throughout the experiments, a criticality coefficient \(\alpha\) value of 0.7 was employed. Given that all activation functions were ReLU, neurons with an activation value greater than 0.0 were considered to be activated.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Model** & \begin{tabular}{c} **\# Trainable** \\ **Params** \\ \end{tabular} & **Architecture** & **Accuracy** & \begin{tabular}{c} **\#Correctly** \\ **Classified** \\ **Samples** \\ \end{tabular} \\ \hline \multirow{4}{*}{MNIST} & MNIST\_1 & 27,420 & 5 * \textless{}30\textgreater{}, \textless{}10\textgreater{} & 96.6 \% & 9631 \\ \cline{2-5}  & MNIST\_2 & 22,975 & 6 * \textless{}25\textgreater{}, \textless{}10\textgreater{} & 95.8 \% & 9581 \\ \cline{2-5}  & MNIST\_3 & 18,680 & 8 * \textless{}20\textgreater{}, \textless{}10\textgreater{} & 95 \% & 9512 \\ \hline \multirow{4}{*}{
\begin{tabular}{c} CIFAR-10 \\ \end{tabular} } & CIFAR\_1 & 411,434 & 2 * \textless{}32@3x3\textgreater{}, 2 * \textless{}64@3x3\textgreater{}, 4 * \textless{}128\textgreater{}, \textless{}10\textgreater{} & 70.1 \% & 7066 \\ \cline{1-1} \cline{2-5}  & CIFAR\_2 & 724,010 & 2 * \textless{}32@3x3\textgreater{}, 2 * \textless{}64@3x3\textgreater{}, 2 * \textless{}64@3x3\textgreater{}, 2 * \textless{}256\textgreater{}, \textless{}10\textgreater{} & 72.6 \% & 7413 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Configuration of all DNN models used in NP-SBFL
### Research Questions

Our experimental evaluation aims to address the following research questions.

* RQ1 (Validation): Does empirical evaluation validate the effectiveness of NP-SBFL in identifying suspicious paths and its ability to outperform DeepFault, a neuron-based suspiciousness selection strategy, in synthesizing adversarial inputs that lead to misclassification of previously correctly classified inputs by DNNs?
* RQ2 (Comparison): How do NP-SBFL-GA, NP-SBFL-MGA, and DeepFault instances with different measurements of suspiciousness compare against each other? The evaluation is conducted by analyzing the results produced by different instances using Tarantula [36], Ochiai [41], and Barinel [29].
* RQ3 (Fault Detection Rate): Which of the selected approaches exhibits a greater fault detection rate?
* RQ4 (Correlation) Is there a correlation between the coverage of critical paths and the number of failed tests in DNN fault localization?
* RQ5 (Locating Unique Suspicious Neurons): Which approach is more effective in locating unique suspicious neurons among the different instances?
* RQ6 (Quality of Synthesized Inputs): How well can NP-SBFL synthesize inputs of acceptable quality?

### Results and Discussion

In this section, we design and conduct specific experiments to address each research question and provide insights into the impact of activating suspicious paths on the model's performance and potential vulnerabilities. Our experiment process involves several steps. Firstly, we choose a group of accurately classified samples from the test set based on the pre-trained model's predictions. Next, we utilize an input synthesis technique to activate suspicious paths in the model. This technique involves modifying the input samples to highlight certain patterns that could trigger misclassifications. Then, we measure the misclassification rate by inputting the synthesized samples into the model and comparing the predicted and actual labels. After that, we calculate the fraction of samples that activate the suspicious paths and lead to misclassification by dividing the number of misclassified samples by the total number of synthesized samples. Lastly, we analyze the ratios obtained to conclude the effectiveness of activating suspicious paths in producing faulty results.

#### RQ1 (Validation)

We use the NP-SBFL workflow to analyze DNNs in Table 4. This analysis involves identifying the K neurons with the highest scores using a suspiciousness measure and synthesizing new input samples from those input samples classified correctly, which exercise the identified neurons. The last column in Table 4 shows the number of correctly

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{DeepFault} & \multicolumn{2}{c}{NP-SBFL-GA} & \multicolumn{2}{c}{NP-SBFL-MGA} \\ \cline{2-7}  & step size & distance & step size & distance & step size & distance \\ \hline \hline MNIST-1 & 1 & 0.1 & 1 & 0.5 & 5 & 0.006 \\ \hline MNIST-2 & 1 & 0.1 & 1 & 0.5 & 5 & 0.006 \\ \hline MNIST-3 & 1 & 0.1 & 1 & 0.5 & 5 & 0.006 \\ \hline CIFAR-1 & 10 & 0.1 & 10 & 0.1 & 5 & 0.01 \\ \hline CIFAR-2 & 10 & 0.1 & 10 & 0.1 & 5 & 0.02 \\ \hline CIFAR-3 & 10 & 0.1 & 10 & 0.1 & 5 & 0.04 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The hyper-parameters of different approaches and models.
classified inputs. We evaluate the DNN's prediction performance using standard metrics like cross-entropy loss and accuracy. The analysis is done per class since inputs from the same class have similar activation patterns.

Tables 6 to 8 show the average loss and accuracy of inputs synthesized by different methods, including DeepFault, NP-SBFL-GA, and NP-SBFL-MGA. They use Tarantula, Ochiai, and Barinel to identify suspicious neurons on MNIST (top) and CIFAR-10 (bottom) models from Table 4. In Tables 6 to 8, each cell value represents an average over the synthesized inputs. The number of synthesized inputs for each model is reported in the last column of Table 4. The loss and accuracy of the synthesized samples demonstrate the effectiveness of the gradient ascent method in generating improved samples to deceive the network.

Table 6 reveals that DeepFault using Tarantula and Barinel achieved significantly lower prediction performance than Ochiai on all models. The performance between Tarantula and Barinel is similar. These results suggest that the identified neurons are suspicious, so their weights are insufficiently trained. In NP-SBFL-GA and NP-SBFL-MGA, Ochiai and Barinel have similar effectiveness on the models and obtain lower performance than Tarantula. These results demonstrate that the identified paths are suspicious, so their weights are insufficiently trained. Therefore, slightly perturbing the inputs correctly classified by the DNN could transform them into adversarial by increasing the activation value of suspicious paths.
\begin{table}
\begin{tabular}{l l l l l l l l l l l} \hline \hline  & & K=1 & K=5 & K=10 & K=1 & K=5 & K=10 & K=1 & K=5 & K=10 \\ \hline \multirow{2}{*}{MNIST-1} & Loss & **0.1553** & 0.0280 & 0.0113 & 0.1340 & **0.0383** & **0.0201** & 0.1340 & **0.0383** & **0.0201** \\  & Accuracy & **25.07** & 66.80 & 81.58 & 34.39 & **64.45** & **75.28** & 34.39 & **64.45** & **75.28** \\ \hline \multirow{2}{*}{MNIST-2} & Loss & 0.0274 & 0.0271 & 0.0208 & **0.0548** & **0.0392** & **0.0273** & **0.0548** & **0.0392** & **0.0273** \\  & Accuracy & 65.41 & 61.51 & 68.40 & **46.65** & **59.38** & **64.46** & **46.65** & **59.38** & **64.46** \\ \hline \multirow{2}{*}{MNIST-3} & Loss & **0.1297** & 0.0275 & 0.0271 & 0.0819 & **0.0574** & **0.0290** & 0.0819 & **0.0574** & 0.0275 \\  & Accuracy & **40.65** & 72.93 & 74.20 & 41.61 & **59.85** & **73.08** & 41.61 & **59.85** & 74.40 \\ \hline \hline \multirow{2}{*}{CIFAR-1} & \multirow{2}{*}{K=10} & K=30 & K=50 & K=10 & K=30 & K=50 & K=10 & K=30 & K=50 \\  & & & & & & & & & \\ \hline \multirow{2}{*}{CIFAR-1} & Loss & 0.0106 & 0.0065 & **0.0052** & **0.0107** & **0.0093** & 0.0049 & **0.0107** & **0.0093** & 0.0051 \\  & Accuracy & **59.97** & 72.95 & **77.73** & 61.47 & **65.49** & 79.63 & 61.47 & **65.49** & 79.06 \\ \hline \multirow{2}{*}{CIFAR-2} & Loss & **0.0146** & 0.0122 & 0.0119 & 0.0132 & **0.0158** & **0.0151** & 0.0132 & **0.0158** & **0.0151** \\  & Accuracy & **47.38** & 51.73 & 52.39 & 48.18 & **45.01** & **45.74** & 48.18 & **45.01** & **45.74** \\ \hline \multirow{2}{*}{CIFAR-3} & Loss & **0.0102** & **0.0074** & **0.0076** & 0.0083 & 0.0062 & 0.0070 & 0.0083 & 0.0062 & 0.0070 \\  & Accuracy & **56.06** & **65.65** & **65.20** & 63.03 & 71.69 & 67.89 & 63.03 & 71.9 & 67.89 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Loss and accuracy of all models on the synthesized data for NP-SBFL-MGA on all the selected models. The best results per suspiciousness measure are shown in bold. K represents the number of suspicious neurons.

\begin{table}
\begin{tabular}{l l l l l l l l l l l} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Measure**} & \multicolumn{4}{c}{**Tarantula**} & \multicolumn{4}{c}{**Ochiai**} & \multicolumn{4}{c}{**Barinel**} \\ \cline{3-10}  & & K=1 & K=5 & K=10 & K=1 & K=5 & K=10 & K=1 & K=5 & K=10 \\ \hline \multirow{2}{*}{MNIST-1} & Loss & **0.0442** & 0.0317 & **0.0679** & 0.0293 & **0.0713** & 0.0392 & 0.0293 & **0.0713** & 0.0392 \\  & Accuracy & 47.52 & 51.68 & **34.18** & **46.53** & **28.43** & 41.0 & **46.53** & **28.43** & 41.0 \\ \hline \multirow{2}{*}{MNIST-2} & Loss & **0.0437** & **0.0450** & **0.0452** & 0.0393 & 0.0394 & 0.0429 & 0.0393 & 0.0394 & 0.0429 \\  & Accuracy & 33.99 & **30.04** & 35.06 & **33.04** & 35.01 & **34.51** & **33.04** & 35.01 & **34.51** \\ \hline \multirow{2}{*}{MNIST-3} & Loss & **0.0423** & 0.0896 & **0.0665** & 0.0201 & **0.0911** & 0.0652 & 0.0201 & **0.0911** & 0.0652 \\  & Accuracy & **26.04** & 25.47 & 30.52 & 38.19 & 19.50 & **24.17** & 38.19 & 19.50 & **24.17** \\ \hline \hline \multirow{2}{*}{CIFAR-1} & \multirow{2}{*}{K=10} & K=30 & K=50 & K=10 & K=30 & K=50 & K=10 & K=30 & K=50 \\  & Accuracy & **0.0160** & **0.0234** & **0.0209** & 0.0158 & 0.0182 & 0.0205 & 0.0158 & 0.0182 & 0.0205 \\  & Accuracy & **30.76** & **23.06** & 21.27 & 31.61 & 26.80 & 21.22 & 31.61 & 26.80 & **19.17** \\ \hline \multirow{2}{*}{CIFAR-2} & Loss & 0.0196 & 0.0233 & 0.0236 & **0.0320** & **0.0469** & **0.0547** & **0.0320** & **0.0469** & **0.0547** \\  & Accuracy & 38.01 & 32.22 & 33.91 & **20.41** & **17.65** & **15.28** & **20.41** & **17.65** & **15.28** \\ \hline \multirow{2}{*}{CIFAR-3} & Loss & 0.0183 & 0.0222 & 0.0255 & **0.0231** & **0.0242** & **0.0363** & **0.0231** & 0.0234 & **0.0363** \\  & Accuracy & 41.19 & 36.57 & 30.59 & **25.50** & **19.57** & **15.03** & **25.50** & 19.83 & **15.03** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Loss and accuracy of all models on the synthesized data for NP-SBFL-MGA on all the selected models. The best results per suspiciousness measure are shown in bold. K represents the number of suspicious neurons.
Figures 2 and 3 show the loss and accuracy of MNIST and CIFAR on the synthesized data for different approaches and suspicious measures. The figures reveal that among the different approaches, the suspicious paths reported by NP-SBFL-MGA are more responsible for insufficient DNN performance because it obtains the lowest accuracy and higher loss, except for MNIST_1, in all variant K. This observation suggests two points: 1) neural pathway fault localization is more successful than neuron-based fault localization in identifying a DNN low performance, and 2) a gradient ascent synthesizer technique is not suitable for activating sequences of neurons. The last point will be further discussed in RQ2.

We used the Wilcoxon rank-sum test [44] for statistical significance at a 95% confidence level and the Vargha and Delaney's A 12 statistics [45] for the effect size measure to compare the performance of NP-SBFL-MGA instances with other methods. Table 9 reports the results. According to the table, NP-SBFL-MGA had a statistically significant difference (p-value \(<\) 0.05) compared to other approaches for all CIFAR-10 models. However, for all MNIST models, NP-SBFL-MGA achieved significantly lower accuracy than other methods. We will further investigate this observation in our future work.

Tables 6 to 8 show that there is a performance difference for all instances when using different K values. Further analysis of the trend of accuracy and loss for each instance, respectively depicted in Figures 4 and 5, revealed that increasing K values during fault localization results in the synthesized inputs increasing accuracy and decreasing loss for all instances except for NP-SBFL-MGA. This trend is reversed for all instances of NP-SBFL-MGA except for MNIST with K = 10 when using Ochiai and Barinel. This achievement is because NP-SBFL-MGA uses a multi-stage gradient ascent technique, a layer-wise paradigm, to maximize the activation values produced by suspicious neural pathways, unlike DeepFault and NP-SBFL-GA.
Figure 2: Loss and accuracy of MNIST on the synthesized data for different approaches and suspicious measures.
Figure 3: Loss and accuracy of CIFAR on the synthesized data for different approaches and suspicious measures.

\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{Approach} & \multirow{2}{*}{Model} & \multirow{2}{*}{Measure} & \multicolumn{2}{c}{Tarantula} & \multicolumn{2}{c}{Ochiai} & \multicolumn{2}{c}{Barinel} \\ \cline{3-8}  & & & A12 & P-value & A12 & P-value & A12 & P-value \\ \hline \hline \multirow{4}{*}{DeepFault} & \multirow{2}{*}{MNIST} & Accuracy & **0.87** & \(<\)**0.001** & **1** & \(<\)**0.001** & **0.92** & \(<\)**0.001** \\ \cline{3-8}  & & Loss & 0.62 & 0.01 & 0.48 & 0.04 & 0.35 & 0.10 \\ \cline{2-8}  & \multirow{2}{*}{CIFAR} & Accuracy & **1** & \(<\)**0.001** & **1** & \(<\)**0.001** & **1** & \(<\)**0.001** \\ \cline{3-8}  & & Loss & **0.97** & \(<\)**0.001** & **0.97** & \(<\)**0.001** & **0.97** & \(<\)**0.001** \\ \hline \hline \end{tabular}
\end{table}
Table 9: The statistical test compares the performance of NP-SBFL-MGA instances with other methods.
Figure 4: Accuracy of MNIST and CIFAR on the synthesized data using different approaches and suspicious measures for a variant number of suspicious neurons, K: MNIST(CIFAR).
_RQ1 (Validation)._ There is empirical evidence of suspicious neural pathways that could be causing inadequate DNN performance. NP-SBFL-MGA instances with different suspiciousness measures significantly improve localizing low performance compared to other approaches for most DNN models (except for the loss case on MNIST models).

Figure 5: Loss of MNIST and CIFAR on the synthesized data using different approaches and suspicious measures for a variant number of suspicious neurons, K: MNIST(CIFAR).
#### RQ2 (Comparison)

We compared suspiciousness measures in all instances and performed pair-wise comparisons using the Wilcoxon rank-sum test [44] for statistical significance at a 95% confidence level and the Vargha and Delaney's A'12 statistics [45] for the effect size measure to explore whether there are significant differences between Tarantula, Ochiai, and Barinel. Tables 11 and 12 present the results of these comparisons for MNIST and CIFAR10, respectively. Our study has concluded that NP-SBFG-MGA delivers superior results on all models when using any suspicious measure. This achievement suggests that the neural pathways identified as suspicious by NP-SBFG-MGA have a greater impact on DNN performance than other methods. Therefore, there is no single best spectrum-based suspiciousness measure. This finding is consistent with traditional fault localization techniques used in software development.

\begin{table}
\begin{tabular}{c
_RQ2 (Comparison)._ NP-SBFG-MGA with any suspiciousness measure is statistically superior to other instances in uncovering the low performance of models. These findings have significant implications for the field of neural network development and suggest that focusing on specific pathways rather than single neurons can lead to more accurate fault localization.

#### RQ3 (Fault Detection Rate)

Table 13 displays the proportions of synthesized samples that activate the critical pathways (C) and the proportions of failed synthesized samples that activate the faulty pathways (F) for each instance of all models. In the table, DF stands for DeepFault, NG for NP-SBFL-GA, and NM represents NP-SBFL-MGA. It can be observed that NP-SBFL-MGA using Tarantula achieves higher values in terms of both criteria C and F for all the models compared to the other instances. The second-best result is obtained by DeepFault using Ochiai. The effectiveness of the generated synthesized samples in activating faulty pathways for all approaches is represented by Figures 6 to 8, using Tarantula, Ochiai, and Barinel, respectively. These figures confirm that higher values of the number of covered critical pathways are associated with higher values of the number of failed tests. This point will be statistically proven in RQ4. Figure 7 further demonstrates that Ochiai effectively detects faulty neurons for all models. The results of Figure 6 indicate that Tarantula performs effectively in detecting faulty pathways across all models. The results of DeepFault using Tarantula and Barinel are not as satisfactory for CIFAR10, while NP-SBFL-GA and NP-SBFL-MGA show similar levels of effectiveness for all models.
Figure 6: A comparison between the averaged ratios of failed synthesized tests and the synthesized samples activating faulty pathways for Tarantula in different models and approaches.
_RQ3 (Fault Detection Rate)._ NP-SBFL-MGA using Tarantula is the most effective approach regarding the ratio of failed synthesized test samples activating faulty pathways for all models. Moreover, Ochiai is an acceptable option for detecting faulty neurons, while Tarantula is a good alternative for detecting faulty pathways.

#### RQ4 (Correlation)

Figure 9 displays the scatter plots illustrating the correlation between the rate of covered critical pathways and failed tests across all instances and models. Except for DeeFault using Ochiai and NP-SBFL-GA using Tarantula, it is observed that an increase in the rate of covered critical pathways leads to a corresponding increase in the rate of failed tests.

We utilized the Spearman correlation coefficient to investigate the statistical correlation between these variables [46]. The reason for selecting the Spearman correlation is its ability to quantify the strength of a monotonic relationship between two variables while remaining agnostic about the relationship's form or the distribution of the data [46]. Our findings indicate a significant positive correlation between the variables. The obtained correlation results are presented in Table 14, encompassing various instances of all DNN models. Bold values within the table denote statistically significant correlations (p-value \(<=\) 0.05). Consequently, our results showcase statistically significant correlations between the rate of covered critical pathways and the percentage of failed tests in all instances.

Figure 8: A comparison between the averaged ratios of failed synthesized tests and the synthesized samples activating faulty pathways for Barinel in different models and approaches.

Figure 7: A comparison between the averaged ratios of failed synthesized tests and the synthesized samples activating faulty pathways for Ochiai in different models and approaches.
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{Approach} & \multirow{2}{*}{Measure} & \multicolumn{2}{c}{MNIST} & \multicolumn{2}{c}{CIFAR} \\ \cline{3-5}  & & Spearman & P-value & Spearman & P-value \\ \hline \multirow{3}{*}{DeepFault} & Tarantula & **0.99** & **< 0.001** & **1** & **0** \\ \cline{2-5}  & Ochiai & **0.93** & **< 0.001** & **0.91** & **< 0.001** \\ \cline{2-5}  & Barinel & **0.99** & **< 0.001** & **1** & **0** \\ \hline \hline \end{tabular}
\end{table}
Table 13: Correlation results between the number of covered critical pathways and the detected faults by different approaches. The bolds refer to statistically significant correlations (p-value \(<\)= 0:05).

Figure 9: The correlation between the rate of covered critical pathways and failed tests across all instances and models.
#### RQ5 (Locating Unique Suspicious Neurons)

Figure 10 displays the ratios of common neurons in inputs synthesized by NP-SBFL-MGA using Tarantula, Ochiai, and Barinel for various K and different models. Notably, from the left part of Figure 10, Ochiai and Barinel detected the same suspicious neurons, albeit with varying levels of suspicion. Conversely, the right part of Figure 10 reveals that Tarantula identified entirely distinct neurons, particularly in convolutional networks. It shows that the suspicious neurons detected by Tarantula yielded higher reliability levels than those detected by Ochiai and Barinel.

Figure 11 illustrates the ratios of common neurons between the best results of DeepFault using Ochiai for K = 10 (50) and the best results of NP-SBFL-MGA using Tarantula for K = 10 (5). As observed in Figure 5, fully-connected networks exhibit more common neurons between NP-SBFL-MGA and DeepFault. However, it is worth noting that suspicious neurons detected by NP-SBFL-MGA differ significantly for larger models such as convolutional networks.

Figure 10: A comparison between the Ratios of common neurons in each layer of a DNN model for NP-SBFL-MGA using Ochiai against Barinel (left) and Tarantula against Ochiai/Barinel (right).
_RQ5 (Locating Unique Suspicious Neurons)._ Ochiai and Barinel detect common suspicious neurons but with varying suspicion levels among the different suspicious measures. However, Tarantula identifies distinct neurons, particularly in convolutional networks, and detects the neurons with higher reliability than those detected by Ochiai and Barinel. Besides, more common neurons between DeepFault and NP-SBFL-MGA are observed in fully-connected networks. The significant differences are in suspicious neurons detected by NP-SBFL-MGA for complex models like convolutional networks.

#### RQ6 (Quality of Synthesized Inputs)

This section examines the quality of the generated inputs for NP-SBFL-MGA and the baseline approach DeepFault. We analyze the distance between the original and synthesized images using various distance metrics, such as \(L_{1}\) Manhattan, \(L_{2}\) Euclidean, and \(L_{\infty}\) Chebyshev. Additionally, we consider the naturalness scores, including inception score (IS) [47] and Frechet Inception Distance (FID) [48], for different values of K (# suspicious neurons) in both approaches.

Table 15 presents the results for the quality of synthesized inputs over different K values. According to the table, for MNIST models, DeepFault exhibits a consistent degree of perturbation regardless of the value of K. On the other hand, NP-SBFL illustrates an increase in distances as K increases. Both approaches yield comparable IS scores. However, DeepFault outperforms NP-SBFL-MGA regarding FID values for CIFAR models, indicating a higher level of naturalness.

Table 16 compares the distances between the original and synthesized images based on different suspiciousness measures employed by various approaches. The scores for these measures are relatively close for all models in a specific approach, except for DeepFault using Ochiai over the CIFAR model, where a notable difference is observed. Specifically, the inputs synthesized by DeepFault instances using Ochiai achieve the best FID value, while those synthesized by NP-SBFL instances using Tarantula demonstrate the best FID value. In the case of MNIST, Tarantula, and Barinel exhibit the same distance for DeepFault and are lower than Ochiai.

These findings provide insights into the quality of the generated inputs and the performance of different approaches in terms of perturbation and naturalness scores. The results highlight the strengths and weaknesses of NP-SBFL-MGA and DeepFault, shedding light on their suitability for specific models and metrics.

_Table 14. A comparison between the quality of synthesized input over different K values. K: MNIST(CIFAR)._

\begin{tabular}{|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{K} & \multirow{2}{*}{Approach} & \multicolumn{3}{c|}{MNIST} & \multicolumn{3}{c|}{CIFAR} \\ \cline{3-8}  & & \(L_{1}\) & \(L_{2}\) & \(L_{\infty}\) & IS (mean) & IS (std) & FID \\ \hline \end{tabular}

Figure 11: A comparison between the ratio of common neurons in each layer of a DNN model for the best results of NP-SBFL-MGA and DeepFault.
### Threats to Validity

**Construct Validity**: When conducting experiments, there is a possibility of challenges to the accuracy of the results. Poor accuracy can occur due to various factors, including the selection of datasets and DNN models. To ensure the reliability of our research, we have taken steps to address these potential concerns. Firstly, we have used well-known and widely studied public datasets, such as MNIST and CIFAR-10. Furthermore, we have applied our NP-SBFL method to multiple DNN models with distinct architectures, which have all shown competitive prediction accuracies, as shown in Table 4. Additionally, we have incorporated established suspiciousness measures from the field of fault localization in software engineering, outlined in Algorithm 1, to mitigate any potential threats related to identifying suspicious neural pathways. These efforts contribute to the overall robustness and validity of our research findings.

**Internal Validity**: To ensure the accuracy of NP-SBFL-MGA's ability to produce new inputs that trigger suspicious neural pathways, we addressed potential threats to internal validity. We utilized various distance metrics to verify that the generated inputs closely resemble the original inputs and are comparable to those generated by DeepFault. We also took measures to prevent NP-SBFL-MGA's suspiciousness measures from accidentally outperforming the baselines. We performed a non-parametric statistical test, specifically the Wilcoxon rank-sum test [44] for statistical significance at a 95% confidence level and Vargha and Delaney's A'12 statistics [45] for the effect size measure to compare the performance of NP-SBFL-MGA and the baselines, and assess any significant differences.

**External Validity:** To address potential issues with external validity, NP-SBFL needs to be able to examine the internal structure of DNN and gather data on the activation patterns of neurons to assess their level of suspicion accurately. To achieve this, we utilized PyTorch in developing NP-SBFL, enabling comprehensive white-box analysis of DNNs. While we have examined various measures of suspicion, we acknowledge the potential existence of other measures. Furthermore, we have validated NP-SBFL against multiple instances of DNNs trained on widely-used datasets to ensure its practicality. However, further experiments are needed to assess the efficacy of NP-SBFL in different domains and networks [48].

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{Measure} & \multirow{2}{*}{Approach} & \multicolumn{4}{c|}{MNIST} & \multicolumn{3}{c|}{CIFAR} \\ \cline{3-8}  & & \(L_{1}\) & \(L_{2}\) & \(L_{\infty}\) & IS (mean) & IS (std) & FID \\ \hline \multirow{2}{*}{Tarantula} & DeepFault & **6022.74** & **334.31** & **25.37** & 1.00 & 0.00 & **46.90** \\ \cline{2-8}  & NP-SBFL-MGA & 12314.54 & 619.14 & 46.27 & 1.00 & 0.00 & 48.22 \\ \hline \multirow{2}{*}{Ochiai} & DeepFault & **7500.54** & **395.60** & **25.49** & 1.01 & 0.00 & **32.47** \\ \cline{2-8}  & NP-SBFL-MGA & 12276.75 & 618.33 & 45.80 & 1.00 & 0.00 & 49.95 \\ \hline \multirow{2}{*}{Barinel} & DeepFault & **6022.74** & **334.31** & **25.37** & 1.00 & 0.00 & **46.90** \\ \cline{2-8}  & NP-SBFL-MGA & 12261.40 & 617.59 & 45.78 & 1.00 & 0.00 & 49.96 \\ \hline \end{tabular}
\end{table}
Table 15: A comparison between the quality of synthesized input over different suspicious measures.
Conclusion

Deep Neural Networks have successfully solved complex tasks in many real-world applications, such as image classification, recognizing human speech, natural language processing, and software engineering. However, despite their high accuracy, DNNs have quality issues and cannot classify samples correctly in real-world applications. Therefore, there is a need for an effective assessment of the quality of DNNs, especially in safety- and security-critical systems. In this paper, we propose a new fault localization method called NP-SBFL that identifies critical neurons using the Layer-wise Relevance Propagation technique and then determines which critical neurons are faulty. The method's effectiveness is demonstrated on two benchmark datasets, MNIST and CIFAR-10, showing its high accuracy and efficiency in fault localization. We also propose a novel methodology to verify the detected faulty paths based on gradient ascent. Our results show that NP-SBFL is highly effective and can achieve an average 96.75% fault detection rate for all models under test. It substantially outperforms the DeepFault technique that analyzes neurons as the root cause of faults in the neural network. Moreover, the multi-stage gradient ascent used in NP-SBFL is more effective than the simple gradient ascent.

In the future, we plan to evaluate NP-SBFL on various other DNNs and datasets. Some examples of these include the steering wheel control of a self-driving car and the Udacity dataset. Additionally, we aim to enhance the suspiciousness-guided synthesis algorithm, expand the creation of synthesized inputs, and explore methods for fixing faulty neural pathways. Furthermore, we intend to design explicit criteria to confine critical pathways, improving syntenies inputs' naturalness. These endeavors are crucial in assessing the resilience of DNNs and facilitating the development of safety cases.

#### Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work the authors used ChatGPT-3.5 in order to improve language and readability of the work. After using this tool/service, the authors reviewed and edited the content as needed and takes full responsibility for the content of the publication.
