MRFI: An Open Source Multi-Resolution Fault Injection Framework for
  Neural Network Processing
To ensure resilient neural network processing on even unreliable hardware,
comprehensive reliability analysis against various hardware faults is generally
required before the deep neural network models are deployed, and efficient
error injection tools are highly demanded. However, most existing fault
injection tools remain rather limited to basic fault injection to neurons and
fail to provide fine-grained vulnerability analysis capability. In addition,
many of the fault injection tools still need to change the neural network
models and make the fault injection closely coupled with normal neural network
processing, which further complicates the use of the fault injection tools and
slows down the fault simulation. In this work, we propose MRFI, a highly
configurable multi-resolution fault injection tool for deep neural networks. It
enables users to modify an independent fault configuration file rather than
neural network models for the fault injection and vulnerability analysis.
Particularly, it integrates extensive fault analysis functionalities from
different perspectives and enables multi-resolution investigation of the
vulnerability of neural networks. In addition, it does not modify the major
neural network computing framework of PyTorch. Hence, it allows parallel
processing on GPUs naturally and exhibits fast fault simulation according to
our experiments.
Haitong Huang, Cheng Liu, Bo Liu, Xinghua Xue, Huawei Li, Xiaowei Li
# MRFI: An Open Source Multi-Resolution Fault Injection Framework for Neural Network Processing

###### Abstract

To ensure resilient neural network processing on even unreliable hardware, comprehensive reliability analysis against various hardware faults is generally required before the deep neural network models are deployed, and efficient error injection tools are highly demanded. However, most existing fault injection tools remain rather limited to basic fault injection to neurons and fail to provide fine-grained vulnerability analysis capability. In addition, many of the fault injection tools still need to change the neural network models and make the fault injection closely coupled with normal neural network processing, which further complicates the use of the fault injection tools and slows down the fault simulation. In this work, we propose MRFI, a highly configurable multi-resolution fault injection tool for deep neural networks. It enables users to modify an independent fault configuration file rather than neural network models for the fault injection and vulnerability analysis. Particularly, it integrates extensive fault analysis functionalities from different perspectives and enables multi-resolution investigation of the vulnerability of neural networks. In addition, it does not modify the major neural network computing framework of PyTorch. Hence, it allows parallel processing on GPUs naturally and exhibits fast fault simulation according to our experiments.

## I Introduction

Deep neural network models are increasingly used in safety-critical applications like autonomous driving and avionics, but computational errors caused by imperfect hardware such as process variations, defects and impurities in materials, and noise can lead to unexpected inference and cause severe consequences in these safety-critical applications. To guarantee resilient deep learning, it is crucial to thoroughly evaluate the reliability of neural network models before their deployment. While it is prohibitively expensive to perform fault injection to realistic chips, simulation based error injection tools such as PyTorchFI [1], Ares [2], TensorFlow [3], TorchFI [4], and MindFI [5] have been extensively developed. These tools typically provide basic functionalities such as error injection and reliability analysis to general neural network processing and significantly alleviate the design efforts required for neural network reliability analysis in presence of various hardware errors.

Nevertheless, it has become evident that existing fault injection tools are insufficient to address complex neural network reliability analysis for the following aspects. First, they generally utilize hardware fault models such as bit flip model to simulate the influence of hardware errors on neural network processing, but they do not model any specific hardware architectures and the correlation between the neuron-wise error injection and realistic hardware-based fault injection remains unclear. The accuracy of the fault simulation needs to be calibrated such that it can be utilized as a common metric to reliability study of neural network processing. Second, there is a lack of standard error injection metrics and neural network evaluation metrics that can be utilized to align the fault simulation results and distinct fault simulation results may be obtained from these different fault injection tools, which dramatically affects the reliability study of neural network processing. For instance, many prior fault injection tools fail to clarify the model quantization during reliability evaluation [6] and biased evaluation results are obtained because of the quantization setups. Third, many of the fault injection tools are closely coupled with the neural network frameworks such as PyTorch. The error model and error injection may vary with the different neural network layers [7], which also complicates the fault simulation. In addition, fine-grained selective fault-tolerant design [8] has also been explored and it poses great challenges to the fault simulation and evaluation of existing fault injection tools. These unique fault simulation and evaluation requirements further make the fault injection frameworks difficult to scale and port to GPUs or other parallel computing systems.

To address the above problems, we present MRFI, a highly configurable open-sourced fault injection tool for reliability investigation and evaluation of neural network processing. On the one hand, it provides unified error models, neural network quantization setups, and error injection mechanisms to ensure consistent metrics of error injection and reliability evaluation. Moreover, it is compared to typical fault injection platforms with more detailed architectural details and calibrated for more fair comparison. On the other hand, it provides a multi-resolution vulnerability investigation of neural network processing from distinct angles and granularities, which can be utilized to guide selective fault-tolerant neural network design conveniently. In addition, MRFI utilizes a tree structure for flexible error injection configurations and reliability evaluation of different neural network modules without modifying the neural network processing engines, which will not affect the underlying parallel computing engines on GPUs and can achieve the state-of-the-art fault simulation.
The contributions of this paper can be summarized as follows:

* We present an open-sourced multi-resolution fault injection framework MRFI and have it calibrated with architecture-specific fault injection frameworks.
* MRFI provides unified error injection and reliability evaluation with flexible simulation configurations. Particularly, it allows fine-grained vulnerability analysis from different angles and can be utilized for selective protection against various hardware errors.
* MRFI is built on top of PyTorch and does not affect the computing engine of PyTorch. Hence, the fault injection and reliability evaluation can be scaled to parallel computing systems like GPUs conveniently and achieve competitive simulation speed.

## II Related Work

Research on the reliability and fault tolerance of deep learning revolves around error simulation, fault evaluation, fault-tolerant model design, and fault-tolerant hardware design. Error simulation methods can be divided into hardware-level simulation and model-level simulation. Hardware-level simulation is generally more accurate but has limited generality and performance due to the need to simulate the underlying logic of specific hardware architectures. Notable works in this area include NVIDIA SASSIFI for fault injection simulation on GPUs [9] and fault simulation on FPGAs [8][10][11][12]. These error simulation methods often require tens to thousands of times the normal execution time.

On the other hand, model-level evaluation and simulation have better performance, making them widely studied in previous works. Some notable works in this area include [1][3][2][13][5][4]. Some works have shown that accurate hardware-level reliability results can be obtained through model-level simulation. For example, in Are [2], the authors obtained consistent measurements with silicon measurement by injecting errors at the model level. Yi He et al. [7] mapped hardware unit error rates to model neurons to obtain accurate model-level evaluation results. Xinghua Xue et al. [14] improved the accuracy of computational evaluation by focusing on operations. Some works aim to improve the performance of error simulation. For example, PyTorchFI [1] optimized the implementation of fault injection simulation, reducing the additional time overhead of single-point fault injection to a negligible level. The closest state-of-the-art tools to our work are PyTorchFI and Ares, which offer more comprehensive functionality compared to other works. Table I shows a summary of their features.

For fault evaluation, recent works have proposed improved statistical metrics and mathematical modeling analysis to further reduce the number of error simulations required. In HarDNN [15], the authors explored various surrogate test metrics to replace simple model accuracy measurements, resulting in a 10-times reduction in simulation count under the same simulation accuracy. BinFI [16] models error propagation and introduces binary search to reduce the number of simulations. [17] achieved a two-order-of-magnitude speedup in fault evaluation by using statistical modeling. Previous fault injection frameworks did not consider these optimizations, whereas our framework's flexibility allows researchers to conveniently perform these evaluations.

After fault evaluation, safety-critical systems often require fault-tolerant design. Fault-tolerant design can be applied at both the software and hardware levels, such as adjusting models for fault tolerance or introducing additional hardware redundancy computations [18][19][20][21][22]. These solutions include Selective Hardening [8], Algorithm-Based Fault Tolerance [23], adjustment of quantization parameter settings [6], adjustment of activation layer values to limit numerical magnitude [24], fault-tolerant retraining [25], and more. Evaluating these solutions requires a more flexible fault
injection framework.

## III MRFI Framework

### _MRFI Overview_

The overall structure of MRFI is illustrated in Figure 1. The framework as a whole can be divided into two parts: the configuration part and the execution modules, which are further divided into several functional modules.

MRFI provides two main approaches to configure different experimental requirements: EasyConfig for simple and preliminary experiments, and a detailed error injection configuration tree for more complex needs. For simple experiments, EasyConfig allows users to specify the layers and fault injection method through a config file. This configuration applies uniformly to the specified set of layers, enabling users to conduct fault injection experiments by loading the corresponding error injection configuration, similar to working with a typical PyTorch model. To address complex and precise error injection requirements without modifying the experimental model, MRFI employs a detailed error injection configuration tree. This tree records the error injection configurations for various modules and layers of a model. Users can easily adjust the parameters within the configuration tree to meet their specific needs. MRFI's error injection module then dynamically performs error injection based on the configured settings.

The execution modules of MRFI offer multiple commonly used injection methods that encompass the majority of error injection requirements for experiments. For enhanced flexibility, users can also customize the execution module according to their specific needs.

The structured configuration approach of MRFI offers several advantages over previous manual error injection implementations. In the traditional manual approach, experimenters typically need to modify both the neural network model code and the error injection parameters to accommodate different error injection requirements. This process is hard especially for modern large-scale models with numerous layers. Additionally, manually adjusting error injection parameters for multiple fine-grained settings in dozens of model layers is cumbersome, making experimental management challenging. MRFI overcomes these issues by decoupling the error injection configuration from the model definition and organizing it into separate modules, facilitating error injection experiments with specific configurations.

### _Integration with PyTorch_

MRFI is built upon PyTorch, the widely adopted neural network training framework. Error injection in MRFI is achieved through PyTorch hook, allowing for seamless integration with the model. PyTorch serves as the foundation for MRFI due to its extensive user base and well-established software ecosystem. Additionally, we have found that PyTorch's dynamic graph approach aligns well with error injection scenarios that necessitate dynamic modification and configuration. The hook function in PyTorch enables dynamic observation and modification of intermediate data within network modules, making it highly suitable for fine-grained error injection experiment configuration. An alternative implementation approach involves adding or replacing layers in the model with fault injection operations. However, this method requires prior knowledge of the original layer implementations, which is impractical given the diverse range of user-defined modules and layers. It is worth noting that error injection at the model level only requires access to the model's weights and feature map information for modification. By leveraging hooks, rewriting of the underlying computation implementation is not necessary. Our execution module also mainly uses PyTorch tensors, which allows the error injection process to be naturally used on the CPU and GPU to maximize performance.

### _Major MRFI Components_

As depicted in Figure 1, MRFI incorporates three key components for fault injection: quantizer, selector, and error model. Additionally, an observer feature is available for monitoring numerical values and their error impact within the model.

#### Iii-C1 Quantizer

In MRFI, we recognize the significance of quantization methods in evaluating fault tolerance. Quantization is commonly employed during inference for fixed-point or integer computations, differing from the floating-point storage and computation used during model training. Previous studies highlight the varying fault-tolerance characteristics of different numerical representations and the influence of quantization choices on model accuracy and fault tolerance. To facilitate accurate fault simulation for quantized models, MRFI offers multiple configurable quantization options. These simulated quantizations encompass the quantization and de-quantization steps, converting tensors from floating-point to integer format and vice versa. The quantizer serves as a wrapper for the fault injection component, enabling the provision of integer tensors to the selector and error model for simulating integer computation and error occurrence on real hardware. It is worth noting that the quantizer is optional, and users can omit it if they solely require simulations with floating-point models.

#### Iii-C2 Selector

The selector determines the locations within the tensor where errors are injected. By defining the selector, users can specify the manner in which errors manifest. For example, a fixed position selector can be employed to evaluate the impact of permanent errors. Conversely, a random position selector serves as a random engine to simulate the occurrence of soft errors. Soft errors are often associated with a bit error rate, and the random position selector efficiently generates erroneous positions based on this rate. Users also have the flexibility to customize filtering masks at the selector level or implement custom selectors to simulate fine-grained selective fault-tolerant protection.

#### Iii-C3 Error model

The error model dictates how errors are injected into the numerical values at the selected error positions. The error model may rely on the original numerical values, such as random bit flips for integers or floating-point numbers, fixed bit flips, or the addition of perturbations following a normal distribution. Alternatively, the error model may be
independent of the original numerical values, involving stuck-at 0 faults, faults with specific fixed numerical values, or random uniform integers or floating-point numbers. MRFI incorporates these common error models, while also allowing users to define custom error models conveniently.

#### Iii-B4 Observer

MRFI provides internal observers to enhance the visibility of numerical values and their error impact within the model. Observers that do not require fault injection can be employed to monitor the distribution of internal activation values. For example, an observer can track the maximum and minimum values to determine the dynamic range of activations, aiding in the determination of model quantization parameters. Another type of observer enables the comparison between fault injection and non-fault injection scenarios. Before initiating fault injection, this observer conducts a golden run comparison analysis. It can utilize a counter to tally the number of numerical values affected by fault injection or employ an RMSE observer to gauge the extent of error propagation.

## IV Experiment Results

In this section, we present the results of various error injection experiments conducted using MRFI. Our focus is on demonstrating experimental approaches not covered by previous error injection tools, highlighting the necessity of utilizing a more fine-grained error injection framework.

For our experiments, we primarily employ VGG and ResNet models evaluated on the ImageNet dataset. In subsection IV-B, we also utilize a smaller dataset, cifar-10, to visualize error propagation.

### _Basic BER-Accuracy fault inject experiment_

We begin by testing and evaluating MRFI's performance under simple configurations. Specifically, we conduct random bit flipping error injection on the quantization weight of the VGG16 model using the fixed-point quantization settings of 2.12 and 3.13. Additionally, MRFI supports different data types and quantization methods. For comparison, we include

Fig. 1: The MRFI tool integrates multiple modules. The configuration part can be used for simple and detailed experiments. The execution module provides a variety of predefined functions.

Fig. 2: Relation between bit error rates and classification error rate of different data type.
the original 32-bit floating point (f32) and layer-wise range quantization (Qlayer).

As illustrated in Figure 2(a)(b), our results of fixed-point quantization with settings 2.12 and 3.13 align with those obtained from Ares [2], exhibiting a significant increase in bit error rates at \(BER=10^{-6}\). In contrast, floating-point types exhibit lower error tolerance, while layered quantization demonstrates superior fault tolerance, corroborating the conclusions from SNR [6]. In Figure 2(c)(d), the injection of activation values exhibits a similar trend, but not entirely the same.

While the above experiments could be conducted separately using different fault injection tools in the past, the advantage of MRFI lies in its integration of diverse functionalities within a unified framework.

### _Observing error propagation_

To gain deeper insights into error propagation within the network, MRFI provides a straightforward operation to apply error observers to all network modules. These observers enhance transparency and offer diverse perspectives for in-depth research on error impact.

Firstly, we employ a simple equation comparison observer to assess the number of neurons affected by a single injection. We perform one-bit flipping on the activation values and weights of the first convolution layer (conv1) of VGG11, and then calculate the average number of errors across 10,000 input images. The golden run accuracy and accuracy after activation injection both yield \(76.49\%\), while the accuracy after weight injection slightly decreases to \(76.34\%\). As shown in Figure 3, the number of affected neurons increases rapidly along the layers, with nearly all neurons in the fully-connected layers displaying differences from their original values. This indicates the network's resilience to errors, while also highlighting that the number of affected neurons is not a good indicator of the severity of error effects.

Next, we evaluate error propagation using more robust metrics. Inspired by [15][17], MRFI provides mean absolute error (MAE) and rooted mean square error (RMSE) observers. In Figure 4, both MAE and RMSE reveal that weight injection has a significantly greater influence than activation value injection when performing single-point injection on the first convolution layer. This finding aligns with the accuracy results. Additionally, although the number of affected neurons rapidly increases during the propagation process, these error measures remain relatively stable.

MRFI also includes observers for recording activation and error values directly, facilitating the visualization of internal feature maps in neural networks. Figure 5 showcases the raw neuron values input by LeNet on a single image from the cifar-10 dataset, along with the errors caused by a single activation injection in the first layer. We observe that errors occur in fixed positions across all channels in the convolution layer's feature map, while nearly all neurons in the fully connected layers experience irregular disturbances.

### _Quantization Settings Resilience Study_

Recent studies have highlighted the close relationship between quantization methods and neural network fault tolerance. Therefore, our framework integrates quantization settings into its configuration. In subsection IV-A, we compared the differences between two quantization methods using a simple global quantization setting. With MRFI, we can now quantize each submodule of the network separately to study these differences at a fine-grained level.

Figure 6 illustrates the differences in RMSE metrics for various error injections on different layers and blocks of ResNet18. A larger RMSE value indicates greater sensitivity to errors. It is evident that global quantization settings exhibit significant variations in fault tolerance and perform poorly at each layer, resulting in lower fault tolerance than layered quantization. Conversely, layerwise quantization demonstrates stable error resilience. Additionally, we observe that weights are more sensitive to errors during global quantization, whereas weights

Fig. 4: Evaluate error propagation by better metrics.

Fig. 3: Number of affected neurons of one bit flip on first layer in VGG-11.
and activation values exhibit similar sensitivities during layer-wise quantization.

The quantization parameters of the model also impact fault tolerance. In MRFI, we employ layerwise quantization and set different quantization scaling factors for each layer of the model. We evaluate accuracy and RMSE in ResNet as an example. As shown in Figure 7, larger scale factors generally lead to worse fault tolerance.

In practical deployment scenarios, models are typically quantized after training, and the quantization parameters depend on both model accuracy and target hardware requirements. The experimental differences outlined above emphasize the need for a thorough assessment of the quantization methods' impact when deploying security-critical neural network models.

### _Evaluate Fault-Tolerant Resilience at Different Level_

MRFI's flexible configuration capabilities enable error injection analysis at various levels, providing robust support for selective fault-tolerant design. In this section, we demonstrate the differences in fault tolerance based on bit, channel and pixel.

#### Iv-D1 Bit sensitivity evaluation

We inject random errors into each bit of the float16 floating-point type and fixed-point 3.13 quantization type of the VGG-11 model, observing the model's accuracy at different injection rates. Figure 8 illustrates the model accuracy for the high-order bits, while the low-order bits demonstrate greater insensitivity to error injection. In float16, the highest bit represents the sign, followed by 5 exponent bits, and finally 10 mantissa bits. From Figure 8, we observe that high-order exponent bits are highly sensitive to errors, whereas error injection into the mantissa bits does not significantly affect the model's accuracy. The sign bit exhibits a sensitivity level lower than that of high exponent bits but higher than that of low exponent and mantissa bits. For fixed-point 3.13 quantization, a regular pattern emerges in the fault tolerance of each bit, with lower bits consistently displaying better fault tolerance than higher bits.

#### Iv-D2 Channel sensitivity evaluation

Figure 9 showcases the differences in sensitivity indicators for each channel in the activation values of the first three layers of VGG-11. The three convolutional layers contain 64, 128, and 256 feature map channels, respectively. By employing layerwise quantization parameters, the differences in fault tolerance among different layers are small. After sorting the RMSE indicators for each channel in ascending order, we observe that the variance in channel sensitivity within a layer surpasses the variance in sensitivity between layers. This suggests that selective fault-tolerant design at the channel level can yield enhanced fault tolerance outcomes.

#### Iv-D3 Pixel sensitivity evaluation

It is also possible to evaluate the pixel sensitivity differences of feature maps at the spatial level. We use MRFI to randomly inject errors into each channel of different pixels with different activation values in the ResNet two layers, resulting in the output RMSE as shown in Figure 10. We can see that pixels in the center position are usually more sensitive to errors, which may be related to the

Fig. 5: Visualize of LeNet internal layers and error propagation.

Fig. 6: The difference in error sensitivity of quantization methods at different network modules.

Fig. 7: Fault tolerance under different quantization scale parameter.
dataset features of ImageNet, as most targets are located at the center of the image.

### _Performance_

In addition to accuracy, the performance of error injection and simulation in neural networks is a crucial factor for conducting experiments. MRFI has implemented optimizations in error injection, resulting in comparable time costs between error injection and the golden run. Below, we will refer the state of the art tools PyTorchFI and Ares to conduct two similar experiments and measure the time consumption. PyTorchFI has been optimized for single point error injection, making its overhead negligible compared to forward propagation. However, PyTorchFI does not provide random error generation based on injection rate, so we compared it with the random sampling method used by Ares as the baseline.

#### Iv-E1 Single point injection

Table II presents the evaluation time for single point injection errors on GPUs and CPUs. It demonstrates that the execution time of the MRFI framework can be disregarded relative to the model's computation. This indicates that although our tool is designed for complex injection configurations, it has also achieved target performance close to PyTorchFI.

#### Iv-E2 Injection according to bit error rate

For random multi-point injection, Ares conducts acceptance-rejection sampling on each bit, while MRFI uses selectors to choose injection error positions based on the expected number calculated using Equation 1. This reduces the number of random number generations required. However, a discrepancy arises when the injection error rate is low, as \(n_{inject}\) is a discrete integer. To address this issue, MRFI adopts the Poisson sampling strategy described in Equation 2, ensuring consistent results in theory. Figure 11 compares the performance and accuracy between direct probability sampling and optimization methods. While using a selector can significantly reduce the cost of sampling calculations, it may introduce jumps at low injection rates. On the other hand, Poisson sampling maintains consistent results with the baseline sampling methodology.

\[n_{inject}=\lfloor N_{neurons}\times BER\rceil \tag{1}\]

\[N_{inject}\sim Poisson(N_{neurons}\times BER) \tag{2}\]

## V Conclusion

In this paper, we have introduced MRFI, an open-source multi-resolution fault injection framework for neural network processing. MRFI enables refined error assessments, considering the impact of model quantization and supporting fine-grained error injection configurations. To simplify configuration, MRFI decompose complex configuration requirements into multiple modules and offers a user-friendly configuration interface. Our experiments have shown that MRFI can evaluate errors from multiple aspects with practical needs.

Fig. 11: Comparison of performance and accuracy of different random sampling methods for random multi-point injection.

Fig. 8: Bit sensitivity evaluation of VGG-11 among all layers.

Fig. 10: Pixel sensitivity evaluation on two convolution layers of ResNet18 using RMSE metric.

Fig. 9: Channel sensitivity evaluation on three convolution layers of VGG-11.