#  Winograd Convolution: A Perspective from Fault Tolerance

Xinghua Xue1, Haitong Huang1, Cheng Liu1, Ying Wang1, Tao Luo2, Lei Zhang1
1Institute of Computing Technology, Chinese Academy of Sciences,
University of Chinese Academy of Sciences, Beijing, China
2Institute of High Performance Computing, A*STAR
{xuexinghua, liucheng, wangying2009<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Winograd convolution is originally proposed to reduce the computing overhead
by converting multiplication in neural network (NN) with addition via linear
transformation. Other than the computing efficiency, we observe its great
potential in improving NN fault tolerance and evaluate its fault tolerance
comprehensively for the first time. Then, we explore the use of fault
tolerance of winograd convolution for either fault-tolerant or energy-
efficient NN processing. According to our experiments, winograd convolution
can be utilized to reduce fault-tolerant design overhead by 27.49% or energy
consumption by 7.19% without any accuracy loss compared to that without being
aware of the fault tolerance.

††copyright: none

##  1\. Introduction

Deep neural network (DNN) has been demonstrated to be successful in numerous
domains of applications such as computer vision, natural language processing,
and data analytics (et al., 2017d). It attracts worldwide attentions and
substantial research efforts have been devoted to improving DNNs from the
perspective of precision, performance, energy efficiency, and so on (et al.,
2020b). Fault tolerance is also a vital DNN metric yet has generally been
overlooked (et al., 2018f).

Although DNNs are inherently fault-tolerant to errors or noise (et al., 2017e)
(Temam, 2012) (et al., 2016b) (et al., 2015a), improving the fault tolerance
of DNNs remains highly demanded as it can potentially bring in multi-folded
benefits. First of all, the underlying computing engines fabricated on silicon
are not perfect and can suffer increasing soft errors because of the
continuously lower power supply and higher transistor density over the years
(et al., 2017a) (et al., 2019e). The influence of soft errors on the DNN
execution is almost unavoidable. They can cause dramatic accuracy drop and
even catastrophic consequences for safety-critical DNN applications such as
autonomous driving and medical diagnosis (et al., 2021d). Although soft errors
can be mitigated with fault-tolerant hardware designs, fault tolerant DNN
models can greatly alleviate or even completely avoid the fault-tolerant
hardware design overhead (et al., 2020h) (et al., 2018a). Second, when DNNs
are applied to applications like entertainment that are insensitive to the
soft errors, the fault tolerance of DNNs can also be explored to relax the
requirements of absolute correctness of the DNN execution, which can improve
the energy efficiency substantially with techniques such as aggressive voltage
scaling(et al., 2016b) (Tu Fengbin, 2018), overclocking (et al., 2015a) (et
al., 2019c), and model pruning (et al., 2020b). Thereby, improving the fault
tolerance of DNNs is beneficial in general and required for safety-critical
applications particularly.

A few work have been proposed to improve the fault tolerance of DNNs from
different angles. The authors in (et al., 2018b) (et al., 2021d) opted to
mitigate soft errors in neural network accelerators with in-situ model
retraining. Despite the fault tolerance to the targeted fault configurations,
they may require additional retraining to adapt to a different fault scenario.
The authors in (et al., 2020h) (et al., 2021b) utilized network architecture
search (NAS) to search both high-precision and fault-tolerant neural networks.
Basically, they have fault tolerance and precision combined into a unified
metric for the network candidate selection during NAS. Hence, the network
design space shrinks because of the more stringent design constraints, and the
resulting DNN models usually get more complex and induce additional overhead.
According to their experiments, the searched neural network models have quite
some duplicated cells. Jack Kosaian et al. (et al., 2021a) proposed a
lightweight algorithm-based fault tolerance (ABFT) approach for error
mitigation of neural network processing on GPUs. Elbruz Ozen et al. (Alex,
2019) proposed Sanity-Check, which employs spatial and temporal checksums to
protect fully-connected and convolutional layers in deep neural networks.
Although these approaches improve the DNN fault tolerance, they generally
require additional computing overhead or compromise the model generality.

Unlike prior work, we aim to enhance the inherent fault tolerance of DNNs
without compromising other features of the models. With the observation that
bit flip errors in input operands of multiplication typically can cause more
severe computing errors over the golden outputs and multiplication is
generally more sensitive to the soft errors compared to addition, we envision
that winograd convolution which is originally proposed to reduce the computing
overhead by reducing the number of multiplications (et al., 2016a) is
potentially beneficial to the fault tolerance of DNN execution. As winograd
convolution offers an efficient way to conduct the DNN computing almost for
free (et al., 2017c) (et al., 2018e) (et al., 2018h) (et al., 2018c), we
investigate the fault tolerance of winograd-based DNN computing
comprehensively in this work. In order to evaluate the fault tolerance of
winograd-based DNNs, we develop an operation-level fault injection approach
rather than reusing existing neuron-level fault injection methods (et al.,
2020i) (et al., 2020a) (et al., 2018f) which can hardly differentiate the
standard convolution and winograd convolution due to the lack of fault
simulation of details in each neuron computing.

On top of the fault tolerance evaluation of winograd convolution, we further
explore two different ways to take advantage of the fault tolerance brought by
winograd convolution. One of them is to reduce DNN fault-tolerant design
overhead. Specifically, we propose a fine-grained triple modular redundancy
(TMR) protection approach and demonstrate the reduced DNN fault-tolerant
design overhead under different fault configurations. The other one is to
reduce DNN energy consumption. Basically, we can conduct more aggressive
voltage scaling of computing engines while the timing errors induced by the
voltage scaling can be tolerated by the winograd convolution. Hence, the
energy consumption can be reduced without any model accuracy loss.

The contributions of this work can be summarized as follows.

  * •

We discover and investigate the fault tolerance of winograd-based DNNs
comprehensively for the first time and explore the use of the enhanced fault
tolerance of DNNs for either soft error mitigation or computing energy
reduction.

  * •

We propose a fine-grained TMR-based redundancy approach for DNNs to make best
use of winograd convolution fault tolerance. According to our experiments,
winograd convolution with being aware of the fault tolerance shows 61.21% and
27.49% less computing overhead compared to standard convolution and winograd
convolution without being aware of the inherent fault tolerance respectively.

  * •

We take advantage of the winograd convolution fault tolerance to enable more
aggressive supply voltage scaling of a typical deep learning accelerator.
According to our experiments, this approach achieves 42.89% and 7.19% energy
reduction compared to voltage scaling with standard convolution and winograd
convolution without being aware of the inherent fault tolerance.

##  2\. Background and Related Work

###  2.1. Fault Tolerance of Neural Networks

Soft errors that are usually caused by high-energy particles hitting on
electronic devices can lead to malfunction and considerable prediction
accuracy loss of neural networks that eventually rely on the silicon-based
computing engines such as neural network accelerators, GPUs, and CPUs.
Although conventional fault-tolerant design approaches such as triple modular
redundancy, ECC, and algorithm-based fault tolerance (ABFT) (et al., 2021a)
(Alex, 2019) can potentially alleviate the influence of soft errors, they
usually require considerable computing overhead. In contrast, it is more
flexible and cost-effective to take advantage of the inherent fault tolerance
of the neural networks and alleviate the fault-tolerant design overhead at
least. The authors in (et al., 2019a) (et al., 2017f) (et al., 2019b) explored
fault tolerance of neural networks with retraining. Prasenjit Dey et al. (et
al., 2017b) proposed to penalize the system errors with regularizing terms to
make MLP robust to various errors such as link failures, multiplicative noise,
and additive noise. Elbruz Ozen et al. (et al., 2021c) also proposed a novel
regularization term to manipulate the parameter distributions at each layer
and tighten error margins. The resulting models show improved resilience
noticeably against soft errors in the neural network variables. Unlike the
work that retain the model architectures, the authors in (et al., 2021b) (et
al., 2020h) proposed to take fault tolerance into consideration in neural
architecture search (NAS) framework and obtain a new network architecture that
fulfills both the fault tolerance metric and accuracy metric at the same time.
More fault-tolerant neural network design approaches can be found in recent
surveys (Sparsh, 2020) (et al., 2020l) (et al., 2017d). In summary, we can
conclude that improving the fault tolerance of neural networks usually takes
only minor or moderate overhead and thus is an effective way to mitigate soft
errors in silicon especially when compared to expensive conventional fault-
tolerant approaches.

Neural network fault tolerance can also be utilized to improve the performance
or energy efficiency of neural network processing other than the soft error
mitigation as demonstrated in prior work (et al., 2016b) (Tu Fengbin, 2018)
(et al., 2019c) (et al., 2015a) (et al., 2019d) (et al., 2020j). For instance,
Fengbin Tu et al. (Tu Fengbin, 2018) proposed to take eDRAM with relaxed
retention as the data buffers of neural network accelerators for lower energy
consumption while the data corruption induced by the aggressive data retention
is tolerated by the neural network models. Thibaut Marty et al. (et al.,
2020j) adopted timing speculation coupled with lightweight error detection as
an approach to improve the performance of hardware accelerators for CNNs.
Brandon Reagen et al. (et al., 2016b) explored the inherent neural network
fault tolerance to lower the neural network energy consumption by scaling the
SRAM supply voltage. These approaches generally achieve significant energy
saving or performance improvement by relaxing the requirements of absolute
correctness of the neural network processing with only minor or no accuracy
drop of the neural networks. These work demonstrate the great potential of
inherent fault tolerance of the neural networks.

###  2.2. Winograd Convolution

Winograd convolution converts the matrix multiplication in standard
convolution into element-wise multiplication by linearly transforming the
input feature map and convolution kernels to a different domain of data
representation. The element-wise multiplication results can be restored to the
standard feature map domain with the corresponding inverse linear
transformation. With the transform-calculation-inverse transformation process,
the number of multiplication operations is considerably reduced and replaced
with more cost-effective addition operations compared to standard convolution,
which significantly enhances the computing efficiency. A basic two-dimensional
winograd convolution is formulated in Equation 1. $Y$ denotes the output
feature, $d$ denotes the input feature, $g$ denotes the filter, $G$ denotes
the linear transformation matrix, $B$ denotes the input transformation matrix,
and $A$ denotes the output transformation matrix. $G$, $B$, and $A$ are all
constant matrices. When winograd convolution is conducted on small windows
including 3×3 filter with unit stride, there will be no accuracy penalty. Even
when the convolution filter and stride are larger, they can also be split to
small ones according to the decomposable winograd method proposed in (et al.,
2020e). In this case, winograd convolution can be processed without any
accuracy penalty. Winograd convolution has been explored from various angles
such as quantization (et al., 2020g) (et al., 2020d), tiling (et al., 2019f),
hardware acceleration (et al., 2018d), and (et al., 2017c) for efficient DNN
computing. As far as we know, there is still a lack of investigation of
winograd convolution from the perspective of fault tolerance.

(1) $Y=A^{T}[[Gg_{k,c}G^{T}]\odot[B^{T}d_{c,b}B]]A$

In this work, we aim to investigate the fault tolerance of winograd-based
convolution neural networks and take advantage of the improved neural network
fault tolerance for either soft error mitigation or energy saving.

##  3\. Fault Tolerance of Winograd DNNs

###  3.1. Fault Injection Platform

Winograd convolution is conducted in layer wise and does not change the value
of the neurons in DNNs with lossless conversion. As a result, neural network
fault injection platforms such as TensorFI (et al., 2020a) and PyTorchFI (et
al., 2020i) that generally have bit errors injected to neurons and weights can
not differentiate the influence of soft errors on neural networks processed
with standard convolution and winograd convolution. FIdelity (et al., 2020f)
has more hardware details considered, but it does not support winograd
convolution yet. To that end, we propose an operation-level fault injection
platform that has random soft errors injected to the results of primitive
operations i.e. multiplication and addition in the neural network processing.
It is implemented on top of PyTorch and open sourced on github. It focuses on
computing of neural network processing rather than a specific computing
engine.

To illustrate the precision of the proposed operation-level fault injection
platform, we take VGG19 quantized with 16bit fixed point on CIFAR-100 dataset
as an example and evaluate the model accuracy when different number of bit
flip errors are injected. Bit error rate that denotes the probability of a bit
flip in an operation is utilized as the soft error metric. We have the model
accuracy of VGG19 with both standard convolution and winograd convolution
evaluated on both neuron-level fault injection platform and operation-level
fault injection platform when the bit error rate ranges from 7E-11 to 9E-10.
The experiment result is shown in Figure 1. It can be seen that there is no
accuracy difference between standard convolution and winograd convolution
under different bit error rate while the difference is clearly observed when
the fine-grained operation-level fault injection platform is adopted. On the
other hand, the general trend of the model accuracy under different bit error
rate obtained from the two different fault injection platforms remains
consistent in spite of the fault simulation accuracy difference.

Figure 1. Comparison of the neuron-level fault injection (FI) and the
operation-level FI.

###  3.2. Fault Tolerance Evaluation

With the operation-level fault injection platform, we investigate the fault
tolerance of winograd DNNs from different angles. First of all, we evaluate
the neural network accuracy calculated with standard convolution and winograd
convolution respectively under distinct bit error rate and present an end-to-
end fault tolerance comparison of the different computing methods of neural
networks. Second, we conduct a layer-wise fault tolerance evaluation of the
neural networks with standard convolution and winograd convolution
respectively to investigate the winograd induced fault tolerance improvement
across the different layers. Third, we evaluate the influence of soft errors
on different types of operations in neural network processing with both
standard convolution and winograd convolution. Finally, we conclude the fault
tolerance comparison of the two different calculation methods.

####  3.2.1. Evaluation Setup

In this evaluation, we have DenseNet169 on ImageNet, ResNet50 on ImageNet,
VGG19 on CIFAR-100, and GoogleNet on CIFAR-10 utilized as the benchmark neural
networks. Each neural network is quantized to a 8bit fixed point version and a
16bit fixed point version respectively. The bit error rate ranges from 0 to
1E-7 to cover the entire model accuracy drop process of the benchmark neural
networks.

####  3.2.2. Network-wise Fault Tolerance Evaluation

Figure 2. Accuracy of benchmark neural networks calculated with standard
convolution and winograd convolution.

We evaluated and compared the accuracy of the benchmark neural networks
implemented with standard convolution and winograd convolution under different
bit error rate. The experiment result is shown in Figure 2. It can be observed
that neural network implemented with winograd convolution generally shows
significant higher accuracy compared to that implemented with standard
convolution. The accuracy improvement of winograd convolution relative to
standard convolution as shown in dotted line in this figure reveals that the
improvement can be up to 35% and it holds true despite the network
architectures, data width, and bit error rate. Nevertheless, the benefits can
vary. For instance, models quantized with int16 are more vulnerable than that
with int8 because bit filp for int16 can cause larger data variation on
average. The network architectures also lead to dramatic difference. DenseNet
and ResNet on top of the same dataset exhibit distinct accuracy improvement
using winograd convolution. DenseNet shows rather sharp model accuracy
improvement while ResNet shows more smooth model accuracy improvement. Hence,
more fine-grained fault analysis is required to understand the fault tolerance
of winograd DNNs comprehensively.

####  3.2.3. Layer-wise Fault Tolerance Evaluation

To gain insight of the fault tolerance of winograd DNNs, we conduct layer-wise
fault analysis of the neural networks with both winograd convolution and
standard convolution, which is also an important basis for selective model
protection. To characterize a layer fault tolerance of neural networks, we
take VGG19 on CIFAR-100 as an example and conduct fault simulation of two
different configurations. In one configuration, we have random bit errors
injected to operations of the entire neural network. On the other
configuration, we have random bit errors injected to operations of the entire
neural networks except the layer under evaluation. The model accuracy obtained
with the first fault simulation is denoted as baseline. The model accuracy
difference of that obtained with the second fault simulation relative to the
baseline represents the fault sensitivity of the layer under evaluation.
Larger fault sensitivity indicates that higher model accuracy can be recovered
from the baseline when the evaluated layer is fault-free. Hence, the
corresponding layer is more critical to the fault tolerance of the entire
neural network. The layer-wise model accuracy of the entire neural network
implemented with both standard convolution and winograd convolution can be
obtained with the same fault simulation approach. They are denoted as ST-Conv,
WG-Conv respectively while the baselines are denoted as ST-Conv-Base and WG-
Conv-Base accordingly.

The experiment result is shown in Figure 3. It reveals that the centering
layers of the neural network are more sensitive to the bit errors while the
layers in the beginning and the end of the network are less sensitive. It is
probably because of the difference in operation number involved in each layer,
as the layer-wise model accuracy is roughly consistent with the number of
multiplication operations involved in each layer according to Figure 3.
Basically, the layers with more operations are likely to induce higher
accuracy improvement when they are fault-free. In general, the layer-wise
accuracy of the neural network implemented with winograd convolution is
generally much higher than that implemented with standard convolution. The
trend of the layer-wise accuracy is roughly consistent for neural networks
implemented with both standard convolution and winograd convolution.

Figure 3. Accuracy of VGG19 on CIFAR-100 with one fault-free layer while the
rest of layers are injected with bit error rate 3E-10. We have the neural
network implemented with standard convolution and winograd convolution
respectively. The base accuracy refers to the occasion when all the neural
network layers are injected with the same bit error rate.

####  3.2.4. Operation Type Fault Tolerance Evaluation

As winograd gre-atly changes the number of multiplication and addition in
neural networks, we also investigate the fault tolerance from the perspective
of operation types. The evaluation metric is similar to that of layer-wise
accuracy. Assume that the entire model is exposed to random bit errors, then
the model accuracy when the multiplication operations are kept fault-free can
be used to measure the sensitivity of the multiplication operations to bit
errors. Higher accuracy indicates that these operations are more vulnerable
and need to be protected with higher priority. Similarly, the sensitivity of
operations in different neural networks under different bit error rate can be
obtained. The experiment result is shown in Figure 4. Note that ST-Conv-Add
and ST-Conv-Mul represent addition and multiplication in standard convolution
respectively while WG-Conv-Add and WG-Conv-Mul represent addition and
multiplication in winograd convolution accordingly. The experiment result
shows that multiplication operations are more vulnerable than addition
operations in all the different neural networks under various bit error rate.
This also holds true for both standard convolution and winograd convolution.
In addition, we notice that the accuracy of WG-Conv-Add with more addition
included is generally higher than that of ST-Conv-Add but it remains much less
important to the accuracy of the entire neural network. In contrast, the
accuracy of WG-Conv-Mul with much less multiplications is still comparable to
that of ST-Conv-Mul, which reveals that winograd convolution is more cost-
effective for protection.

Figure 4. Accuracy of neural networks with fault-free addition or fault-free
multiplication under different bit error rate. Neural network implemented with
standard convolution and winograd convolution are considered as well.

##  4\. Exploring Winograd Fault Tolerance

We already demonstrate significant fault tolerance improvement with winograd
convolution and the internal difference in terms of layers and operation types
in subsection 3.2. We further illustrate how the winograd convolution fault
tolerance can be explored for more efficient fault tolerance and higher energy
efficiency.

###  4.1. Using Winograd for Fault Tolerance

TMR is a key strategy to mitigate soft errors in silicon. Unlike prior layer-
wise TMR (et al., 2021d), we propose a fine-grained heuristic TMR instead. The
basic idea is to select the most vulnerable layer of a neural network with
layer-wise vulnerability factor but protecting only a fraction of the
operations in the layer rather than the entire layer. Note that the
vulnerability factor of each layer is defined as the accuracy improvement when
it is fault-free over when it is faulty. It can be obtained from the analysis
in subsubsection 3.2.3. The operations to be selected is randomly chosen to
ensure that the proposed TMR protection approach can be implemented
efficiently on various computing engines. According to the analysis in
subsubsection 3.2.4, multiplication operations are much more vulnerable. Thus,
we have multiplication operations selected with higher priority. The TMR
protection is performed iteratively and a fraction of the operations will be
protected in each iteration until the design goal i.e. accuracy is fulfilled.

We have the proposed TMR protection approach applied to three different neural
network implementations including standard convolution (ST-Conv), winograd
convolution without being aware of the fault tolerance (WG-Conv-W/O-AFT), and
winograd convolution with being aware of the fault tolerance (WG-Conv-W/AFT).
Note that WG-Conv-W/O-AFT evaluates the vulnerability factor of each layer
with standard convolution and utilizes the same TMR protection option with ST-
Conv because it is not aware of the fault tolerance of winograd convolution.
The major difference between WG-Conv-W/O-AFT and ST-Conv is that WG-
Conv-W/O-AFT conducts the neural network processing and protection on top of
winograd convolution while ST-Conv conducts on standard convolution. In
contrast, WG-Conv-W/AFT conducts both the layer-wise vulnerability factor
analysis and TMR on top of winograd convolution. VGG19 quantized with int16 on
CIFAR-100 is used as the benchmark example and its original model accuracy is
72.6%. The bit error rate is set to be 3E-10. Since the TMR protection is
essentially a design trade-off between TMR overhead and model accuracy. We set
the model accuracy as the design goal and it ranges from 45% to 70% in this
experiment. Then, we evaluate the TMR design overhead for the different design
goals in this experiment. The TMR overhead is normalized to that of ST-Conv.
The experiment result is presented in Figure 5. It can be observed that WG-
Conv-W/O-AFT that greatly reduces the number of multiplication with winograd
convolution achieves the same accuracy with much less TMR overhead. The
benefit is mainly attributed to both the much less operations to be protected
and part of the inherent fault tolerance of winograd convolution because of
the lack of accurate layer vulnerability evaluation for the winograd
convolution. In contrast, WG-Conv-W/AFT that fully explores the winograd
convolution fault tolerance further reduces the TMR overhead by 27.49% on
average.

Figure 5. Normalized TMR overhead with different model accuracy requirements
when bit error rate is 3E-10.

###  4.2. Using Winograd for Energy Efficiency

We can also utilize neural network fault tolerance in combination with voltage
scaling of the underlying computing engines for more energy-efficient
processing (et al., 2020c) (et al., 2018i) (et al., 2020k) (et al., 2018k).
Basically, neural network fault tolerance relaxes the requirement of absolute
computing of neural networks and we can have the computing engines for neural
network processing working at lower voltage as long as the lower voltage
induced timing errors and computing result variations can be tolerated by the
neural networks. We take a typical neural network accelerator proposed in (et
al., 2018i) and VGG19 quantized with int16 for the experiment. The accelerator
enables voltage scaling from 0.9V to 0.7V at 667MHz clock. The relationship
between voltage and bit error rate as well as model accuracy is presented in
Figure 6. Note that the voltage is constrained between 0.77V and 0.82V to
highlight the major difference of interest in this experiment. We apply
voltage scaling to three different implementations including standard
convolution (ST-Conv), winograd convolution without being aware of the fault
tolerance (WG-Conv-W/O-AFT), and winograd convolution with being aware of the
fault tolerance (WG-Conv-W/AFT) under different model accuracy loss
constraints ranging from 1% to 10%. Note that WG-Conv-W/O-AFT is a
straightforward implementation of ST-Conv. The energy consumption depends on
both power consumption of the accelerator according to (et al., 2018i) and the
runtime of the neural network processing which can be estimated with a
simulator modified on top of Scale-Sim (et al., 2018g).

The energy consumption of the three different implementations is further
normalized to a standard convolution implementation without voltage scaling
denoted as baseline which has the voltage set to be 0.9V. The comparison is
shown in Figure 7. It can be seen that ST-Conv i.e. standard convolution
taking advantage of its inherent fault tolerance with voltage scaling shows
significant energy reduction compared to the baseline. We further explored the
use of winograd convolution fault tolerance to improve the energy efficiency
of the original DNN processing through voltage scaling. When we have ST-Conv
implemented with winograd directly which is essentially WG-Conv-W/O-AFT,
substantial energy reduction is obtained because of the reduced multiplication
with winograd transformation, and reduce power consumption according to (et
al., 2018j). Nevertheless, we can see that the energy consumption can be
further reduced by 7.19% on average according to the comparison between WG-
Conv-W/O-AFT and WG-Conv-W/AFT. In summary, exploring the fault tolerance of
winograd convolution enables further voltage scaling and additional energy
reduction.

Figure 6. Bit error rate of the neural network accelerator and model accuracy
of VGG19 on CIFAR-100 under different voltage scaling setups. Figure 7.
Voltage scaling assisted energy consumption of VGG19 quantized with int16 on
CIFAR-100 implemented with ST-Conv, WG-Conv-W/O-AFT, and WG-Conv-W/-AFT under
different accuracy loss constraints.

##  5\. Conclusion

This paper discovers and studies the fault tolerance of winograd convolution
that has been generally overlooked comprehensively for the first time. With
this evaluation, we further explore the use of winograd fault tolerance for
hardware-efficient fault-tolerant design and energy-efficient neural network
processing respectively. For the hardware-efficient fault tolerance, we
propose a novel fine-grained TMR protection approach which protects the neural
network processing with layer-wise vulnerability analysis yet operation-wise
TMR redundancy. The experiment reveals that winograd convolution fault
tolerance can be utilized to reduce the TMR overhead by 61.21% and 27.49% on
average compared to standard convolution and that without being aware of the
winograd convolution fault tolerance respectively. For the energy-efficient
neural network processing, we take advantage of the winograd convolution fault
tolerance to enable more aggressive supply voltage scaling of a typical neural
network accelerator. The experimental result demonstrates 42.89% and 7.19%
energy reduction compared to standard convolution and that without being aware
of winograd convolution fault tolerance respectively.

## References

  * (1)
  * Alex (2019) Ozen Elbruz Alex. 2019\.  Sanity-Check: Boosting the Reliability of Safety-Critical Deep Neural Network Applications. In _2019 IEEE 28th Asian Test Symposium (ATS)_. 7–75.  https://doi.org/10.1109/ATS47505.2019.000-8
  * et al. (2011) Avirneni Naga et al. 2011\.  Low overhead soft error mitigation techniques for high-performance and aggressive designs.  _IEEE Trans. Comput._ 61, 4 (2011), 488–501.
  * et al. (2017a) Bohr Mark T et al. 2017a.  CMOS scaling trends and beyond.  _IEEE Micro_ 37, 6 (2017), 20–29.
  * et al. (2020a) Chen Zitao et al. 2020a.  TensorFI: A flexible fault injection framework for TensorFlow applications. In _2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 426–435.
  * et al. (2015a) Deng Jiacnao et al. 2015a.  Retraining-based timing error mitigation for hardware neural networks. In _2015 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 593–596.
  * et al. (2020b) Deng Lei et al. 2020b.  Model compression and hardware acceleration for neural networks: A comprehensive survey.  _Proc. IEEE_ 108, 4 (2020), 485–532.
  * et al. (2017b) Dey Prasenjit et al. 2017b.  Regularizing multilayer perceptron for robustness.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 48, 8 (2017), 1255–1266.
  * et al. (2019a) Dutta Sanghamitra et al. 2019a.  CodeNet: Training large scale neural networks in presence of soft-errors.  _arXiv preprint arXiv:1903.01042_ (2019).
  * et al. (2020c) Du Yuxuan et al. 2020c.  An Energy-Efficient Time-Domain Binary Neural Network Accelerator with Error-Detection in 28nm CMOS. In _2020 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS)_. IEEE, 70–73.
  * et al. (2020d) Fernandez Marques et al. 2020d.  Searching for winograd-aware quantized networks.  _arXiv preprint arXiv:2002.10711_ (2020).
  * et al. (2020e) Huang Di et al. 2020e.  DWM: a decomposable winograd method for convolution acceleration. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 4174–4181.
  * et al. (2019b) Hacene Ghouthi et al. 2019b.  Training modern deep neural networks for memory-fault robustness. In _2019 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 1–5.
  * et al. (2018a) He Xin et al. 2018a.  AxTrain: Hardware-oriented neural network training for approximate inference. In _Proceedings of the International Symposium on Low Power Electronics and Design_. 1–6.
  * et al. (2020f) He Yi et al. 2020f.  Fidelity: Efficient resilience analysis framework for deep learning accelerators. In _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 270–281.
  * et al. (2018b) Jia Kaige et al. 2018b.  Calibrating process variation at system level with in-situ low-precision transfer learning for analog neural network processors. In _Proceedings of the 55th Annual Design Automation Conference_. 1–6.
  * et al. (2018c) Jia Zhen et al. 2018c.  Optimizing N-dimensional, winograd-based convolution for manycore CPUs. In _Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_. 109–123.
  * et al. (2021a) Kosaian Jack et al. 2021a.  Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs.  _arXiv preprint arXiv:2104.09455_ (2021).
  * et al. (2016a) Lavin Andrew et al. 2016a.  Fast algorithms for convolutional neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4013–4021.
  * et al. (2020g) Li Guangli et al. 2020g.  Lance: efficient low-precision quantized winograd convolution for neural networks based on graphics processing units. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 3842–3846.
  * et al. (2017c) Lu Liqiang et al. 2017c.  Evaluating fast algorithms for convolutional neural networks on FPGAs. In _2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_. IEEE, 101–108.
  * et al. (2018d) Lu Liqiang et al. 2018d.  SpWA: An efficient sparse winograd convolutional neural networks accelerator on FPGAs. In _Proceedings of the 55th Annual Design Automation Conference_. 1–6.
  * et al. (2019c) L. Li et al. 2019c.  Squeezing the Last MHz for CNN Acceleration on FPGAs. In _2019 IEEE International Test Conference in Asia (ITC-Asia)_. 151–156.  https://doi.org/10.1109/ITC-Asia.2019.00039
  * et al. (2019d) Liu Shanshan et al. 2019d.  Exploiting asymmetry in eDRAM errors for redundancy-free error-tolerant design.  _IEEE Transactions on Emerging Topics in Computing_ (2019).
  * et al. (2017d) Liu Weibo et al. 2017d.  A survey of deep neural network architectures and their applications.  _Neurocomputing_ 234 (2017), 11–26.
  * et al. (2020h) Li Wenshuo et al. 2020h.  FTT-NAS: Discovering Fault-Tolerant Neural Architecture. In _2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC)_. 211–216.  https://doi.org/10.1109/ASP-DAC47756.2020.9045324
  * et al. (2018e) Liu Xingyu et al. 2018e.  Efficient sparse-winograd convolutional neural networks.  _arXiv preprint arXiv:1802.06367_ (2018).
  * et al. (2020i) Mahmoud Abdulrahman et al. 2020i.  Pytorchfi: A runtime perturbation tool for dnns. In _2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W)_. IEEE, 25–31.
  * et al. (2020j) Marty Thibaut et al. 2020j.  Safe Overclocking for CNN Accelerators Through Algorithm-Level Error Detection.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ 39, 12 (2020), 4777–4790.
  * et al. (2021b) Ning Xuefei et al. 2021b.  FTT-NAS: Discovering fault-tolerant convolutional neural architecture.  _ACM Transactions on Design Automation of Electronic Systems (TODAES)_ 26, 6 (2021), 1–24.
  * et al. (2021c) Ozen Elbruz et al. 2021c.  SNR: Squeezing Numerical Range Defuses Bit Error Vulnerability Surface in Deep Neural Networks.  _ACM Transactions on Embedded Computing Systems (TECS)_ 20, 5s (2021), 1–25.
  * et al. (2019e) Pandey Pramesh et al. 2019e.  GreenTPU: Improving Timing Error Resilience of a Near-Threshold Tensor Processing Unit. In _2019 56th ACM/IEEE Design Automation Conference (DAC)_. 1–6.
  * et al. (2016b) Reagen Brandon et al. 2016b.  Minerva: Enabling low-power, highly-accurate deep neural network accelerators. In _2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 267–278.
  * et al. (2018f) Reagen Brandon et al. 2018f.  Ares: A framework for quantifying the resilience of deep neural networks. In _2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC)_. IEEE, 1–6.
  * et al. (2020k) Rathore Mallika et al. 2020k.  Error Probability Models for Voltage-Scaled Multiply-Accumulate Units.  _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ 28, 7 (2020), 1665–1675.
  * et al. (2018g) Samajdar Ananda et al. 2018g.  Scale-sim: Systolic cnn accelerator simulator.  _arXiv preprint arXiv:1811.02883_ (2018).
  * et al. (2018h) Shen Junzhong et al. 2018h.  Towards a uniform template-based architecture for accelerating 2D and 3D CNNs on FPGA. In _Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_. 97–106.
  * et al. (2019f) Shen Junzhong et al. 2019f.  Toward an efficient deep pipelined template-based architecture for accelerating the entire 2-D and 3-D CNNs on FPGA.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ 39, 7 (2019), 1442–1455.
  * et al. (2020l) Shafique Muhammad et al. 2020l.  Robust Machine Learning Systems: Challenges,Current Trends, Perspectives, and the Road Ahead.  _IEEE Design Test_ 37, 2 (2020), 30–57.  https://doi.org/10.1109/MDAT.2020.2971217
  * et al. (2017e) Torres-Huitzil Cesar et al. 2017e.  Fault and error tolerance in neural networks: A review.  _IEEE Access_ 5 (2017), 17322–17341.
  * et al. (2015b) Tan Li et al. 2015b.  Investigating the interplay between energy efficiency and resilience in high performance computing. In _2015 IEEE International Parallel and Distributed Processing Symposium_. IEEE, 786–796.
  * et al. (2018i) Whatmough Paul et al. 2018i.  DNN engine: A 28-nm timing-error tolerant sparse deep neural network processor for IoT applications.  _IEEE Journal of Solid-State Circuits_ 53, 9 (2018), 2722–2731.
  * et al. (2018j) Xygkis Athanasios et al. 2018j.  Efficient winograd-based convolution kernel implementation on edge devices. In _2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC)_. IEEE, 1–6.
  * et al. (2021d) Xu Dawen et al. 2021d.  R2F: A Remote Retraining Framework for AIoT Processors With Computing Errors.  _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ 29, 11 (2021), 1955–1966.  https://doi.org/10.1109/TVLSI.2021.3089224
  * et al. (2018k) Zhang Jeff et al. 2018k.  Thundervolt: enabling aggressive voltage underscaling and timing error resilience for energy efficient deep learning accelerators. In _Proceedings of the 55th Annual Design Automation Conference_. 1–6.
  * et al. (2017f) Zhao Lei et al. 2017f.  AEP: An error-bearing neural network accelerator for energy efficiency and model protection. In _2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_. IEEE, 1047–1053.
  * Sparsh (2020) Mittal Sparsh. 2020\.  A survey on modeling and improving reliability of DNN algorithms and accelerators.  _Journal of Systems Architecture_ 104 (2020), 101689.
  * Temam (2012) Olivier Temam. 2012\.  A defect-tolerant accelerator for emerging high-performance applications. In _2012 39th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 356–367.
  * Tu Fengbin (2018) et al. Tu Fengbin. 2018\.  RANA: Towards efficient neural acceleration with refresh-optimized embedded DRAM. In _2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 340–352.