Safe Use of Neural Networks
Neural networks in modern communication systems can be susceptible to
internal numerical errors that can drastically effect decision results. Such
structures are composed of many sections each of which generally contain
weighting operations and activation function evaluations. The safe use comes
from methods employing number based codes that can detect arithmetic errors in
the network's processing steps. Each set of operations generates parity values
dictated by a code in two ways. One set of parities is obtained from a
section's outputs while a second comparable set is developed directly from the
original inputs. The parity values protecting the activation functions involve
a Taylor series approximation to the activation functions. We focus on using
long numerically based convolutional codes because of the large size of data
sets. The codes are based on Discrete Fourier Transform kernels and there are
many design options available. Mathematical program simulations show our
error-detecting techniques are effective and efficient.
George Redinbo
# Safe Use of Neural Networks

###### Abstract

Neural networks in modern communication systems can be susceptible to internal numerical errors that can drastically effect decision results. Such structures are composed of many sections each of which generally contain weighting operations and activation function evaluations. The safe use comes from methods employing number-based codes that can detect arithmetic errors in the network's processing steps. Each set of operations generates parity values dictated by a code in two ways. One set of parities is obtained from a section's outputs while a second comparable set is developed directly from the original inputs. The parity values protecting the activation functions involve a Taylor series approximation to the activation functions. We focus on using long numerically-based convolutional codes because of the large size of data sets. The codes are based on DFT kernels and there are many design options available. MatLab simulations show our error-detecting techniques are effective and efficient.

Neural networks, convolutional codes, error detection, soft errors, matrix operations, activation functions, DFT-based convolutional codes, algebraic-based fault tolerance (ABFT)

## 1 Introduction

Communication systems can use neural networks in many parts as is outlined in an article describing many applications of neural networks [1]. Neural networks have many processing operaions that are susceptible to random internal numerical errors that can drastically alter their decision outputs. Safe use needs to know when errors have appeared. Networks in commercial situations on standard computing hardware are extremely reliable.

On the other hand, there are situations involving neural networks that operate in what we term, hostile environments, where radiation particles can disrupt normal numerical operations causing erroneous decisions and improper tuning. For example, remote sensing in earth orbit or on foreign planets face disruptions. Neural networks can be used in orbital control systems or in medical systems within high-energy environments. Control systems in heavy industrial locations can be influenced dramatically. This paper addresses a standard neural network configuration and proposes protective methods that can detect when errors have affected numerical calculations voiding safe use.

Neural networks appear in many forms. They all have some common operations in stages forming the network, both in a forward direction when yielding decision results and in a backward propagation needed for tuning and training the network [Chapter 5, 2]. Each stage in the network whether a forward or backward type involves weighting data, scale by coefficients and summing, or passing through activation functions, nonlinear operations with limited output range. We propose numerically based error-detecting codes wherein the code word symbols are numerical values with coding operations defined on an arithmetic field.

The purpose of this paper is to guarantee safe us by detecting errors in operations supporting neural networks. Error-detecting codes are applied to generic models of the stages in neural networks, not aimed for any specific implementation. In this way, these new methods can be appropriately modified to address any practical neural network implementation

The concerns about errors in neural networks have been expressed in many papers with quite different viewpoints and sometimes with new approaches for increasing the protection levels. There are many, many articles concerning neural networks in the literature and some of them address reliability issues. We mention some in particular that seem in the direction of results in this article.

Some papers [3-5] evaluate various architectures that mitigate the effects of single-event upset (SEU) errors in networks of various kinds. A fault-tolerant method in [6] employs training algorithms combined with error-correction codes to separate the decision result allowing errors to be more noticeable. The training procedures for this approach are complicated. A series of papers [7-10] make hardware changes to the underlying implementation devices to avoid errors. Paper [10] diversifies the decisions steps allowing better detection of errors. When memistors are employed [11-13] several clustering techniques including binary error-correcting codes on the network's outputs offer some protection.

Several papers are concerned with the impact of SEU errors on the memory system supporting the network [14-16]. One approach focuses on storage of the weight values [16]. Another article addresses modifying the activation func
tion (models of neurons) so that failures during these evaluations can be checked more easily.

After we had completed our developments of error-detection codes for protecting the weighting operations and activation function outputs, we discovered a short conference paper (2pages) [18] that started in the same direction as our approach. We recognized their approach as using algorithm-based fault tolerance (ABFT) methods [19]. Their results relied on BCH codes upgraded to real number block codes, which they applied to a three-stage network for simulation purposes. We believe our new results employing numerically based convolutional codes for both weighting actions and activation functions provide a much broader scope for protection techniques.

The next section describes the general features of most neural networks. The following section explains our novel technique for detecting numerical errors in both forward and backward stages aligning with the data flow of the network. The use of wavelet codes, convolutional codes with numbers as symbols and parities, permit the large size of the data vectors to be handled offering protection through and across the weighting and activation function computations. A special method is developed for protecting the calculations producing the new weighting matrices that are determined in the backward stages. Thus, the infrequent tuning of the network as implemented by the backward sections are protected. The last section evaluates the effectiveness of these new codes, showing unusually broad detecting performances. A short appendix outlines the design details of the new wavelet codes.

## 2 Modeling Neural Networks for Protection Evaluation

There are many different neural network configurations for implementing artificial intelligence operations. Our goal is to demonstrate methods for detecting any errors that occur in arithmetic operations in the network. Computational errors arise from the underlying electronic structures' failures such as by soft errors [3]. They are very infrequent and their sources are difficult to pinpoint but can have a devastating effect on the results of a neural network. Accordingly, we will adopt a reasonable model that considers all normal operations that appear in neural networks. We employ a model offered by a text on neural networks [Chapter 5, 2]. The common arithmetic processing features involve large arithmetic weighting operations and activation functions. All are included in Fig. 1, which depicts the weighting sections \(\mathrm{W}^{(p)}\) and activation operator \(\mathrm{A}\).

The data processed in stages are collected into vectors \(\mathrm{Y}\), dimension \(\mathrm{K}\), as will be more formally described shortly. The number of data samples passed between stages can be different sometimes, but for exposition purposes, we assume the same size through the stages. The weighting functions are implemented by a matrix \(\mathrm{W}\) with its scaling features. The outputs of the weighting operations are denoted by \(\mathrm{\underline{S}}\) whereas the outputs to the next stage are from the activation functions, each of which have a limited output range. The role of the forward stages is to produce decision variables outputs in \(\mathrm{\underline{Y}}^{(M)}\). These final outputs of the forward network yield the decision variables.

The neural network is adjusted, possibly infrequently, by the actions of the backward propagation stages that process errors between the actual outputs and the desired outputs of the forward stages. These error values are passed through the backward stages. Each backward stage processes errors from the previous stage. Then, based on the perceived errors using the outputs of a comparable indexed forward stage, a new set of weights are computed in a backward stage. These new weights will be employed in a future forward stages. In addition, the newly defined weights are used to continue through the backward propagation processing stages. This approach also detects any errors in the control operations since they ultimately lead to numerical errors.

The arithmetic operations in a typical forward stage, labeled by index \(\mathrm{p}\), out of \(\mathrm{M}\) forward stages is detailed further in Fig. 2. We consider the arithmetic operations as separate entities whether by fixed-point or floating-point nature. Thus, arithmetic values are the fundamental quantities in this work. The inputs to this stage \(\mathrm{p}\) are outputs of activation functions from the previous stage. These inputs are collected into a vector \(\mathrm{\underline{Y}}^{(p)}\), \(1\times\mathrm{K}\). They are combined with the weighting matrix \(\mathrm{W}^{(p)},\mathrm{K}\times\mathrm{K}\), yielding outputs \(\mathrm{\underline{S}}^{(p)},1\times\mathrm{K}\). The activation function \(g(\mathrm{x})\) is applied to each variable in \(\mathrm{\underline{S}}^{(p)}\) providing the outputs of stage \(\mathrm{p}\), \(\mathrm{\underline{Y}}^{(p)}\). Since the weightings are applied linearly, the processing from \(\mathrm{\underline{Y}}^{(p)}\) to \(\mathrm{\underline{S}}^{(p)}\) is a matrix-vector equation.

Figure 1: Arithmetic Stages of Neural Network
\[\underline{S}^{(p)}=\underline{Y}^{(p-1)}W^{(p)} \tag{1}\]

In a compressed notation, the outputs \(\underline{Y}^{(p)}\) are expressed by applying \(g(x)\), activation function, to each component of \(\underline{S}^{(p)}\).

\[\underline{Y}^{(p)}=g\left(\underline{S}^{(p)}\right)\quad\text{ Activation Function Outputs} \tag{2}\]

The activation function can take several nonlinear forms, e.g. tanh(x) ReLU(x), [20]. It is useful to have a good derivative since the backward stages use derivatives of the forward function. (Some other functions besides tanh(x) may not have derivatives at all places but that can be handled [20])

The forward stages produce decision variables that give the choices for the output. If the forward network is to be trained, adjusted, the decision variables are compared to the desired outputs and any discrepancies are used by the backward propagation stages to adjust the weighting values in \(W^{(p)}\)'s. Any future adjusted network will use the new values contained in the new weighting matrices \(W^{(p)ww}\). The role of the backward propagation stages is indicated in Fig. 1 by the adjustable symbol under backward stages. Keep in mind that the activation function in the backward section is the derivative of the one in the correspondingly indexed forward stage.

The update of each weighting matrix is done in two phases. The output of a backward stage, \(\underline{\delta}^{(p)}\), as indicated in Fig. 3 showing a generic backward stage p is derived using new weights called \(W^{(p)ww}\). This is to designate the new variables in backward stages Fig. 3. They are derived using the propagating error vector \(\underline{\delta}^{(p-1)}\) from the previous backward stage (p+1) and the input vector \(\underline{Y}^{(p-1)}\) saved from the comparably indexed FORWARD stage.

\[W^{(p)ww}_{j}=W^{(p)}_{j}+\eta\delta^{(p+1)}_{j}Y^{(p-1)}_{i} \tag{3}\] \[;\eta\text{ learning rate, }Y^{(p-1)}_{i}\text{forward network values}\]

The output vector \(\underline{\delta}^{(p-1)}\) is calculated using the new weights. Remember, the backward propagation stages are not used during normal decision operations.

## 3 Protecting Processing Operations

There are two major processing operations involved in every forward and backward stage (Fig. 1). The filter weighting calculations may be viewed as a large matrix structure, far larger than most matrix-vector products. The second part of each stage has a bank of activation functions. It might seem that the backward feedback which adjusts the filter weightings could eventually correct any errors introduced in the updating of the weights. However, it is easy to conjure up situations where any errors would continue through to the feedback. We propose using error-detecting codes defined over large arithmetic fields to determine when computational errors in either part of a stage are present. When errors are detected, the processing steps need to be repeated to support safe use.

### Weighting Operations

The weighting of input data in stages is effectively a vector-matrix operation. For example, the data input to stage p of forward section, Fig. 2, weights a vector \(\underline{Y}\), 1\(\times\)K, with matrix \(W,K\times\)K. The output values are placed in a K vector \(\underline{S}\).

\[\underline{S}=\underline{Y}W\quad\quad\quad\quad 1\times\text{K vector} \tag{4}\]

There can be occasional infrequent sparse numerical errors in the resulting vector caused by failures in the vector-multiply operations. (Later, we will model these errors as additive errors inserted in the components of W.)

Error-detecting codes can be employed to sense any er

Figure 3: Backward Propagation Stage of Neural Network

Figure 2: Forward Stage of Neural Network
rors in \(\underline{\mathrm{S}}\). We are considering block codes first for describing the concepts. Later, we will expand the approach to employ wavelet codes. Think of a large length code word in systematic form (data and check symbols are distinct), defined by an encoding matrix \(\mathrm{G_{s}=\left(I_{1}\,P\right)}\). The parity-generating matrix for a linear code is \(\mathrm{P}\), \(\mathrm{K\times(N\text{-}K)}\), for a length \(\mathrm{N}\) code with \(\mathrm{K}\) data positions.

We will use a methodology called algorithm-based fault tolerance (ABFT) [19] where the parity values associated with output vector \(\underline{\mathrm{S}}\) will be computed in two independent ways and then compared. One set of parity values can be developed:

\[\underline{\rho}^{\text{--}}\underline{\mathrm{S}}\mathrm{P}\qquad\qquad 1 \times\left(\text{N-}\text{K}\right)\] (5a) However, \[\underline{\mathrm{S}}\] results from using matrix \[\mathrm{W}\] applied on the inputs \[\underline{\mathrm{Y}}\]. Thus, an alternative version for the parity values can be defined. \[\underline{\rho}_{\text{--}}\underline{\mathrm{Y}}\mathrm{WP}\qquad\qquad \qquad 1\times\left(\text{N-}\text{K}\right)\] (5b) When parities in \[\underline{\rho}\] and \[\underline{\rho}_{\text{a}}\] are compared, if they disagree in even one place, errors appear in \[\underline{\mathrm{S}}\] or \[\underline{\mathrm{Y}}\] or \[\mathrm{both}\] (up to error-detecting capability of code).

The matrix (WP) appearing in developing \(\underline{\rho}_{\text{a}}\) is smaller than \(\mathrm{W}\); \(\mathrm{WP}\) is \(\mathrm{K\times(N\text{-}K)}\). It can be formed beforehand independently. The overhead in this ABFT scheme is in the calculation \(\underline{\mathrm{Y}}\mathrm{WP}\) (5b) and \(\underline{\mathrm{S}}\mathrm{P}\) (5a). Note the computation of the output \(\underline{\mathrm{Y}}\) using \(\mathrm{W}\) is already required for the forwarding data. The efficiency of the ABFT method relies on the value (N-K) being much smaller than \(\mathrm{K}\) the data size. The overall concept of ABFT is shown in Fig. 4.

The comparison of parities in \(\underline{\rho}\) and \(\underline{\rho}_{\text{a}}\) allows a small tolerance in each position, particularly when no errors are present since roundoff noise can enter the two different calculations. Of course, it is always possible to overwhelm any error-detecting structure by encountering too many errors. This is true for all error-detecting codes. No errors guarantees safe processing.

We can use the same parity-check matrix \(\mathrm{P}\) to produce parities associated with the calculation of the new weighting function in each stage in the backward direction. A generic backward stage first develops an updated weighting matrix, which is employed in the next forward stages and henceforth until the next training sessions. We can describe a method for verifying proper calculations using a generic backward stage with this new updated weighting called \(\mathrm{W^{raw}}\) with formula similar to (3).

\[\eta\] is learning factor \[\mathrm{W^{raw}=W+}\ \eta\underline{\delta}^{\mathrm{T}}\underline{ \mathrm{Y}}; \underline{\mathrm{\delta}}, 1\times\text{$\mathrm{K}$ error gradient}\] \[\underline{\mathrm{Y}}, 1\times\text{$\mathrm{K}$ input data forward}\] \[\mathrm{W}, \text{$\mathrm{K}\times\mathrm{K}$ matrix forward stage} \tag{6}\]

The parity-check matrix \(\mathrm{P}\) is applied to \(\mathrm{W^{raw}}\) generating a row of parity \(\rho_{\text{i}}^{raw}\).

\[\left(\left(\rho_{\text{i}}^{raw}\right)\right)=\mathrm{W^{raw}P}\ \ \text{$\mathrm{K \times}\left(\text{N-}\text{K}\right)$}\] (7a) Each of the \[\mathrm{K}\] rows of \[\left(\left(\rho_{\text{i}}^{raw}\right)\right)\] hold (N-K) parity values.

However, similar parity values can be computed by applying \(\mathrm{P}\) to the two parts of (6) individually and adding.

\[\left(\left(\rho_{\text{i}}^{raw}\right)\right)=\mathrm{W^{+}}\eta\underline{ \delta}^{\mathrm{T}}\underline{\mathrm{Y}}\mathrm{P} \tag{7b}\]

Note, two items in this parity equation have been calculate in the forward operation. \(\underline{\mathrm{Y}}\mathrm{P}\), \(1\times\left(\text{N-}\text{K}\right)\), was formed in the forward stage as was \(\mathrm{WP}\). Then scaling the \(\underline{\mathrm{Y}}\mathrm{P}\) by the error gradient \(\underline{\delta}^{\mathrm{T}}\) produces a matrix \(\mathrm{K\times(N\text{-}K)}\) which when adjusted by \(\eta\) and added to \(\mathrm{WP}\) yields \(\mathrm{K}\) new row vectors each \(\times\left(\text{N-}\text{K}\right)\). When \(\left(\left(\rho_{\text{i}}^{raw}\right)\right)\) rows are compared to the rows of \(\left(\rho_{\text{i}}^{raw}\right)\), any mismatch indicates a detected error in the calculation of \(\mathrm{W^{raw}}\).

\[\left(\left(\rho_{\text{i}}^{raw}\right)\right)-\left(\left(\rho_{\text{i}}^{ \text{-}}\right)\right)\] Parity Comparisons (7c)

Once each updated weighting vector is computed and checked, it is employed in the backward stage. Now checking this backward stage, its output shown generically as vector \(\underline{\sigma}\), \(1\times\text{K}\), can produce (N-K) parity values using check matrix \(\mathrm{P}\).

\[\underline{\rho}=\underline{\sigma}\mathrm{P}\] (8a) However, as before, the inputs to backward stages can be used to generate another set of comparable parities using the new weighting matrix just computed. This alternate parity vector is designated \[\underline{\rho}_{\text{a}}=\underline{\mathcal{Q}}\left(\mathrm{W^{raw}P} \right)\quad 1\times\left(\text{N-}\text{K}\right)\ \underline{\mathcal{Q}}, 1\times\text{$\mathrm{K}$ input vector} \tag{8b}\]

Fig.5 shows this ABFT method for a generic backward step.

Long error-detecting block codes have very poor performances, regardless whether finite field or numerically based forms. One way to obtain long error-detecting codes over numerical fields is through convolutional codes. We pioneered a new form of convolutional codes over the complex numbers [24]. Their construction uses the popular discrete Fourier transforms (DFT). A brief description with defining equations is contained in Appendix A.

Convolutional codes employ a sliding memory segment of data, linearly producing parity samples. This memory

Figure 4: Algorithm-Based Fault Tolerance Method

Figure 5: Algorithm-Based Fault Tolerance
usually called the constraint length of data provides error detection along the codeword stream. We propose such codes to protect the large-sized weighting operations, again employing the concept of algorithm-based fault-tolerance.

The codeword in a standard systematic form intermingles the parity samples among the data segment allowing detecting operations to proceed using a sliding memory. However, it is always possible to handle the parity parts separately. The coding structure we developed actually uses a subcode giving a slightly shorter information parameter. The underlying DFT codes have parameter n length, (k-1) information positions and a constraint length \(\mu\) (memory length parameter). There is a constraint among these parameters required.

\[\text{n}\textgreater\big{(}\mu+1\big{)}\text{(n-k);}\quad\text{n,k and u, parameter DFT code} \tag{9}\]

We change the notation for the processing and coding to match the traditional form of convolutional codes even though they are viewed here as a "block" code. The data are contained in a column vector as are the associated parity values. If the K data samples are collected in a wide vector \(\underline{\text{Y}}\) of L, (k-1) subvectors while the affiliated parity values are contained in a vector \(\underline{\rho}\), L(n-k) long

\[\underline{\text{Y}}\textgreater=\big{(}\underline{Y}_{0},\underline{Y}_{1}, \ldots,\underline{Y}_{L+1}\big{)};\quad\text{each }\underline{Y}_{1}\text{\times(k-1), }\underline{Y}\text{\times L}\text{(k-1), }\text{K=L}\text{(k-1)}\] (10a) \[\underline{\rho}=\underline{\text{Y}}\Gamma^{\text{T}}\quad;\quad\begin{array}{c}1 \times\text{L(n-k+1)}\text{\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \ \leavevmode\nobreak\ \ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\nobreak\ \ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\nobreak\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\ \leavevmode\
This is an expansion around \(\underline{S}=0\). Several coefficients in this expansion are known constants and will be designated by elements \(A_{i}\).

\[A_{i}=\frac{1}{i!}\frac{d^{i}g}{d\underline{S}^{i}}\left[{}_{\underline{S}=0} \right] \tag{17}\]

Again, using vector notation, the output vector \(\underline{Y}{=}g\left(\underline{S}\right)\) can be written as an approximation out to (m+1) terms.

\[g\left(\underline{S}\right)=\sum_{i=0}^{m}A_{i}S^{i}\ ;\text{where}\ \underline{S}^{i}=\left(S^{i}_{1},S^{i}_{2}, \ldots,S^{i}_{k}\ \right)^{\text{T}} \tag{18}\]

An alternate set of parities may be determined using the approximation to \(g(X)\) and incorporating the parity-generating matrix \(\Gamma^{\text{T}}\).

\[\underline{\rho}_{i}{=}g\left(\underline{S}\right)\Gamma^{\text{T}}=\sum_{i=0 }^{m}\left(A_{i}\Gamma^{\text{T}}\right)\underline{S}^{i} \tag{19}\]

The linearity of the expansion permits the parity matrix \(\Gamma^{\text{T}}\) to be applied to individual powers. The parity matrix guarantees that the codewords have good separation insuring detection of errors up to a certain level [25]. Nevertheless, Fig. 7 outlines the ABFT protection scheme using the Taylor series expansion.

Of course, the powers of items in vector \(\underline{S}\) have to be computed and the expansion in (16) may be the basis for computing the activation function itself.

The Taylor series expansion (18) is only valid over a limited range of the components of \(\underline{S}\). So, extending the range to develop the alternate parity calculations (19) requires approximating \(\tanh(X)\) outside \(\mid X\mid<1\). The Taylor series for \(\tanh(X)\) of with up to 10 terms is called \(g(X)\).

\[\begin{split} g(x){=}& x{-}\left(\frac{1}{3}\right) x^{3}+\left(\frac{2}{15}\right)x^{5}-\left(\frac{17}{315}\right)x^{7}\\ &+\left(\frac{63}{2835}\right)x^{9}\end{split} \tag{20}\]

On way to do this extension approximation is to assign fixed values over segments outside the good range. A reasonable approximation for \(\tanh(X)\) is given \(g_{s}(x)\).

\[g_{s}(x){=}\begin{cases}g(x)&\left|x\right|{\leq}1\\ 0.8049&1<\left|x\right|{\leq}1.25\\ 0.8767&1.25<\left|x\right|{\leq}1.5\\ 0.9233&1.5<\left|x\right|{\leq}1.75\\ 0.9527&1.75<\left|x\right|{\leq}2\\ 0.9820&2<\left|x\right|{\leq}5\\ 1&5<\left|x\right|\end{cases} \tag{21}\]

The fixed segments of range of \(X\) where constant values are defined is shown in Fig. 8. The constant values are chosen as the midpoint value between endpoints of the segment. The approximation error introduced by these segment values are on the order of \(10^{-2}\). The approximation errors are reflected into the parity calculations when \(g_{s}(x)\) is substituted in (19).

Similar protection methods can be applied for the derivative of the activation function appearing in the forward sections. The derivative of \(\tanh(X)\) is \(\tanh^{\text{T}}(X)=\left[1-\tanh^{\text{T}}(x)\right]\) which has a diminishing curve from 1 to 0 for positive values of \(X\). Also, the Taylor series for \(g^{\prime}\left(X\right)\) is extracted for the expansion for \(g(X)\) by taking derivatives inside (20).

\[\begin{split} g^{\prime}\left(\underline{S}\right)=&\\ +\frac{dg}{d\underline{S}}\left[{}_{\underline{S}=0}\ \underline{S}+\frac{1}{2!}\frac{d^{2}g}{d \underline{S}^{2}}\left|{}_{\underline{S}=0}\ \underline{S}^{2}+\cdots+\frac{1}{\left(m^{+1}\right)!}\frac{d^{m+1}g}{d \underline{S}^{m+1}}\left[{}_{\underline{S}=0}\ \underline{S}^{n}\right.\right.\\ &\left.\left.+\text{error terms}\right.\right.\end{split} \tag{22}\]

Then an approximation using segmented pieces like \(g_{s}(x)\) in (21) is easily established for protection purposes.

On the other hand, if the evaluation functions are implemented by look-up tables, a simple straightforward and effective error-detecting scheme uses duplication. Two compatible outputs from the functions employing two different look-up operations per position. This simple but brute force detection philosophy is outlined in Fig. 9.

## 4 Evaluations

Will the protection methods work, how well and what are the overhead processing costs? Two parts of the neural network stages are addressed by our detection techniques. The large weighting operations, viewed as matrix multiplication, are protected by long numerical-based error-detecting codes. In addition, the bank of activation functions can be protected by similar codes using a segmented Taylor series approximation for parity generation.

Figure 8: Approximations of \(\tanh(X)\) by Taylor Series and Constant Value Segments

Figure 7: Parity Detection Method Using Taylor Series Expansion of Activation Function
The large filtering operations are efficiently covered by DFT motivated codes. An algorithm-based fault tolerance (ABFT) methodology produces two sets of comparable parity values related to the output data weighted by matrix W. Equations (12) and (13) outline this approach relying on a parity-generating matrix \(\Gamma^{\mathrm{T}}\). (See Fig. 6) The matrix W, L (k-1)\(\times\)L(k-1), gives an output \(\mathbb{S}\) that in turn yields a set of Ln_k =L(n-k+1) parity values. The extra cost of the parity generation is a matrix-vector multiplication \(\underline{\mathrm{SI}}\Gamma^{\mathrm{T}}\). Using the number of multiplications to form the parity components in \(\rho\) (5a) as an indication of extra cost gives Ln_k (L (k-1))\({}^{2}\) multiplications. (\(\Gamma^{\mathrm{T}}\) (A-17) contains only short spans of nonzero pieces, which could greatly reduce this number.)

The other set of parities in \(\rho\), (13) results from applying the combined parity matrix \(\left(\mathrm{WT}^{\mathrm{T}}\right)\) which is smaller at L(n-k+1)\(\times\)L(k-1). This matrix can be computed off-line (not required at data processing steps). It operates on the input data in \(\underline{\mathrm{Y}}\) to matrix W. This second set of parities also needs products. L(n-k+1)\(\left(\mathrm{L(k-1)}\right)^{2}\). Thus, the parity calculations involve 2L(n-k+1)\(\left(\mathrm{L(k-1)}\right)^{2}\) extra multiplications. This is the operational overhead required to insert error-detecting codes in the stages.

We ran extensive MatLab simulations concerning the processing of data with weighting matrix W. These simulations focus on errors inserted in the numerical values during processing. The experiments considered three instances of matrix processing, one modeling the forward section and two backward sections. One of the background steps used the same matrix from a similarly indexed forward section while the other backward propagation section used the new weighting matrix W\({}^{\mathrm{m}}\) (6), resulting from the updating actions of the forward matrix W.

The simulations go through three executions of the matrix iterations with randomly generated input data for each pass. This guarantees virtually everything is random to test all operations. The total number of passes numbered in the millions. The errors in multiplications in the processing steps are modeled by randomly selecting locations in the matrix W into which random sized errors are added. The additive error values follow a Gaussian-Bernoulli probability law. The positions where these random errors are inserted are selected uniformly distributed over all positions in the matrix with independent probability \(\varepsilon\). Fig. 10 indicates the main loop for one pass of the simulation.

The operations are protected by a DFT-based convolutional code (wavelet code in Appendix A). The parameters of each code are basic length of pieces n, (k-1) information content, and number of parity values per code piece (n-k+1) with constraint length \(\mu\). The two parity sets for each section of a pass are compared and if any position differs between the two, an error is detected. (Many positions of parity could deviate, still counting as a detection.) The independent insertion probability \(\varepsilon\) was varied over 13 values ranging from \(\varepsilon\) = 10\({}^{-3}\) to \(\varepsilon\) = 0.1 and during each value, 10\({}^{7}\) passes were executed. The number of errors detected were collected for each pass. The performance of the coding was so good that no errors were missed. Thus, the probability of detection for runs in these ranges was 1.0

The simulations were performed for four different numerical-valued convolutional codes whose different parameters are given in Table 1. The sizes of the matrix W used in each suite of simulations depended on a parameter L=12 and the parameters k and n-k. The matrix W was L(k-1)\(\times\)L(k-1) and the number of parity values used in each case was L(n-k+1).

The number of errors that were detected were counted. For each data point, 4 million passes were made. The codes are intrinsically powerful so that only a few errors were missed (under 5 per pass). However, if the insertion probability \(\varepsilon\) is raised above 0.5, errors were missed because they exceeded the capabilities of the codes. The number of errors detected for each insertion probability \(\varepsilon\) and for each of the codes employed is plotted in Fig.11.

Table 1

Simulation Code Parameters

\begin{table}
\begin{tabular}{|c|c|c|c|c|} Code & k-1 & (n-k+1) & \(\mu\) & L(k-1) & L(n-k+1) \\ \hline
1 & 8 & 4 & 2 & 96 & 48 \\ \hline
2 & 11 & 4 & 2 & 132 & 48 \\ \hline \end{tabular}
\end{table}
Table 1: Simulation Code Parameters

Figure 10: Basic Simulation Loop, Filter Section
For the case of activation function processing, simulations used inserting errors on top of the normal outputs. The first parity was directly from these outputs, possibly corrupted with infrequent additive errors. The other parity set of values used the \(\tanh()\) approximation in (21), Fig. 8. A comparison of these parity values provide detection capabilities. However, this approximation increased the level of small errors appearing in this set of parities. The Taylor series part utilizes powers of input samples. It is hard to estimate its overhead because of the randomness of samples.

The look-up part of the segmented Taylor series-based approximation introduces errors when normal error-free data are processed through it. Thus, the threshold for determining if parities are mismatched has to be increased. Nevertheless, the segmented part could cause ( infrequently) miss-detections. These, in turn, can prompt some processing to be repeated after incorrectly detecting errors in this part. We simulated millions of passes with no errors present and quickly found that there was a discernable threshold above which normal error-free data processing caused no false detections. The threshold was increased by a factor of 10 over that used in the weighting simulations.

Once, we had set a threshold properly to avoid false errors being detected, simulated errors were properly detected all the time. We ran long simulations inputting random data and selecting error positions in the output places to add Gaussian errors. The results were gathered as before, and detection levels are plotted in Fig. 13. We note the number of detected errors are of course much less since there are only L(k-1) places where errors are inserted compared to the case of weighting matrix W that had \(\left(\text{L}\left(\text{k-1}\right)\right)^{2}\) places for error insertions. The detection performances, when approximations for the activation function are employed, are shown in Fig. 13.

## 5 Summary

Neural networks employ several stages that each include weighting operations and activation functions. Safe operation requires no errors appearing in any stage. The weighting parts involve a large number of arithmetic operations, multiplications and additions, which could be susceptible to infrequent numerical errors. The activation functions in the stages, which are nonlinear and have limiting outputs, can also suffer numerical errors. In all cases, errors can drastically change the final decisions at the outputs of the network.

This paper proposes and evaluates methods using numerically based error-detecting codes to sense any processing errors that can led to erroneous decisions. The elements in the code are arithmetic symbols, not bits or groups of bits as used in more familiar error-detecting codes. We explored long codes in both sections of a network stage, weighting and activation functions. The detecting procedures involve generating parity values dictated by the code in two ways; one set from the outputs and the other directly

Figure 11: Detection Values of Protected Matrix Operations

Figure 12: Simulation Activation Functions Section
from inputs to the process stage. Protecting activation functions' operations uses a segmented Taylor series to generate one set of the necessary parity pairs. We also showed a technique for detecting numerical errors in computing the updated weighting functions implemented in the backward stages.

Extensive MatLab simulations modeling errors in a hostile environment of both weighting and activation function operations show this approach is extremely effective in detecting randomly occurring errors in the processing steps. The Taylor series approximation method for generating checking parities gives good results.

## Appendix A Appendix Codes over Number Fields to "Safe Use of Neural Networks"

The proposed protection method for the major two parts,same sign depending on the size of t, even or odd. The product forward and backward processing stages, of neural networkwill be + if t is even while product is - if t is odd. Of course, configurations involve error-detecting codes. Our codes are \(\mathrm{arg}_{\texttt{n}\texttt{k}}=1\), a product of all X terms. Consequently, the sign of the defined over standard arithmetic fields. We outline heremhowary coefficients in matrix G trace back to the coefficients in details of these types of codes, block or convolutional, so as \(\mathrm{log}(\texttt{X})\).

All coding features for protecting arithmetic operations should use codes in systematic form; the information symbols are completely distinguished from the easily identified parity symbols. This requires the encoding matrix to have a particular form.

\[\mathrm{g}(\texttt{X})\mathrm{=}\mathrm{g}_{\texttt{n}\texttt{k}}\mathrm{X^{ \texttt{n}\texttt{k}}}+\mathrm{g}_{\texttt{n}\texttt{k}\texttt{l}}\mathrm{X^{ \texttt{n}\texttt{k}\texttt{l}}}+\cdots+\mathrm{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}+\mathrm{g}_{\texttt{t}}\mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{ t}}}\mathrm{\texttt{g}_{\texttt{t}}\mathrm{X^{\texttt{t}}}}\] (A-1a)

\[\mathrm{g}(\texttt{X})\mathrm{=}\mathrm{g}_{\texttt{n}\texttt{k}}\mathrm{X^{ \texttt{n}\texttt{k}}}+\mathrm{g}_{\texttt{n}\texttt{k}\texttt{l}}\mathrm{X^{ \texttt{n}\texttt{k}\texttt{l}}}+\cdots+\mathrm{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}+\mathrm{g}_{\texttt{t}}\mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}} \mathrm{\texttt{g}_{\texttt{t}}\mathrm{X^{\texttt{t}}}\mathrm{\texttt{g}_{ \texttt{t}}\mathrm{X^{\texttt{t}}}}\] (A-1b)

There is a difficultly in transferring the binary code to a real number code. In the binary field, -1=+1 so the binary coefficients in the original code when converted to the encoding matrix each 1 must be distinguished whether it is +1 or -1 in the real field. These distinctions can be made by examining the coefficients of \(\mathrm{g}(\texttt{X})\) when transferred.

The proper sign for the translating coefficients can be established by noting the generator polynomial is a product of first-degree polynomial factors in an extension field.

\[\mathrm{g}(\texttt{X})\mathrm{=}\mathrm{\prod}_{\texttt{i=0}}^{\texttt{n} \texttt{k}}\mathrm{(X\texttt{-}b_{\texttt{i}})}\mathrm{;}\mathrm{\ }\mathrm{b}_{\texttt{i}}\mathrm{\ }\mathrm{root\ in\ extension\ field}\] (A-2)

The coefficients in \(\mathrm{g}(\texttt{X})\) (A-1a) are products of the roots taken the proper number of times. For example, the conditions \(\mathrm{g}_{\texttt{n}\texttt{k}\texttt{l}}\) is a sum of all combinations of roots taken t at

\[\mathrm{a\ time:\ }\sum(-\mathrm{b}_{\texttt{i}})(-\mathrm{b}_{\texttt{i}}) \cdots(-\mathrm{b}_{\texttt{i}})\,.\] (A-3)

All items in the sum have the same sign depending on the size of t, even or odd. The product was on the BCH class that can be constructed for large lengths, 74-150 symbols. As mentioned in a recent paper [18], real num-\(\mathrm{G}_{\texttt{n}\texttt{k}}=1\), a product of all X terms. Consequently, the sign of the her codes can be declared by treating the binary bits as real numbers. This is a well-known result and all defining matrices now have real number 1's. [Theorem 2, 22] The error- detecting capabilities are preserved.

BCH binary codes [Sect, 6.5, 23] are a form of cyclic codes that is described by a generator polynomial \(\mathrm{g}(\texttt{X})\) whose coefficients in turn define a generator matrix. The generator polynomial of degree (n-k) for code of length n and information content k, has coefficients that also translate into a generator matrix G.

\[\mathrm{g}(\texttt{X})\mathrm{=}\mathrm{g}_{\texttt{n}\texttt{k}}\mathrm{X^{ \texttt{n}\texttt{k}}}+\mathrm{g}_{\texttt{n}\texttt{k}\texttt{l}}\mathrm{X^{ \texttt{n}\texttt{k}\texttt{l}}}+\cdots+\mathrm{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}\mathrm{X^{\texttt{t}}}+\mathrm{g}_{\texttt{t}}\mathrm{X^{\texttt{t }}}\mathrm{X^{\texttt{t}}}\mathrm{\texttt{g}_{\texttt{t}}\mathrm{X^{\texttt{t}}}}\] (A-1a)

\[\mathrm{g}(\texttt{X})\mathrm{=}\mathrm{g}_{\texttt{n}\texttt{k}}\mathrm{X^{ \texttt{n}\texttt{k}\texttt{l}}}+\mathrm{g}_{\texttt{n}\texttt{k}\texttt{l}} \mathrm{X^{\texttt{n}\texttt{k}\texttt{l}}}+\cdots+\mathrm{g}_{\texttt{t}} \mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}}+\mathrm{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}\mathrm{X^{\texttt{t}}}\] (A-1a)

\[\mathrm{g}(\texttt{X})\mathrm{=}\mathrm{g}_{\texttt{n}\texttt{k}}\mathrm{X^{ \texttt{n}\texttt{k}\texttt{l}}}+\mathrm{g}_{\texttt{n}\texttt{k}\texttt{l}} \mathrm{X^{\texttt{n}\texttt{k}\texttt{l}}}+\cdots+\mathrm{g}_{\texttt{t}} \mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}}+\mathrm{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{\texttt{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{\texttt{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{\texttt{g}_{\texttt{t}}\mathrm{X^{ \texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}} \mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{X^{\texttt{t}}}\mathrm{X^{ \texttt{t}}}\mathrm{X^
two subvectors.

\[\underline{x}=\left(\underline{u}\text{:}\underline{\varepsilon}\right)\text{:} \underline{\varepsilon}\text{ is 1}\times\text{n-k, parity vector}\] (A-4)

The parity check symbols are related to the data symbols through the submatrix P (A-3).

\[\underline{\varepsilon}=\text{P}\underline{u}\text{:}\text{1}\times\text{n-k, parity vector}\] (A-5)

These checking symbols are used to determine if errors have\(\underline{u}=\left(\underline{u}_{0},\ \underline{u}_{1},\ \ldots,\underline{u}_{p},\ \ldots.\right)\) data stream been added to the code vector \(\underline{x}\).

BCH codes are defined only for certain lengths and information content due to their construction using roots in a binary extension field [23]. However, the information content, factor k, can be adjusted by setting a number of the normal information positions to 0. The information content is now (k-r)ng by setting r information positions to 0. The length is also short-acting data is a checking matrix H. This checking matrix annened to (n-r). This can be described by a generator matrix likabilates all codewords \(\underline{x}\), i.e., \(\underline{0}=\text{H}\underline{x}\). When additive errors (A-1b) by selecting k-r rows of original G and removing \(r\) colorpers, the parity-checking matrix produces a syndrome vector \(\underline{S}\), which is composed of syndrome, subvectors \(\underline{S}_{\text{s}}\).

\[\begin{array}{ccccccccc}\underline{S}=\text{H}\underline{x}\\ \underline{S}=\left(\underline{S}_{0},\ \underline{S}_{1},\ \ldots,\underline{S}_{p},\ \ldots.\right)\text{;}\ \underline{S}_{p}=\left(\underline{s}_{p0},\ \underline{s}_{p1},\ \ldots.\right)\end{array}\] (A-8)

The parity-checking matrix H is formed using a finite number of submatrices so that it engages only a finite length of the codeword stream at a time. These submatrices is involved with constraint length of the code, m.

\[\begin{array}{ccccccccc}\underline{H}=\left(\begin{array}{ccccccccc}H_{0} &0&0&\cdots&&&&\\ H_{1}&H_{0}&0&0&\cdots&&&0&0\\ H_{2}&H_{1}&H_{0}&0&0&\cdots&&&\\ \vdots&&\vdots&\ddots&\ddots&&&\\ H_{n}&H_{n-1}&\cdots&\cdots&H_{1}&H_{0}&0&\cdots&0\\ 0&H_{n}&H_{n-1}&&&H_{1}&H_{0}&0&\cdots\\ 0&0&&\ddots&\ddots&&\ddots&&0\\ &&&&\ddots&&\ddots&&\ddots\\ &&&&\ddots&&\ddots&\end{array}\right)\end{array}\] (A-9)

Each group of syndrome subvectors \(\underline{S}_{\text{s}}\) in \(\underline{S}\) involves the product of a set of submatrices in H, collected as \(\text{H}_{\text{sSG}}\).

\[\begin{array}{ccccccccc}\text{H}_{\text{sSG}}=\left(\begin{array}{ccccccccc}H_{n} &H_{n-1}&\cdots&H_{1}&H_{0}\end{array}\right)\text{;}\
emerge. This class of codes, sometimes called Piret convolutional codes [24], does require a constraint among governingpressed showing two parts, data and parity.

\[\begin{array}{l}\big{(}\text{m+1}\big{)}\big{(}\text{n-k}\big{)}\text{<n}\\ \end{array}\] (A-11)

This requirement guarantees that there are enough DFT vec-with (A-16) shows the syndromes as the sum of two items, eager describing all details for constructing these types of codes is given [23] along with many other features of such codes. It is also possible to develop the generators of these DFT-based convolutional codes to have real-valued coefficients, following.

For our use, it is best to describe these codes using a polyphase representation [26] wherein the vectors and matrices of the sequences of symbols. When thematic form (A-15) indicates that the submatrices of each element in a submatrix is labeled accord-\(\Xi_{i}\), (n-k+1)\(\times\)(k-1), i=0,1,...,\(\mu\) are useful in determining paring to its location in the submatrix, the Z-transform spreadsry values. Expanding on the use of the convolutional code, the data components for this subcode can be collected into a semi-infinite vector containing well-defined pieces, subvectors each (k-1) long.

\[\underline{\text{Y}}=\left(\underline{\text{Y}}_{0},\,\underline{\text{Y}}_{ 1},\,\ldots,\underline{\text{Y}}_{\text{i}},\,\underline{\text{Y}}_{\text{i +1}},\ldotsldots\,\,\right)^{\text{T}}\ ;\ \underline{\text{Y}}_{\text{i}}\ 1\times\text{(k-1) data (A-19)}\]

The semi-infinite parity vector is composed of subvectors P, 1\(\times\)(n-k+1), each calculated using the submatrices \(\Xi_{i}\) shown shortly.

Z-P=\(\left(\underline{\text{P}}_{0},\,\underline{\text{P}}_{1},\ldots,\underline{\text{P}}_{ \text{i+1}},\ldots\,\,\right)^{\text{T}}\ ;\ \underline{\text{P}}_{\text{i}}\ 1\times\text{(n-k+1) parity}\) (A-20a)

All subvectors result from using a parity-generating matrix \(\Gamma\) with the same form as H above.

\[\Gamma=\left(\begin{array}{ccccccccc}-\Xi_{0}&0&0&\ldots&&&&\\ -\Xi_{1}&-\Xi_{0}&0&0&\ldots&&&&\\ -\Xi_{2}&-\Xi_{1}&-\Xi_{0}&0&0&\ldots&&&&\\ \vdots&&\vdots&\ddots&\ddots&&&&\\ -\Xi_{\mu}&-\Xi_{\mu+1}&\ldots&\ldots&-\Xi_{1}&-\Xi_{0}&0&\ldots\\ 0&-\Xi_{\mu}&-\Xi_{\mu+1}&&-\Xi_{1}&-\Xi_{0}&0&\ldots\\ 0&0&&\ddots&\ddots&&\ddots&&0\\ &&&&&&\ddots&&\ddots&\\ &&&&&&\ddots&&\ddots&\\ &&&&&&\ddots&&\ddots&\\ &&&&&&\end{array}\right)\]

\[\underline{\text{P}}=\Gamma\underline{\text{Y}}\] (A-20c)

Each parity subvector P involves engaging a finite number of the subvectors (A-19)\(\underline{\text{Y}}_{\text{i}}\).

\[\underline{\text{P}}_{\text{i}}=\left(\begin{array}{c}-\Xi_{0},-\Xi_{1},\ \cdots,-\Xi_{1}\\ \vdots\\ \underline{\text{Y}}_{\text{i+1}}\\ \underline{\text{Y}}_{\text{i+1}}\\ \underline{\text{Y}}_{\text{i+1}}\\ \end{array}\right)\] (A-21) \[\times\left(\mu+1\right)\times\left(\mu+1\right)\ \ \ \left(\mu+1\right)\times\text{1}\]
Note that \(\mathbb{P}_{q^{+1}}\) parity subvectors involves input subvectors \(\underline{y}_{q^{+1}}\), \(\underline{y}_{q^{+2}},^{-},\underline{y}_{q^{+1}\mu}\) which overlap those involve in, \(\underline{P}\). Hence, parity symbols are generated by matrix \(\Xi=(-\underline{z}_{0^{+}},-\underline{z}_{1},\ \cdots,-\underline{z}_{i})\)

The data samples in stream \(\underline{y}\) can be segmented into a piece-by \((\mu^{+}1)(\textbf{k}\text{-}1)\times 1\) and then applied to matrix \(\Xi\). If there are errors in this segment of \(\underline{y}\), the resulting parity values will not match those computed in a different way and method. The error detection process can proceed as segments of the observed data-progress.

The ABFT technique in the text uses a modified weighting matrix combining \(\Gamma\) and weighting matrix \(\mathbb{W}\), as \(\Gamma\mathbb{W}\). The network structure of \(\Gamma\) with the submatrices in \(\Xi\) appearing in limited parts of \(\Gamma\) clearly visible in (A-20b) means there are many fewer multiplications really in forming \(\Gamma\mathbb{W}\). There are only \((\mu^{+}1)(\textbf{k}\text{-}1)\) columns nonzero when applying \(\Gamma\) to \(\mathbb{W}\).
