Noise robust neural network architecture
In which we propose neural network architecture (dune neural network) for
recognizing general noisy image without adding any artificial noise in the
training data. By representing each free parameter of the network as an
uncertainty interval, and applying a linear transformation to each input
element, we show that the resulting architecture achieves decent noise
robustness when faced with input data with white noise. We apply simple dune
neural networks for MNIST dataset and demonstrate that even for very noisy
input images which are hard for human to recognize, our approach achieved
better test set accuracy than human without dataset augmentation. We also find
that our method is robust for many other examples with various background
patterns added.
Xiong Yunuo, Xiong Hongwei
# Noise robust neural network architecture

###### Abstract

In which we propose neural network architecture (dune neural network) for recognizing general noisy image without adding any artificial noise in the training data. By representing each free parameter of the network as an uncertainty interval, and applying a linear transformation to each input element, we show that the resulting architecture achieves decent noise robustness when faced with input data with white noise. We apply simple dune neural networks for MNIST dataset and demonstrate that even for very noisy input images which are hard for human to recognize, our approach achieved better test set accuracy than human without dataset augmentation. We also find that our method is robust for many other examples with various background patterns added.

Keywords: Noisy image recognition, neural networks, robustness, noise immunity, additive white gaussian noise
Introduction

The problem of devising neural network architectures that can recognize noisy image without artifically adding any noise in the training process has been long standing in the field of deep learning [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14]. For example, it has been observed that state-of-the-art neural networks for ImageNet failed completely when we add small and special noises to the test images (the so-called adversarial examples [15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27]), and no efficient solutions have been found to tackle this issue and the presence of general noises. Therefore, noisy image recognition remains an active area of research [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27] and in this work, we propose a different neural network architecture which takes the effect of noises explicitly into account.

In the dune neural network we present here, each free parameter of the model is represented as an uncertainty interval. At each training iteration, we sample from dune neural network using Monte Carlo method and propagate through the sampled network in the usual manner, then uncertainty intervals in the original dune network are dynamically updated in response to the loss derivatives. The advantages of using dune neural networks are twofold: first, the uncertainty interval representation of free parameters tends to make the model easier to optimize, due to how finite width trajectories make the exploration of loss function landscape easier, as contrast to other "probabilistic" models such as Bayesian neural network [28], where model optimization proved to be much harder than optimization for standard architectures. Second, dune neural networks naturally encode robustness with respect to parameter perturbation, namely, all parameters in the given region defined by the uncertainty intervals should yield networks with similar performance. This property helps ensure that the training process is more stable as well.

Inspired by the notion of uncertainty interval for dune neural networks, we also propose applying a linear transformation to each element of the noisy input data. This transformation does not eliminate the noise added to input data in any way, but maps the input uncertainty interval into a new uncertainty interval that achieves excellent noise robustness when fed into dune neural network. When we combine the input transformation with dune neural networks, we show that without dataset augmentation, the network realized high accuracy when evaluated on the noisy test set. Specifically, without dataset augmentation, in the examples we consider here, the noisy test set accuracy decreases roughly exponentially with increasing noise to signal ratio for traditional method with simple neural network;
whereas for our approach here, the noisy test set accuracy only decreases approximately linearly. Therefore, we are able to achieve reasonable accuracy for very noisy images that are otherwise hard for a human to recognize, by training on the original training set only without the use of dataset augmentation. The work we present here may have potential value for the field of noisy image recognition and deep learning noise problem in general, and in particular may have a wide range of practical applications where we do not know or can not train the noises/disturbing background patterns in advance, e.g., in medical image recognition [29] and astronomical image classification [30; 31].

In the following, we give detailed account of dune neural network and its associated transformation of noisy input data, then we perform tests on MNIST dataset for simple neural network architectures to demonstrate the power of our approach to combat noisy data and disturbing background pattern.

## II Method

### Architecture

Dune neural network converts each free parameter in an ordinary neural network into an uncertainty interval represented by two separate parameters. Let \(\theta\) denotes the collection of free parameters from the original network, then the corresponding dune neural network has parameters \(\mathbf{\Theta}\) where \([\Theta_{i1},\Theta_{i2}]\) is the uncertainty interval corresponding to the \(i\)th free parameter \(\theta_{i}\). So if the original network has \(n\) parameters, the associated dune neural network has \(2n\) parameters. Whenever we need to perform any operations on the dune neural network, we start by treating the uncertainty intervals as uniform distributions and sampling to obtain a set of real values, then construct an ordinary neural network based on those values:

\[\tilde{\theta_{i}}\in[\Theta_{i1},\Theta_{i2}]. \tag{1}\]

This is called the instantiation step.

To perform forward propagation on dune neural network, we simply carry out the propagation on the sampled, ordinary neural network. For example, the loss function can be written as

\[L_{\mathbf{\Theta}}(\tilde{\theta_{1}},...,\tilde{\theta_{n}}). \tag{2}\]
To update the uncertainty intervals of a dune neural network dynamically based on the loss function, we first calculate the gradient with respect to the sampled variables in the usual manner with backward propagation:

\[\mathbf{g}=\nabla_{\tilde{\theta}}L_{\mathbf{\Theta}}(\tilde{\theta_{1}},..., \tilde{\theta_{n}}). \tag{3}\]

Then we may use any optimization algorithm (stochastic gradient descent, momentum, Adam, and so forth) we want to calculate an update on \(\tilde{\theta}\) values based on the gradient and other relevant hyperparameters. Let us denote that step as

\[\tilde{\theta}^{(new)}=optimization\_algorithm(\tilde{\theta}^{(old)},\mathbf{g},...). \tag{4}\]

Then the parameters in the corresponding dune neural network are updated as

\[\Theta_{i1}^{(new)}=\Theta_{i1}^{(old)}+(1-p)(\tilde{\theta_{i}}^{(new)}- \tilde{\theta_{i}}^{(old)}),\]

\[\Theta_{i2}^{(new)}=\Theta_{i2}^{(old)}+p(\tilde{\theta_{i}}^{(new)}-\tilde{ \theta_{i}}^{(old)}), \tag{5}\]

where \(p\) is defined as

\[p=\frac{\tilde{\theta_{i}}^{(old)}-\Theta_{i1}^{(old)}}{\Theta_{i2}^{(old)}- \Theta_{i1}^{(old)}}. \tag{6}\]

In case \(\Theta_{i2}^{(old)}=\Theta_{i1}^{(old)}\) and \(p\) is undefined, we use the following rule to update \(\mathbf{\Theta}\)

\[\Theta_{i1}^{(new)}=\Theta_{i1}^{(old)}+(\tilde{\theta_{i}}^{(new)}-\tilde{ \theta_{i}}^{(old)}),\]

\[\Theta_{i2}^{(new)}=\Theta_{i2}^{(old)}+(\tilde{\theta_{i}}^{(new)}-\tilde{ \theta_{i}}^{(old)}). \tag{7}\]

This ensures that when there are no uncertainty intervals (i.e. \(\Theta_{i2}^{(old)}=\Theta_{i1}^{(old)}\)), the usual parameter update rule for deterministic neural networks is recovered.

The dynamics of uncertainty intervals in dune neural network resembles the movement of a dune in high dimensional space, hence the name. At each training iteration of dune neural network, we resample the parameter values from each uncertainty interval for better statistics. During testing process, we make a single sample from the dune neural network and treat the output of that sampled ordinary network as the final output. Usually, there is little gain from making multiple samples for a single iteration of dune neural network which decreases the efficiency. In its current form, dune neural networks have the same efficiency as traditional deterministic neural networks, and in deeper networks, dune neural networks
have a good chance to converge to a solution faster than other networks under the same circumstances, due to the introduction of uncertainty intervals making it easier to explore the cost function landscape.

To initialize a dune neural network, we first obtain a collection of parameters \(\theta\) for an ordinary network from any initialization scheme. Then we specify a prior uncertainty width \(d\) and \(\mathbf{\Theta}\) are set as

\[\Theta_{i1}=\theta_{i}-d,\ \Theta_{i2}=\theta_{i}+d. \tag{8}\]

Also, to prevent any uncertainty interval from growing too large during training, we can add a regularization term for the width of each uncertainty interval to the loss function, as follows:

\[L^{(reg)}=L+\sum_{i=1}^{n}\beta(\Theta_{i2}-\Theta_{i1})^{2}, \tag{9}\]

where \(\beta\) is a hyperparameter similar to the one from weight decay. And to prevent the uncertainty interval from growing too small, which would recover a deterministic network, we can impose a hard threshold for the smallest allowed uncertainty interval width, by specifying another hyperparameter \(w_{min}\). If after an update is made, an uncertainty interval has a width less than \(w_{min}\) (i.e. \(\Theta_{i2}-\Theta_{i1}<w_{min}\)), then we readjust that uncertainty interval as

\[\Theta_{i1} =\frac{\Theta_{i1}+\Theta_{i2}}{2}-\frac{w_{min}}{2},\] \[\Theta_{i2} =\frac{\Theta_{i1}+\Theta_{i2}}{2}+\frac{w_{min}}{2}. \tag{10}\]

### Data representation

In image recognition, we normalize the input data so that each pixel is represented as a single real value in the interval \(x\in[0,1]\) for grayscale images. For color images each pixel would be represented as three numbers each in the interval \([0,1]\). In this work, we first consider white noise as applied to the input images, by using this formula to add noise to all pixels in the image:

\[x^{noisy}=\frac{x^{old}}{n_{s}+1}+\frac{n_{s}}{n_{s}+1}U(0,1), \tag{11}\]

where \(U(0,1)\) denotes a random number from the uniform distribution \([0,1]\).

It is easy to see that the noise-to-signal ratio in the above formula is \(n_{s}\), the larger \(n_{s}\) the more noise is applied to the image. Regardless of whether there are noises present in
the image, each pixel is always restricted to lie in the interval \([0,1]\). Based on the idea of dune neural network, we may treat the range of pixel values \([0,1]\) as an uncertainty interval too. In particular, we propose to apply the same linear transformation to each pixel value, before feeding those values as input to the dune neural network, with the following formula:

\[\tilde{x}^{noisy}=(1+2h_{s})x^{noisy}-h_{s}. \tag{12}\]

It is straightforward to see that \(\tilde{x}^{noisy}\) now lies in the interval \([-h_{s},1+h_{s}]\).

It is also worthy to note that the above transformation does not eliminate or weaken the noise in the noisy input data in any way, since the transformation is applied after the noise adding procedure given by Eq. (11). The \(h_{s}\) in the above equation is to be treated as a hyperparameter specified by the user. We will see later that after applying the transformation to the input data, the capability of dune neural network to recognize noisy images is dramatically improved. We call the input data representation as defined by Eq. (12) magic shift (abbreviated Magics).

In the following, we apply the approach introduced in this section to MNIST dataset with noisy test images. We show that even without dataset augmentation, our architecture and data representation achieved excellent noise robustness compared with previous methods [13; 9; 1].

## III Results

We test our approach on MNIST dataset for simple dune neural networks. Specifically, we use dune network counterpart of simple fully connected network for image classification to verify the principle of our method, with ReLU activation and softmax classifier. We do not use dataset augmentation so the training set always remains the same, and when we need to add noise to the testing set, we add white noise using Eq. (11) (\(n_{s}=0\) corresponds to the case of no noise).

To begin with, we check that the combination of dune neural networks and Magics is working correctly, by monitoring the accuracy on the original testing set (without noise). The result is shown in Fig. 1; as expected, the accuracy converges toward 100% at a stable pace.

Then we turn our attentions to the case of noisy test set. In Fig. 2, we tested dune
Figure 1: Test set (without noise) accuracy as a function of epochs for two layer fully connected dune neural network of hidden units 150 and 10. We used Adam optimization algorithm and the learning rate is 0.001, \(\beta=0.1\) and \(w_{min}=0.15\). In this simple setting, both the cases with Magics (\(h_{s}=0.5\)) and without Magics (\(h_{s}=0\)) converge toward 100% accuracy.

Figure 2: Noisy test set accuracy (\(n_{s}=1.5\), no dataset augmentation) as a function of epochs for two layer fully connected dune neural network of hidden units 150 and 10, for different \(w_{min}\) values. The common hyperparameters are: learning rate 0.001, \(\beta=0.1\), \(h_{s}=1.8\). We can see that using different \(w_{min}\) values can have a slight effect on the test set accuracy, with \(w_{min}=0.15\) corresponding to the optimal choice. In practice one may need to experiment with different \(w_{min}\) to find the best choice.
neural network with Magics on noisy test set with \(n_{s}=1.5\), without dataset augmentation. We give results for different \(w_{min}\) values, we can see that \(w_{min}=0.15\) gives optimal result.

Next, we investigate the effects of Magics on the noisy test set accuracy, without dataset augmentation. As we mentioned before, the combination of dune neural network and Magics can improve the ability to recognize noisy images dramatically, so we try different Magics hyperparameter \(h_{s}\) (\(h_{s}=0\) is the special case with no Magics). The result is shown in Fig. 3, we can clearly see the improvement in noisy test set accuracy by using positive \(h_{s}\) values. As \(h_{s}\) increases the accuracy begins to converge.

We note that the traditional method with simple neural network can be combined with Magics as well. To compare the noise robustness of these different approaches, we plot the noisy training set accuracy versus noise-to-signal ratio for three methods, as shown in Fig. 4. As shown by the yellow line based on the traditional method, the test set accuracy decreases exponentially with increasing noise-to-signal ratio. This exponential decay is also observed in many other works [3; 4] with more complex neural network, e.g. for Gaussian blur and Gaussian noise in Ref. [3]. One of the most challenging problem in noisy image recognition

Figure 3: Noisy test set accuracy (\(n_{s}=1.5\), no dataset augmentation) as a function of epochs for two layer fully connected dune neural network of hidden units 150 and 10, for different \(h_{s}\) values. The common hyperparameters are: learning rate \(0.001\), \(\beta=0.1\), \(w_{min}=0.15\). \(h_{s}=0\) corresponds to the case without Magics and the network performs poorly. By using positive \(h_{s}\) values, we see that there is a dramatical improvement in the noisy test set accuracy. With large enough \(h_{s}\), the accuracy converges to about 91%.
is to eliminate this exponentially decaying behavior, which is successfully realized in Fig. 4, as shown by blue line based on the combination of dune neural network and Magics, and the green line based on the combination of the traditional method and Magics.

We also pick a specific case and plot the accuracy as a function of epochs for these three different methods, as shown in Fig. 5. We can see that the accuracy of traditional method decreases quickly with epochs due to overfitting, whereas the two methods with Magics both achieved decent noise robustness and did not overfit over the training course of 30 epochs.

Figure 4: Noisy test set accuracy as a function of noise-to-signal ratio \(n_{s}\) (no dataset augmentation) for two layer fully connected dune neural network of hidden units 150 and 10, for different methods. The hyperparameters are: learning rate \(0.001\), \(\beta=0.1\), \(w_{min}=0.15\). For \(n_{s}\geq 1\), we set \(h_{s}=1.8\), otherwise we set \(h_{s}=2n_{s}\). Traditional method shown by yellow line (without dune neural network and without Magics) has a poor performance: with increasing noise-to-signal ratio the test set accuracy decreases exponentially. The combination of dune neural network and Magics (shown by blue line) achieves the best performance, while the combination of ordinary neural network and Magics (shown by green line) doing slightly worse. The dashed line in the figure corresponds to the threshold of human’s ability to recognize noisy images, we show a selection of images from MNIST for different \(n_{s}\) in the bottom of the figure.
We emphasize that in our method there is no artificial noise added in the training dataset for recognizing test images with added white noise. Hence, it is expected that our method can be applied to other types of noise or background pattern. To support this conjecture, we also performed experiments on different types of noises and found that the combination of dune neural network and Magics achieved decent noise robustness for all the different noises we considered. A selection of images with various disturbing background patterns along with the test set accuracy are shown in Fig. 6. The improvement on the test set accuracy of noisy image recognition is significant, compared with previous results with more complex methods [1; 9; 13].

Figure 5: Noisy test set accuracy (\(n_{s}=1.5\), no dataset augmentation) as a function of epochs for two layer fully connected dune neural network of hidden units 150 and 10, for different methods. The hyperparameters are: learning rate 0.001, \(\beta=0.1\), \(w_{min}=0.15\). For the two methods with Magics, we set \(h_{s}=1.8\).
## IV Conclusion

As a summary, in this work we propose a noise robust neural network architecture, based on the combination of dune neural network and Magics as applied to the input data. We tested on MNIST for different noises and noise strength to demonstrate the improvement of the current method over traditional method. In particular, we achieved decent noisy test set accuracy even without dataset augmentation, showing that our method is indeed noise robust for noisy image recognition.

We believe the method present here has a broad applicability when faced with general noise problem in the field of deep learning. For example, all existing neural network architectures can be promoted to their dune neural network version without much extra difficulty, so our approach can naturally be applied to tasks other than image classification in the future.

Figure 6: The same image with different types of noises applied to it, the upper left corner is the original image with added border. At the top right corner of each image, we show the noisy test set accuracy (without dataset augmentation), using the combination of dune neural network and Magics. A decent noise robustness is achieved in each case, showing that our method here can be used to deal with a variety of different noises and disturbing background patterns.
Moreover, a possible future direction is to gain a deeper understanding of Magics, by applying it to other kinds of noisy input data to see if we can achieve noise robustness in other applications. The purpose of the present work is not to consider adversarial examples [15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27], which may deserve further studies based on our method.

**Acknowledgments** The authors would like to acknowledge support from Hubei Polytechnic University.

**Author Contributions** YN designed the study, proposed the idea, implemented the methods and wrote the manuscript. HW performed some numerical experiments and revised the manuscript.

**Funding** This work is partly supported by the National Natural Science Foundation of China under grant numbers 11175246, and 11334001.

**Declaration of competing interest**

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

**Availability of data and material** The data that support the findings of this study are available from the corresponding author upon reasonable request.

**Declarations**

**Confact of interest** The authors declare no conflicts of interest.

**Ethics approval** Not Applicable.

**Consent to participate** Not Applicable.

**Consent for publication** Not Applicable.

**Code availability** All code associated with this paper is publicly available from [https://github.com/xiongyunuo/XNN](https://github.com/xiongyunuo/XNN)
