Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed  perturbations which are imperceptible to humans but can cause the network to give incorrect outputs. In this paper, we design a new CNN architecture that by itself has good robustness. We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures. We show that CNN with Random Mask achieves state-of-the-art performance against black-box adversarial attacks without applying any adversarial training. We next investigate the adversarial examples which “fool” a CNN with Random Mask. Surprisingly, we find that these adversarial examples often “fool” humans as well. This raises fundamental questions on how to define adversarial examples and robustness properly. Deep learning (LeCun et al., 2015) , especially deep Convolutional Neural Network (CNN) (LeCun et al., 1998) , has led to state-of-the-art results spanning many machine learning fields, such as image classification BID12 BID15 Huang et al., 2017; Simonyan & Zisserman, 2014) , object detection (Redmon et al., 2016; BID7 Ren et al., 2015) , image captioning (Vinyals et al., 2015; Xu et al., 2015) and speech recognition BID1 BID13 .Despite the great success in numerous applications, recent studies have found that deep CNNs are vulnerable to some well-designed input samples named as Adversarial Examples (Szegedy et al., 2013) BID2 . Take the task of image classification as an example, for almost every commonly used well-performed CNN, attackers are able to construct a small perturbation on an input image to cause the model to give an incorrect output label. Meanwhile , the perturbation is almost imperceptible to humans. Furthermore , these adversarial examples can easily transfer among different kinds of CNN architectures (Papernot et al., 2016b) .Such adversarial examples raise serious concerns on deep neural network models as robustness is crucial in many applications. Just as BID8 suggests , both robustness and traditional supervised learning seem fully aligned. Recently, there is a rapidly growing body of work on this topic. One important line of research is adversarial training (Szegedy et al., 2013; Madry et al., 2017; BID9 Huang et al., 2015) . Although adversarial training gains some success, a major difficulty is that it tends to overfit to the method of adversarial example generation used at training time BID3 . Xie et al. (2017) and BID11 propose defense methods by introducing randomness and applying transformations to the inputs respectively. BID5 introduces random drop during the evaluation of a neural network. However, BID0 contends that such transformation and randomness only provide a kind of "obfuscated gradient" and can be attacked by taking expectation over transformation (EOT) to get a meaningful gradient. Papernot et al. (2016a ) and Katz et al. (2017) consider the non-linear functions in the networks and try to achieve robustness by adjusting them. There are also detection-based defense Our main contributions are summarized as follows:• We develop a very simple but effective method, Random Mask. We show that combining with Random Mask, existing CNNs can be significantly more robust while maintaining high generalization performance. In fact, CNNs equipped with Random Mask achieve state-of-the-art performance against several black-box attacks, even when comparing with methods using adversarial training (See Table 1 ).• We investigate the adversarial examples generated against CNNs with Random Mask. We find that adversarial examples that can "fool" a CNN with Random Mask often fool humans as well. This observation requires us to rethink what are the right definitions of adversarial examples and robustness. In conclusion, we introduce and experiment on Random Mask, a modification of existing CNNs that makes CNNs capture more information including the pattern of feature distribution. We show that CNNs with Random Mask can achieve much better robustness while maintaining high test accuracy. More specifically, by using Random Mask, we reach state-of-the-art performance in several black-box defense settings. Another insight resulting from our experiments is that the adversarial examples generated against CNNs with Random Mask actually change the semantic information of images and can even "fool" humans. We hope that this finding can inspire more people to rethink adversarial examples and the robustness of neural networks. A RANDOM SHUFFLE Figure 5 : An example image that is randomly shuffled after being divided into 1 × 1, 2 × 2, 4 × 4 and 8 × 8 patches respectively.In this part, we show results of our Random Shuffle experiment. Intuitively, by dropping randomly selected neurons in the neural network, we may let the network learn the relative margins and features better than normal networks. In randomly shuffled images, however, some global patterns of feature distributions are destroyed, so we expect that CNNs with Random Mask would have some trouble extracting feature information and might have worse performance than normal networks. In order to verify our intuition, we compare the test accuracy of a CNN with Random Mask to that of a normal CNN on randomly shuffled images. Specifically speaking, in the experiments, we first train a 0.7-Shallow network along with a normal network on ImageNet dataset. Then we select 5000 images from the validation set which are predicted correctly with more than 99% confidence by both normal and masked networks. We resize these images to 256 × 256 and then center crop them to 224 × 224. After that, we random shuffle them by dividing them into k × k small patches k ∈ {2, 4, 8}, and randomly rearranging the order of patches. Figure 5 shows one example of our test images after random shuffling. Finally, we feed these shuffled images to the networks and see their classification accuracy. The results are shown in Table 3 . DISPLAYFORM0 Normal ResNet-18 99.58% 82.66% 17.56% 0.7-Shallow 97.36% 64.00% 11.94% Table 3 : The accuracy by using normal and masked networks to classify randomly shuffled test images.From the results, we can see that our network with Random Mask always has lower accuracy than the normal network on these randomly shuffled test images, which indeed accords with our intuition. By randomly shuffling the patches in images, we break the relative positions and margins of the objects and pose negative impact to the network with Random Mask since it may rely on such information to classify. Note that randomly shuffled images are surely difficult for humans to classify, so this experiment might also imply that the network with Random Mask is more similar to human perception than the normal one.
We propose a technique that modifies CNN structures to enhance robustness while keeping high test accuracy, and raise doubt on whether current definition of adversarial examples is appropriate by generating adversarial examples able to fool humans. This paper proposes a simple technique for improving the robustness of neural networks against black-box attacks. The authors propose a simple method for increasing the robustness of convolutional neural networks against adversarial examples, with surprisingly good results.