Neural networks are vulnerable to small adversarial perturbations. Existing literature largely focused on understanding and mitigating the vulnerability of learned models. In this paper, we demonstrate an intriguing phenomenon about the most popular robust training method in the literature, adversarial training: Adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution. Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarial trained model that is both trained and evaluated on the new distribution. Our discovery of such sensitivity on data distribution is based on a study which disentangles the behaviors of clean accuracy and robust accuracy of the Bayes classifier. Empirical investigations further confirm our finding. We construct semantically-identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve comparable clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies. This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves. Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon. Neural networks have been demonstrated to be vulnerable to adversarial examples BID22 BID3 . Since the first discovery of adversarial examples, great progress has been made in constructing stronger adversarial attacks BID12 BID18 BID17 BID6 . In contrast, defenses fell behind in the arms race BID5 BID1 . Recently a line of works have been focusing on understanding the difficulty in achieving adversarial robustness from the perspective of data distribution. In particular, BID24 demonstrated the inevitable tradeoff between robustness and clean accuracy in some particular examples. BID20 showed that the sample complexity of "learning to be robust" learning could be significantly higher than that of "learning to be accurate".In this paper, we contribute to this growing literature from a new angle, by studying the relationship between adversarial robustness and the input data distribution. We focus on the adversarial training method, arguably the most popular defense method so far due to its simplicity, effectiveness and scalability BID12 BID13 BID15 BID17 BID8 . Our main contribution is the finding that adversarial robustness is highly sensitive to the input data distribution:A semantically-lossless shift on the data distribution could result in a drastically different robustness for adversarially trained models.Note that this is different from the transferability of a fixed model that is trained on one data distribution but tested on another distribution. Even retraining the model on the new data distribution may give us a completely different adversarial robustness on the same new distribution. This is also in sharp contrast to the clean accuracy of standard training, which, as we show in later sections, is insensitive to such shifts. To our best knowledge, our paper is the first work in the literature that demonstrates such sensitivity.Our investigation is motivated by the empirical observations on the MNIST dataset and the CIFAR10 dataset. In particular , while comparable SOTA clean accuracies (the difference is less than 3%) are achieved by MNIST and CIFAR10 BID10 , CIFAR10 suffers from much lower achievable robustness than MNIST in practice. 1 Results of this paper consist of two parts. First in theory , we start with analyzing the difference between the regular Bayes error and the robust error, and show that the regular Bayes error is invariant to invertible transformations of the data distribution, but the robust error is not. We further prove that if the input data is uniformly distributed, then the perfect decision boundary cannot be robust. However, we also manage to find a robust model for the binarized MNIST dataset (semantically almost identical to MNIST, later described in Section 3). The certification method by BID26 guarantees that this model achieves at most 3% robust error. Such a sharp contrast suggests the important role of the data distribution in adversarial robustness, and leads to our second contribution on the empirical side: we design a series of augmented MNIST and CIFAR10 datasets to demonstrate the sensitivity of adversarial robustness to the input data distribution.Our finding of such sensitivity raises the question of how to properly evaluate adversarial robustness. In particular, the sensitivity of adversarial robustness suggests that certain datasets may not be sufficiently representative when benchmarking different robust learning algorithms. It also raises serious concerns about the deployment of believed-to-be-robust training algorithm in a real product. In a standard development procedure , various models (for example different network architectures) would be prototyped and measured on the existing data. However, the sensitivity of adversarial robustness makes the truthfulness of the performance estimations questionable, as one would expect future data to be slightly shifted. We illustrate the practical implications in Section 4 with two practical examples: 1) the robust accuracy of PGD trained model is sensitive to gamma values of gamma-corrected CIFAR10 images. This indicates that image datasets collected under different light conditions may have different robustness properties; 2) both as a "harder" version of MNIST, the fashion-MNIST BID27 and edge-fashion-MNIST (an edge detection variant described in Section 4.2) exhibit completely different robustness characteristics. This demonstrates that different datasets may give completely different evaluations for the same algorithm.Finally, our finding opens up a new angle and provides novel insights to the adversarial vulnerability problem, complementing several recent works on the issue of data distributions' influences on robustness. BID24 hypothesize that there is an intrinsic tradeoff between clean accuracy and adversarial robustness. Our studies complement this result, showing that there are different levels of tradeoffs depending on the characteristics of input data distribution, under the same learning settings (training algorithm, model and training set size). BID20 show that different data distributions could have drastically different properties of adversarially robust generalization, theoretically on Bernoulli vs mixtures of Gaussians, and empirically on standard benchmark datasets. From the sensitivity perspective, we demonstrate that being from completely different distributions (e.g. binary vs Gaussian or MNIST vs CIFAR10) may not be the essential reason for having large robustness difference. Gradual semantics-preserving transformations of data distribution can also cause large changes to datasets' achievable robustness. We make initial attempts in Section 5 to further understand this sensitivity. We investigated perturbable volume and inter-class distance as the natural causes of the sensitivity; model capacity and sample complexity as the natural remedies. However, the complexity of the problem has so far defied our efforts to give a definitive answer. In this paper we provided theoretical analyses to show the significance of input data distribution in adversarial robustness, which further motivated our systematic experiments on MNIST and CI-FAR10 variants. We discovered that, counter-intuitively, robustness of adversarial trained models are sensitive to semantically-preserving transformations on data. We demonstrated the practical implications of our finding that the existence of such sensitivity questions the reliability in evaluating robust learning algorithms on particular datasets. Finally, we made initial attempts to understand this sensitivity. DISPLAYFORM0 Then we apply Markov's inequality, for all real number t > 0: DISPLAYFORM1 Finally, we observe that the longest (in terms of 2 norm) such ∞ attacks vector to HP 2 are parallel to the normal vector 1 to HP 2 . They have 2 distance √ d. The set these attacks cover is characterized by {x DISPLAYFORM2 Let t = 2 d, we have: DISPLAYFORM3 In the case of zero-one loss, RR DISPLAYFORM4 A.2 PROOF FOR THEOREM 2.1Proof. (First Inequality for Cube) The proof here follows that of BID16 , but we track of the tight constants so as to give tighter adversarial robustness calculations.Let Φ be one dimensional standard normal cumulative distribution function and let µ d denote d dimensional Gaussian measures. Consider the map T : DISPLAYFORM5 T pushes forward µ d defined on R d into a probability measure P on (0, 1) d : DISPLAYFORM6 for A ⊂ (0, 1) d . Next we have the following Gaussian isoperimetric/concentration inequality BID16 : DISPLAYFORM7 Now for A ⊂ (0, 1) d , we have: DISPLAYFORM8 where the first inequality follows from that T has Lipschitz constant DISPLAYFORM9 , and thus T −1 has Lipschitz constant √ 2π; and the second one follows from Gaussian isoperimetric inequality. DISPLAYFORM10 Additionally, the inequality Φ(x) ≥ 1 − e x 2 2 implies the last inequality in the theorem.
Robustness performance of PGD trained models are sensitive to semantics-preserving transformation of image datasets, which implies the trickiness of evaluation of robust learning algorithms in practice. Paper clarifies the difference between clean and robust accuracy and shows that changing the marginal distribution of the input data P(x) while preserving its semantic P(y|x) affects the robustness of the model. This paper investigates the origin of the lack of robustness of classifiers to perturbations of adversarial inputs under l-inf bounded perturbations.