It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples. In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear. Here we show that distributions of logit differences have a universal functional form. This functional form is independent of architecture, dataset, and training protocol; nor does it change during training. This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation. We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD). Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness. Finally, we study the effect of network architectures on adversarial sensitivity. To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10. Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.
 An intriguing aspect of deep learning models in computer vision is that while they can classify images with high accuracy, they fail catastrophically when those same images are perturbed slightly in an adversarial fashion BID17 BID1 . The prevalence of adversarial examples presents challenges to our understanding of how deep networks generalize and pose security risks in real world applications BID11 BID5 . Several techniques have been proposed to defend against adversarial examples. Adversarial training BID1 augments the training data with adversarial examples. It has been shown that using stronger adversarial attacks in adversarial training can increase the robustness to stronger attacks, but at the cost of a decrease in clean accuracy (i.e. accuracy on samples that have not been adversarially perturbed) BID8 . Defensive distillation BID12 , feature squeezing BID22 , and Parseval training BID0 have also been shown to make models more robust against adversarial attacks.The goal of this work is to study the common properties of adversarial examples. We calculate the adversarial error, defined as the difference between clean accuracy and adversarial accuracy at a given size of adversarial perturbation ( ). Surprisingly, adversarial error has a similar dependence on small values of for all network models and datasets we studied, including linear, fully-connected, simple convolutional networks, Inception v3 BID19 , Inception-ResNet v2, Inception v4 BID20 , ResNet v1, ResNet v2 BID2 , NasNet-A BID24 BID25 , adversarially trained Inception v3 BID6 and Inception-ResNet v2 BID21 , and networks trained on randomly shuffled labels of MNIST. Adversarial error due to the Fast Gradient Sign Method (FGSM), its L2-norm variant, and Projected Gradient Descent (PGD) attack grows as a power-law like A B with B between 0.9 and 1.3. By contrast, we find that adversarial error caused by one-step least likely class method (step l.l.) also scales as a power-law where B is between 1.8 and 2.5 for small . This observed universality points to a mysterious commonality between these models and datasets, despite the different number of channels, pixels, and classes present. Adversarial error caused by FGSM on the training set of randomly shuffled labels of MNIST (LeCun & Cortes) also has the power-law form where B = 1.2, which implies that the universality is not a result of the specific content of these datasets nor the ability of the model to generalize.To discover the mechanism behind this universality we show how, at small , the success of an adversarial attack depends on the input-logit Jacobian of the model and on the logits of the network. We demonstrate that the susceptibility of a model to FGSM and PGD attacks is in large part dictated by the cumulative distribution of the difference between the most likely logit and the second most likely logit. We observe that this cumulative distribution has a universal form among all datasets and models studied, including randomly produced data. Together, we believe these results provide a compelling story regarding the susceptibility of machine learning models to adversarial examples at small .We show that training with single-step adversarial examples offers protection against large attacks (between 0.2 and 32), but does not help appreciably at defending against small attacks (below 0.2). At = 0.2, all ImageNet models we studied incur 10 to 25% adversarial error, and surprisingly, vanilla NASNet-A (best clean accuracy in our study) has a lower adversarial error than adversarially trained Inception-ResNet v2 or Inception v3 BID6 (Fig. 1(a ) ). In light of these results, we explore a different avenue to adversarial robustness through architecture selection. We perform neural architecture search (NAS) using reinforcement learning BID24 BID25 . These techniques allow us to find several architectures that are especially robust to adversarial perturbations. In addition , by analyzing the adversarial robustness of the tens-of-thousands of architectures constructed by NAS, we gain insights into the relationship between size of a model, its clean accuracy, and its adversarial robustness. In summary , the key contributions of our work are:• We study the functional form of adversarial error and logit differences across several models and datasets, which turn out to be universal. We analytically derive the commonality in the power-law tails of the logit differences, and show how it leads to the commonality in the form of adversarial error.• We observe that although the qualitative form of logit differences and adversarial error is universal, it can be quantitatively improved with entropy regularization and better network architectures.• We study the dependence of adversarial robustness on the network architecture via NAS.We show that while adversarial accuracy is strongly correlated with clean accuracy, it is only weakly correlated with model size. Our work leads to architectures that are more robust to white-box and black-box attacks on CIFAR10 BID4 ) than previous studies. In this paper we studied common properties of adversarial examples across different models and datasets. We theoretically derived a universality in logit differences and adversarial error of machine learning models. We showed that architecture plays an important role in adversarial robustness, which correlates strongly with clean accuracy. such that t β = 1 if β = γ for some γ and t β = 0 otherwise. We assume our network gets the answer correct so that h γ > h β for all β = γ. Then we apply the adversarial perturbation, DISPLAYFORM0 Note that we can write DISPLAYFORM1 Where we associate J αβ = ∂h β /∂x α with the input-to-logit Jacobian linking the inputs to the logits and δ = ∂L/∂h β the error of the outputs of the network. We can compute the change to the logits of the network due to this perturbation. We find, DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 where we have plugged in for eq. (11). Expressing the above equation in terms of the Jacobian, it follows that we can write the effect of the adversarial perturbation on the logits by, DISPLAYFORM5 as postulated. To make progress we will again make a mean field approximation and assume that each of the logits are i.i.d. with arbitrary distribution P (h). We denote the cumulative distribution F (h). While it is not obvious that the factorial approximation is valid here, we will see that the resulting distribution of P (∆ 1j ) shares many qualitative similarities with the distribution observed in real networks.We first change variables from the logits to a sorted version of the logits, r i . The ranked logits are defined such that r 1 = max({h i }), r 2 = max({h i }\{r 1 }), · · · . Our first result is to compute the resulting joint distribution between r 1 and r j , P j (r 1 , r j ) = A(N, j)F N −j (r j ) [F (r 1 ) − F (r j )] j−2 P (r j )P (r 1 )where A(N, j) = N (N − 1) N −2 j−2 is a combinatorial factor. Eq. (18) has a simple interpretation. F N −j (r j ) is the probability that there are N − j variables less than r j ; [F (r 1 ) − F (r j )] j−2 is the probability that j − 2 variables are between r j and r 1 ; P (r j )P (r 1 ) is the probability that there is one variable equal to each of r 1 and r j . The combinatorial factor can be understood since there are N ways of selecting r 1 , N − 1 ways of selecting r j , and N −2 j−2 ways of choosing j − 2 variables out of the remaining N − 2 to be between r j and r 1 .In terms of eq. FORMULA0 we can compute the distribution over ∆ 1j to be given by, P (∆ 1j ) = drP j (r + ∆ 1j , r)= A(N, j) drF N −j (r) [F (r + ∆ 1j ) − F (r)] j−2 P (r)P (r + ∆ 1j ).We can analyze this equation for small ∆ 1j . Expanding to lowest order in ∆ 1j , P (∆ 1j ) ≈ A(N, j) drF N −j (r) [F (r) + ∆ 1j P (r) − F (r)] j−2 P (r) P (r) + ∆ 1j dP (r) dr ( Since the term in the integral does not depend on ∆ 1j the result follows with, DISPLAYFORM6 6.2.3 ARCHITECTURES FIG3 : Left: Best architecture from Experiment 1. Right: Architecture of NAS Baseline. We note that the architecture from Experiment 1 is "longer" and "narrower" than previous architectures found by NAS for higher clean accuracy BID24 BID25 .
Adversarial error has similar power-law form for all datasets and models studied, and architecture matters.