Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space. While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs. We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data. We find that adversarial training increases the robustness of classifiers against shared perturbations. Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations. Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs. However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or  Gaussian blurring. It thus makes successful attacks on the classifier in the physical world less likely. Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs. While deep learning is relatively robust to random noise , it can be easily fooled by so-called adversarial perturbations BID26 . These perturbations are generated by adversarial attacks BID8 BID2 ) that generate perturbed versions of the input which are misclassified by a classifier and, at the same time, remain quasi-imperceptible for humans. There have been different approaches for explaining properties of adversarial examples and why they exist in the first place BID8 BID27 b) . Adversarial perturbations often transfer between different network architectures BID26 . Moreover, adversarial perturbations have been shown to be relatively robust against various kind of image transformations and can even be successful when being placed as artifacts in the physical world BID24 BID4 . While adversarial perturbations are data-dependent in that they were generated to fool a classifier on a specific input, there also exist universal perturbations which mislead a classifier on the majority of the inputs b; BID16 .Several methods have been proposed for increasing the robustness of deep networks against adversarial examples such as adversarial training BID8 BID13 , virtual adversarial training BID17 , ensemble adversarial training BID28 , defensive distillation BID23 BID22 , stability training BID30 , robust optimization BID14 , and Parseval networks BID3 ). An alternative approach for defending against adversarial examples is to detect and reject them as malicious BID15 . While some of these approaches actually improve robustness against adversarial examples, the classifier remains vulnerable against adversarial perturbations on a non-negligible fraction of the inputs.While adversarial training is arguably the most popular approach for increasing the robustness of deep networks, few works have investigated its effect on the resulting classifiers and adversarial perturbations. Recently, BID29 investigated the effect of adversarial training on the classifier's decision boundary and found that it displaces the boundary slightly but not sufficiently for preventing black-box attacks (attacks where the adversarial perturbation is generated based on a source model that is different from the target model). While BID29 focus on the transferability of perturbations in a black-box scenario, we investigate in this work properties of adversarial perturbations under adversarial training in a white-box scenario, where source and target model are identical.We first study a property of adversarial perturbations which we denote as sharedness. We define a perturbation as shared if it fools the classifier on many inputs (at least 2%). We empirically find that adversarial training is effective in removing shared perturbations but does not improve robustness against singular perturbations, which only fool the classifier on a very small subset of the data. Moreover, we study how adversarial training affects the existence of universal perturbations, the robustness of adversarial perturbations against image transformations, and the detectability of adversarial perturbations. In summary, we find that adversarial training is very effective in removing universal perturbations and that the remaining (non-universal, singular) adversarial perturbations are less robust against most transformations. Thus, adversarial training is more promising in preventing certain kind of attacks on system safety, such as those performed in the physical world, than might have been assumed. Moreover, we find that adversarial training leaves the remaining singular adversarial perturbations easily detected. Thus, these perturbations must exhibit some generalizable patterns that can be detected. Understanding and exploiting those patterns might ideally lead to identifying new ways for improving adversarial training. We have empirically investigated the effect of adversarial training on adversarial perturbations. We have found that adversarial training increases robustness against shared perturbations and even more against universal perturbations. A non-negligible part of the inputs, however, exhibit singular perturbations, which are effective only for the specific input. While adversarial training is not successful in removing singular perturbations, we found that it makes the remaining singular perturbations less robust against image transformations such as changes to brightness, contrast, or Gaussian blurring. Because of the classifier's reduced vulnerability against universal perturbations and the reduced robustness of the remaining singular perturbations, adversarial training appears very promising for preventing attacks on system safety such as physical-world attacks presented for face recognition BID24 or road sign classification BID4 . We strongly recommend that future work in those directions investigates the feasibility of attacks not only against undefended classifiers but also against classifiers that have undergone adversarial training or other defense mechanisms since our results indicate that adversarial training might affect the feasibility of such attacks severely.Interestingly, while singular perturbations are very specific for certain inputs, we found them nevertheless to be sufficiently regular for being detectable. While detection of adversarial perturbationsis not yet an effective defense mechanism BID1 , our results on detectability indicate that there seem to be stable patterns in adversarial examples but that adversarial training fails to make the classifier itself robust against those patterns. Understanding and using what makes up stable patterns in adversarial examples may be a promising direction for improving adversarial training and increasing robustness against safety-relevant attacks in the physical world. We leave a closer investigation of this to future work.
We empirically show that adversarial training is effective for removing universal perturbations, makes adversarial examples less robust to image transformations, and leaves them detectable for a detection approach. Analyses adversarial training and its effect on universal adversarial examples as well as standard (basic iteration) adversarial examples and how adversarial training affects detection.  The authors show that adversarial training is effective in protecting against "shared" adversarial perturbation, in particular against universal perturbation, but less effective to protect against singular perturbations.