Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning. While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms. In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks. After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods. Based on the evaluation results, we draw several important findings and provide insights for future research. Recent progress in deep learning (DL) has led to substantial improvements in a wide range of domains, such as image understanding (Krizhevsky et al., 2012; He et al., 2016) , speech recognition (Graves et al., 2013) , and natural language processing (Devlin et al., 2019) . However, the existing DL models are highly vulnerable to adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015) , which are maliciously generated by an adversary to make a model produce erroneous predictions. As DL models have been integrated into various security-sensitive applications (e.g., autonomous driving, healthcare, and finance), the study of the adversarial robustness issue has attracted increasing attention with an enormous number of adversarial attack and defense methods proposed. Therefore, it is crucial to conduct correct and rigorous evaluations of these methods for understanding their pros and cons, comparing their performance, and providing insights for building new methods (Carlini et al., 2019) . The research on adversarial robustness is faced with an "arms race" between attacks and defenses: a defense method proposed to prevent existing attacks was soon evaded by new attacks, and vice versa (Carlini & Wagner, 2017a; b; He et al., 2018; Athalye et al., 2018a; Uesato et al., 2018; Zhang et al., 2019b) . For instance, defensive distillation (Papernot et al., 2016c) was proposed to improve the robustness, but was later shown to be ineffective against a strong attack (Carlini & Wagner, 2017b) . Many methods were introduced to build robust models by causing obfuscated gradients, which can be defeated by the adaptive ones (Athalye et al., 2018a; Uesato et al., 2018) . As a result, it is particularly challenging to understand their effects, identify the real progress, and advance the field. Moreover, the current attacks and defenses are often evaluated incompletely. First, most defenses are only tested against a small set of attacks under limited threat models, and many attacks are evaluated on a few models or defenses. Second, the robustness evaluation metrics are too simple to show the performance of these methods. The accuracy of a defense against an attack for a given perturbation budget (Kurakin et al., 2018) and the minimum distance of the adversarial perturbation (Brendel et al., 2018b) are used as the primary evaluation metrics, which are often insufficient to characterize the behavior of the attacks and defenses totally. Consequently, the incomplete evaluation cannot provide a comprehensive understanding of the strengths and limitations of the attack and defense methods. In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness, which can provide a comprehensive understanding of the effects of existing methods under different scenarios, with a hope to facilitate the future research. In particular, we focus on the robustness of image classifiers under the p norm threat models, since the adversarial robustness issue has been extensively studied on image classification tasks with the p additive noises. We incorporate a lot of typical and state-of-the-art attack and defense methods for robustness evaluation, including 15 attack methods and 16 defense models-8 on CIFAR-10 (Krizhevsky & Hinton, 2009 ) and 8 on ImageNet (Russakovsky et al., 2015) . To fully demonstrate the performance of these methods, we adopt two complementary robustness curves as the major evaluation metrics to present the results. Then, we carry out large-scale experiments on the cross evaluation of the attack and defense methods under complete threat models 1 , including 1) untargeted and targeted attacks; 2) ∞ and 2 attacks; 3) white-box, transfer-based, score-based, and decision-based attacks. By analyzing the quantitative results, we have some important findings. First, the relative robustness between defenses against an attack could be different under varying perturbation budgets or attack iterations. So it is hard to conclude that a defense is more robust than another against an attack by using a specific configuration. However, this is common in previous works. Second, although various defense techniques have been proposed, the most robust defenses are still the adversarially trained models. The robustness of these defenses can also generalize to other threat models, under which they are not trained to be robust. Third, defenses based on randomization are generally more robust to black-box attacks based on the query feedback. More detailed discussions can be found in Sec. 5.3. All evaluation experiments are conducted on a new adversarial robustness platform 2 developed by us, since the existing platforms (e.g., CleverHans (Papernot et al., 2016a) , Foolbox (Rauber et al., 2017) , etc) cannot fully support our comprehensive evaluations (details in Appendix A). We hope that our platform could continuously incorporate and evaluate more methods, and be helpful for future works. Based on the above results and more results in Appendix C, we highlight some key findings. First, the relative robustness between defenses against the same attack could be different under varying attack parameters, such as the perturbation budget or the number of attack iterations. Not only the results of PGD-AT and TRADES in Fig. 1 can prove it, but also the results in many different scenarios show the similar phenomenon. Given this observation, the comparison between defenses at a specific attack configuration cannot fully demonstrate the superiority of a method upon another. We therefore strongly advise the researchers to adopt the robustness curves as the major evaluation metrics to present the robustness results. Second, among the defenses studied in this paper, we find that the most robust models are obtained by PGD-based adversarial training. Their robustness not only is good for the threat model under which they are trained (i.e., the ∞ threat model), but can also generalize to other threat models (e.g., the 2 threat model). However, adversarial training usually leads to a reduction of natural accuracy and high training cost. A research direction is to develop new methods that maintain the natural accuracy or reduce the training cost. And we have seen several works (Shafahi et al., 2019) in this direction. Third, we observe that the defenses based on randomization are quite resistant to score-based and decision-based attacks, which rely on the query feedback of the black-box models. We argue that the robustness of the randomization-based defenses against these attacks is due to the random predictions given by the models, making the estimated gradients or search directions unreliable for attacks. A potential research direction is to develop more powerful score-based and decision-based attacks that can efficiently evade the randomization-based defenses. Fourth, the defenses based on input transformations (e.g., JPEG, Bit-Red) can sightly improve the robustness over the undefended models, and sometimes get much higher accuracy against black-box attacks. Since these methods are quite simple, they may be combined with other types of defenses to build more powerful defenses. Fifth, we find that different transfer-based attack methods exhibit similar performance on CIFAR-10, while the recent methods (e.g., MIM, DIM) can improve the transferability of adversarial examples over BIM on ImageNet. One potential reason is that the input dimension of the models on ImageNet is much higher than that on CIFAR-10, and thus the adversarial examples generated by BIM can easily "overfit" to the substitute model (Dong et al., 2018) , resulting in poor transferability. And the recent methods proposed to solve this issue can generate more transferable adversarial examples. Note that these findings are based on our current benchmark, which may be strengthened or falsified in the future if new results are given. In this paper, we established a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness of image classifiers. We performed large-scale experiments with two robustness curves as the fair-minded evaluation criteria to facilitate a better understanding of the representative and state-of-the-art adversarial attack and defense methods. We drew some key findings based on the evaluation results, which may be helpful for future research. , 2018) , etc. However, we observe that these platforms do not totally support our comprehensive evaluations in this paper. First, some attacks evaluated in this paper are not included in these platforms. There are less than 10 out of the 15 attacks adopted in this paper that are already implemented in each platform. And most of the available methods are white-box methods. Second, although these platforms incorporate a few defenses, they do not use the pre-trained models. But we use the original source codes and pre-trained models to perform unbiased evaluations. Third, the evaluation metrics defined by the two robustness curves in this paper are not provided in the existing platforms. Therefore, we develop a new adversarial robustness platform to satisfy our requirements.
We provide a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness of deep learning models. This paper presents an evaluation of different kinds of classification models under various adversarial attack methods. A large-scale empirical study comparing different adversarial attack and defense techniques, and use of accuracy vs. perturbation budget and accuracy vs. attack strength curves to evaluate attacks and defenses.