Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD).   In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice.   Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost.   Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy at epsilon=8/255 in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at epsilon=2/255 in 12 hours, in comparison to past work based on ``free'' adversarial training which took 10 and 50 hours to reach the same respective thresholds. Although deep network architectures continue to be successful in a wide range of applications, the problem of learning robust deep networks remains an active area of research. In particular, safety and security focused applications are concerned about robustness to adversarial examples, data points which have been adversarially perturbed to fool a model (Szegedy et al., 2013) . The goal here is to learn a model which is not only accurate on the data, but also accurate on adversarially perturbed versions of the data. To this end, a number of defenses have been proposed to mitigate the problem and improve the robustness of deep networks, with some of the most reliable being certified defenses and adversarial training. However, both of these approaches come at a non-trivial, additional computational cost, often increasing training time by an order of magnitude over standard training. This has slowed progress in researching robustness in deep networks, due to the computational difficulty in scaling to much larger networks and the inability to rapidly train models when experimenting with new ideas. In response to this difficulty, there has been a recent surge in work that tries to to reduce the complexity of generating an adversarial example, which forms the bulk of the additional computation in adversarial training Shafahi et al., 2019) . While these works present reasonable improvements to the runtime of adversarial training, they are still significantly slower than standard training, which has been greatly accelerated due to competitions for optimizing both the speed and cost of training (Coleman et al., 2017) . In this work, we argue that adversarial training, in fact, is not as hard as has been suggested by this past line of work. In particular, we revisit one of the the first proposed methods for adversarial training, using the Fast Gradient Sign Method (FGSM) to add adversarial examples to the training process (Goodfellow et al., 2014) . Although this approach has long been dismissed as ineffective, we show that by simply introducing random initialization points, FGSM-based training is as effective as projected gradient descent based training while being an order of magnitude more efficient. Moreover, FGSM adversarial training (and to a lesser extent, other adversarial training methods) can be drastically accelerated using standard techniques for efficient training of deep networks, including e.g. cyclic learning rates (Smith & Topin, 2018) , mixed-precision training (Micikevicius et al., 2017) , and other similar techniques. The method has extremely few free parameters to tune, and can be easily adapted to most training procedures. We further identify a failure mode that we call "catastrophic overfitting", which may have caused previous attempts at FGSM adversarial training to fail against PGD-based attacks. The end result is that, with these approaches, we are able to train (empirically) robust classifiers far faster than in previous work. Specifically, we train an ∞ robust CIFAR10 model to 45% accuracy at = 8/255 (the same level attained in previous work) in 6 minutes; previous papers reported times of 80 hours for PGD-based training (Madry et al., 2017) and 10 hours for the more recent "free" adversarial training method (Shafahi et al., 2019) . Similarly, we train an ∞ robust ImageNet classifier to 43% top-1 accuracy at = 2/255 (again matching previous results) in 12 hours of training (compared to 50 hours in the best reported previous work that we are aware of (Shafahi et al., 2019) ). Both of these times roughly match the comparable time for quickly training a standard non-robust model to reasonable accuracy. We extensively evaluate these results against strong PGDbased attacks, and show that they obtain the same empirical performance as the slower, PGD-based training. Thus, we argue that despite the conventional wisdom, adversarially robust training is not actually more challenging than standard training of deep networks, and can be accomplished with the notoriously weak FGSM attack. Our findings show that FGSM adversarial training, when used with random initialization, can in fact be just as effective as the more costly PGD adversarial training. While a single iteration of FGSM adversarial training is double the cost of free adversarial training, it converges significantly faster, especially with a cyclic learning rate schedule. As a result, we are able to learn adversarially robust classifiers for CIFAR10 in minutes and for ImageNet in hours, even faster than free adversarial training but with comparable levels of robustness. We believe that leveraging these significant reductions in time to train robust models will allow future work to iterate even faster, and accelerate research in learning models which are resistant to adversarial attacks. By demonstrating that extremely weak adversarial training is capable of learning robust models, this work also exposes a new potential direction in more rigorously explaining when approximate solutions to the inner optimization problem are sufficient for robust optimization, and when they fail. Tramèr et al. (2017) and the various changes for the version of FGSM adversarial training done in this paper, over 10 random seeds.
FGSM-based adversarial training, with randomization, works just as well as PGD-based adversarial training: we can use this to train a robust classifier in 6 minutes on CIFAR10, and 12 hours on ImageNet, on a single machine. This paper revisits Random+FGSM method to train robust models against strong PGD evasion attacks faster than previous methods. The main claim of this paper is that a simple strategy of randomization plus fast gradient sign method (FGSM) adversarial training yields robust neural networks.