Despite the remarkable performance of deep neural networks (DNNs) on various tasks, they are susceptible to adversarial perturbations which makes it difficult to deploy them in real-world safety-critical applications. In this paper, we aim to obtain robust networks by sparsifying DNN's latent features sensitive to adversarial perturbation. Specifically, we define vulnerability at the latent feature space and then propose a Bayesian framework to prioritize/prune features based on their contribution to both the original and adversarial loss. We also suggest regularizing the features' vulnerability during training to improve robustness further. While such network sparsification has been primarily studied in the literature for computational efficiency and regularization effect of DNNs, we confirm that it is also useful to design a defense mechanism through quantitative evaluation and qualitative analysis. We validate our method, \emph{Adversarial Neural Pruning (ANP)} on multiple benchmark datasets, which results in an improvement in test accuracy and leads to state-of-the-art robustness. ANP also tackles the practical problem of obtaining sparse and robust networks at the same time, which could be crucial to ensure adversarial robustness on lightweight networks deployed to computation and memory-limited devices. In the last many years, deep neural networks (DNNs) have achieved impressive results on various artificial intelligence tasks, e.g., image classification , face and object recognition (He et al., 2015; Deng et al., 2018) , semantic segmetnation (Badrinarayanan et al., 2015; He et al., 2017) and playing games (Silver et al., 2016; . The groundbreaking success of DNNs has motivated their use in a broader range of domains, including more safety-critical environments such as medical imaging (Esteva et al., 2017; Rajpurkar et al., 2017) and autonomous driving (Bojarski et al., 2016; Li et al., 2017) . However, DNNs are shown to be extremely brittle to carefully crafted small adversarial perturbations added to the input (Szegedy et al., 2013; Goodfellow et al., 2014) . These perturbations are imperceptible to human eyes but have been intentionally optimized to cause miss-classification. While the field has primarily focused on the development of new attacks and defenses, a 'cat-andmouse' game between attacker and defender has arisen. There has been a long list of proposed defenses to mitigate the effect of adversarial examples defenses (Papernot et al., 2015b; Xu et al., 2017b; Buckman et al., 2018; Dhillon et al., 2018; Xie et al., 2018; Tramèr et al., 2018; Liu et al., 2019) , followed by round of successful attacks (Carlini & Wagner, 2016; Uesato et al., 2018; designed in light of the new defense. Since it shows that any defense mechanism that once looks successful could be circumvented with the invention of new attacks, we try to tackle the problem by identifying a more fundamental cause of the adversarial vulnerability of deep neural networks. What makes deep neural networks vulnerable to adversarial attacks? We conjecture that the adversarial vulnerability of deep neural networks is mostly due to the distortion in the latent feature space. If any perturbation at the input level is successfully suppressed in the latent feature space at any layer of the neural network, such that clean and adversarial samples cannot be distinguished in the latent feature space, then it will not lead to misclassification. However, not all latent features will contribute equally to the distortion in the latent feature space; some latent features may have larger distortion, by amplifying the perturbations at the input level while others will remain relatively static. We consider a novel problem of distortion in latent features of a network in the presence of adversarial perturbation, where the model observes different degrees of distortion for different features (brighter red indicates higher level of distortion). To solve this problem, our proposed method learns a bayesian pruning mask to suppress the higher distorted features in order to maximize it's robustness on adversarial perturbations. In this paper, based on the motivation that adversarial vulnerability comes from distortion in the latent feature space, we first formally define the vulnerability of the latent features and propose to minimize the feature-level vulnerability to achieve adversarial robustness with DNNs. One way to suppress the vulnerability in the feature space is by adding a regularization that minimizes it. However, a more effective and irreversible means is to set the vulnerability to zero, by completely dropping the latent features with high vulnerability. This is shown in Figure 2 (a), where sparse networks are shown to have a much smaller degree of vulnerability (average perturbation of the latent feature across all layers). However, naive sparsification approaches will prune both the robust and vulnerable features, which will limit its effectiveness as a defense mechanism. Moreover, when the sparsity is pushed further, it will prune out robust features which will hurt the model robustness. To overcome this limitation, we propose the so-called adversarial neural pruning (ANP) method that adversarially learns the pruning mask, such that we can prune out vulnerable features while preserving robust ones. Our method requires little or no modification of the existing network architectures, can be applied to any pre-trained networks and it effectively suppresses the distortion in the latent feature space (See Figure 1) and thus obtains a model that is more robust to adversarial perturbations. We validate our model on multiple heterogeneous datasets including MNIST, CIFAR-10, and CIFAR-100 for its adversarial robustness. Our experimental results show that ANP achieves significantly improved adversarial robustness, with significantly less memory and computational requirements. In summary, the contribution of this paper is as follows: • We consider the vulnerability of latent features as the main cause of DNN's susceptibility to adversarial attacks, and formally describe the concepts of vulnerable and robust latent features, based on the expectation of the distortion with respect to input perturbations. • We show that while sparsity improves the robustness of DNNs by zeroing out distortion at the pruned features, it is still orthogonal to robustness and even degenerates robustness at a high degree, via experimental results and visualization of the loss landscape. • Motivated by the above findings, we propose the ANP method that prunes out vulnerable features while preserving robust ones, by adversarially learning the pruning mask in a Bayesian framework. During training, we also regularize the vulnerability of the latent features to improve robustness further. • The proposed ANP framework achieves state-of-the-art robustness on CIFAR-10 and CIFAR-100 datasets, along with a large reduction in memory and computation. While our major focus is on achieving robustness with DNNs, we found that ANP also achieves higher accuracy for clean/non-adversarial inputs, compared to the baseline scheme of adversarial training (see the results of CIFAR datasets in Table 1 ). This is due to the fact that sparsification helps to regularize models and is also an important benefit of ANP as it has been well known that adversarial training schemes tend to hurt the accuracy of the DNNs on non-adversarial samples Tsipras et al., 2019; Zhang et al., 2019) . Moreover, our method enables to obtain a robust and lightweight network, which is useful when working with resource-limited devices. We propose a novel adversarial neural pruning and vulnerability suppression loss, as a defense mechanism to achieve adversarial robustness as well as a means of achieving a memory and computationefficient deep neural networks. We observe that the latent features of deep networks have a varying degree of distortion/robustness to the adversarial perturbations to the input and formally defined the vulnerability and robustness of a latent feature. This observation suggests that we can increase the robustness of the model by pruning out vulnerable latent features and by minimizing the vulnerability of the latent features, we show that sparsification thus leads to certain degree of robustness over the base network for this obvious reason. We further propose a Bayesian formulation that trains the pruning mask in an adversarial training, such that the obtained neurons are beneficial both for the accuracy of the clean and adversarial inputs. Experimental results on a range of architectures with multiple datasets demonstrate that our adversarial pruning is effective in improving the model robustness. Further qualitative analysis shows that our method obtains more interpretable latent features compared to standard counterparts, suppresses feature-level distortions in general while zeroing out perturbations at many of them, and obtains smooth loss surface.
We propose a novel method for suppressing the vulnerability of latent feature space to achieve robust and compact networks. This paper proposes "adversarial neural pruning" method of training a pruning mask and a new vulnerability suppression loss to improve accuracy and adversarial robustness.