Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks. However, recently it has been shown that it is also vulnerable to adversarial perturbations. In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm. We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer. We empirically proved this by experiments on various neural network architectures and datasets. Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm. In spite of their impressive performance on challenging tasks in computer vision such as image classification and semantic segmentation, deep neural networks (DNNs) are shown to be highly vulnerable to adversarial examples, i.e. carefully crafted samples which look similar to natural images but designed to mislead a trained neural network model (Goodfellow et al., 2014; Nguyen et al., 2015; Carlini & Wagner, 2017) . Designing defense mechanisms against these adversarial perturbations has been subjected to much research recently (Xie et al., 2019; Madry et al., 2017; Tramèr et al., 2017; Papernot et al., 2016) . Meanwhile, Batch Normalization (BatchNorm or BN) (Ioffe & Szegedy, 2015) has successfully proliferated throughout all areas of deep learning as it enables stable training, higher learning rates, faster convergence, and higher generalization accuracy. Initially, the effectiveness of the BatchNorm has been attributed to its ability to eliminate the internal covariate shift (ICS), the tendency of the distribution of activations to drift during training. However, later on, alternative reasons including avoiding exploding activations, smooth loss landscape, reducing the sensitivity to initialization, etc. have also been proposed as the basis of BatchNorm's success (Santurkar et al., 2018; Bjorck et al., 2018; Luo et al., 2018) . While there exist a plethora of reasons for the adversarial vulnerability of deep neural networks (Jacobsen et al., 2018; Simon-Gabriel et al., 2018) , a recent study by Galloway et al. (2019) showed that BatchNorm is one of the reasons for this vulnerability. Specifically, they empirically showed that removing the BatchNorm layer enhances robustness against adversarial perturbations. However, removal of BatchNorm also means a sacrifice of benefits such as the use of higher learning rates, faster convergence, and significant improvement in the clean test set accuracy among many others. In this paper, we propose a new perspective regarding the adversarial vulnerability of the BatchNorm layer. Specifically, we probe why BatchNorm layer causes the adversarial vulnerability. We hypothesize that the use of different normalization statistics during training and inference phase (mini-batch statistics for training and moving average of these statistics also called tracking, at inference time) is the cause of this adversarial vulnerability of the BatchNorm layer. Our experiments show that by removing this part of the BatchNorm, the robustness of the network increases by 20%. Similarly, robustness can further be enhanced by up to 30% after adversarial training. However, by removing the tracking part, the test accuracy on the clean images drops significantly ( though better than without normalization). To circumvent this issue, we propose Robust Normalization (RobustNorm or RN). Our experiments demonstrate that RobustNorm not only significantly improve the test performance of adversarially-trained DNNs but is also able to achieve the comparable test accuracy to that of BatchNorm on unperturbed datasets. We perform numerical experiments over standard datasets and DNN architectures. In almost all of our experiments, we obtain a better adversarial robustness performance on perturbed examples for training with natural as well as adversarial training. Addition of maliciously crafted noise in normal inputs, also called adversarial examples has proven to be deceptive for neural networks. While there are many reasons for this phenomena, recent work has shown BatchNorm to be a cause of this vulnerability as well. In this paper, we have investigated the reasons behind this issue and found that tracking part of BatchNorm causes this adversarial vulnerability. Then, we showed that by eliminating it, we can increase the robustness of a neural network. Afterward, based on the intuitions from the work done for the understanding of BatchNorm, we proposed RobustNorm which has much higher robustness than BatchNorm for both natural as well as adversarial training scenarios. In the end, we have shown how tracking can be a necessary evil and argued that it requires further careful investigation. In this section, we provide more detailed results for our experiments for both CIFAR10 and CIFAR100 datasets. In this section, we put results of increasing adversarial noise on CIFAR100 dataset. A.3 ANOTHER ASPECT OF ROBUSTNORM As we have discussed, ICS hypothesis has been negated by a few recent studies. One of these studies (Santurkar et al., 2018) suggested that based on the results, " it might be valuable to perform a principled exploration of the design space of normalization schemes as it can lead to better performance." In this way, we can see RobustNorm with tracking as a new normalization scheme which is based on alternative explanations yet having performance equal to BatchNorm which, in a way, weakens ICS hypothesis. See Figure 7 for a comparison of accuracies over different models for CIFAR10 and CIFAR100 datasets.
Investigation of how BatchNorm causes adversarial vulnerability and how to avoid it.  This paper addresses vulnerability to adversarial perturbations in BatchNorm, and proposes an alternative called RobustNorm, using min-max rescaling instead of normalization. This paper investigates the reason behind the vulnerability of BatchNorm and proposes Robust Normalization, a normalization method that achieves significantly better results under a variety of attack methods.