ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing
Recent studies have shown that higher accuracy on ImageNet usually leads to
better robustness against different corruptions. Therefore, in this paper,
instead of following the traditional research paradigm that investigates new
out-of-distribution corruptions or perturbations deep models may encounter, we
conduct model debugging in in-distribution data to explore which object
attributes a model may be sensitive to. To achieve this goal, we create a
toolkit for object editing with controls of backgrounds, sizes, positions, and
directions, and create a rigorous benchmark named ImageNet-E(diting) for
evaluating the image classifier robustness in terms of object attributes. With
our ImageNet-E, we evaluate the performance of current deep learning models,
including both convolutional neural networks and vision transformers. We find
that most models are quite sensitive to attribute changes. A small change in
the background can lead to an average of 9.23\% drop on top-1 accuracy. We also
evaluate some robust models including both adversarially trained models and
other robust trained models and find that some models show worse robustness
against attribute changes than vanilla models. Based on these findings, we
discover ways to enhance attribute robustness with preprocessing, architecture
designs, and training strategies. We hope this work can provide some insights
to the community and open up a new avenue for research in robust computer
vision. The code and dataset are available at
https://github.com/alibaba/easyrobust.
Xiaodan Li, Yuefeng Chen, Yao Zhu, Shuhui Wang, Rong Zhang, Hui Xue
# ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing

###### Abstract

Recent studies have shown that higher accuracy on ImageNet usually leads to better robustness against different corruptions. Therefore, in this paper, instead of following the traditional research paradigm that investigates new out-of-distribution corruptions or perturbations deep models may encounter, we conduct model debugging in distribution data to explore which object attributes a model may be sensitive to. To achieve this goal, we create a toolkit for object editing with controls of backgrounds, sizes, positions, and directions, and create a rigorous benchmark named ImageNet-E(diting) for evaluating the image classifier robustness in terms of object attributes. With our ImageNet-E, we evaluate the performance of current deep learning models, including both convolutional neural networks and vision transformers. We find that most models are quite sensitive to attribute changes. A small change in the background can lead to an average of 9.23% drop on top-1 accuracy. We also evaluate some robust models including both adversarially trained models and other robust trained models and find that some models show worse robustness against attribute changes than vanilla models. Based on these findings, we discover ways to enhance attribute robustness with preprocessing, architecture designs, and training strategies. We hope this work can provide some insights to the community and open up a new avenue for research in robust computer vision. The code and dataset are available at [https://github.com/alibaba/easyrobust](https://github.com/alibaba/easyrobust).

+
Footnote †: This research is supported in part by the National Key Research and Development Program of China under Grant No.2020AAA0140000.

## 1 Introduction

Deep learning has triggered the rise of artificial intelligence and has become the workhorse of machine intelligence. Deep models have been widely applied in various fields such as autonomous driving [27], medical science [32], and finance [37]. With the spread of these techniques, the robustness and safety issues begin to be essential, especially after the finding that deep models can be easily fooled by negligible noises [15]. As a result, more researchers contribute to building datasets for benchmark
ing model robustness to spot vulnerabilities in advance.

Most of the existing work builds datasets for evaluating the model robustness and generalization ability on out-of-distribution data [21, 6, 29] using adversarial examples and common corruptions. For example, the ImageNet-C(orruption) dataset conducts visual corruptions such as Gaussian noise to input images to simulate the possible processors in real scenarios [21]. ImageNet-R(enditions) contains various renditions (_e.g._, paintings, embroidery) of ImageNet object classes [20]. As both studies have found that higher accuracy on ImageNet usually leads to better robustness against different domains [21, 50]. However, most previous studies try to achieve this in a top-down way, such as architecture design, exploring a better training strategy, _etc_. We advocate that it is also essential to manage it in a bottom-up way, that is, conducting model debugging with the in-distribution dataset to provide clues for model repairing and accuracy improvement. For example, it is interesting to explore whether a bird with a water background can be recognized correctly even if most birds appear with trees or grasses in the training data. Though this topic has been investigated in studies such as causal and effect analysis [8], the experiments and analysis are undertaken on domain generalization datasets. How a deep model generalizes to different backgrounds is still unknown due to the vacancy of a qualified benchmark. Therefore, in this paper, we provide a detached object editing tool to conduct the model debugging from the perspective of object attribute and construct a dataset named ImageNet-E(diting).

The ImageNet-E dataset is a compact but challenging test set for object recognition that contains controllable object attributes including backgrounds, sizes, positions and directions, as shown in Fig. 1. In contrast to ObjectNet [5] whose images are collected by their workers via posing objects according to specific instructions and differ from the target data distribution. This makes it hard to tell whether the degradation comes from the changes of attribute or distribution. Our ImageNet-E is automatically generated with our object attribute editing tool based on the original ImageNet. Specifically, to change the object background, we provide an object background editing method that can make the background simpler or more complex based on diffusion models [24, 46]. In this way, one can easily evaluate how much the background complexity can influence the model performance. To control the object size, position, and direction to simulate pictures taken from different distances and angles, an object editing method is also provided. With the editing toolkit, we apply it to the large-scale ImageNet dataset [41] to construct our ImageNet-E(diting) dataset. It can serve as a general dataset for benchmarking robustness evaluation on different object attributes.

With the ImageNet-E dataset, we evaluate the performance of current deep learning models, including both convolutional neural networks (CNNs), vision transformers as well as the large-scale pretrained CLIP [39]. We find that deep models are quite sensitive to object attributes. For example, when editing the background towards high complexity (see Fig. 1, the \(3\)rd row in the background part), the drop in top-1 accuracy reaches 9.23% on average. We also find that though some robust models share similar top-1 accuracy on ImageNet, the robustness against different attributes may differ a lot. Meanwhile, some models, being robust under certain settings, even show worse results than the vanilla ones on our dataset. This suggests that improving robustness is still a challenging problem and the object attributes should be taken into account. Afterward, we discover ways to enhance robustness against object attribute changes. The main contributions are summarized as follows:

* We provide an object editing toolkit that can change the object attributes for manipulated image generation.
* We provide a new dataset called ImageNet-E that can be used for benchmarking robustness to different object attributes. It opens up new avenues for research in robust computer vision against object attributes.
* We conduct extensive experiments on ImageNet-E and find that models that have good robustness on adversarial examples and common corruptions may show poor performance on our dataset.

## 2 Related Work

The literature related to attribute robustness benchmarks can be broadly grouped into the following themes: robustness benchmarks and attribute editing datasets. Existing robustness benchmarks such as ImageNet-C(orruption) [21], ImageNet-R(endition) [20], ImageNet-Stylized [13] and ImageNet-3DCC [29] mainly focus on the exploration of the corrupted or out-of-distribution data that models may encounter in reality. For instance, the ImageNet-R dataset contains various renditions (_e.g._, paintings, embroidery) of ImageNet object classes. ImageNet-C analyzes image models in terms of various simulated image corruptions (_e.g._, noise, blur, weather, JPEG compression, _etc._). Attribute editing dataset creation is a new topic and few studies have explored it before. Among them, ObjectNet [5] and ImageNet-9 (_a.k.a._ background challenge) [50] can be the representative. Specifically, ObjectNet collects a large real-world test set for object recognition with controls where object backgrounds, rotations, and imaging viewpoints are random. The images in ObjectNet are collected by their workers who image objects in their homes. It consists of \(313\) classes which are mainly household objects. ImageNet-9 mainly creates a suit of datasets that help disentangle the impact of foreground and background signals on classification. To achieve this goal, it uses coarse-grained classes
with corresponding rectangular bounding boxes to remove the foreground and then paste the cut area with other backgrounds. It can be observed that there lacks a dataset that can smoothly edit the object attribute.

## 3 Preliminaries

Since the editing tool is developed based on diffusion models, let us first briefly review the theory of denoising diffusion probabilistic models (DDPM) [24, 46] and analyze how it can be used to generate images.

According to the definition of the Markov Chain, one can always reach a desired stationary distribution from a given distribution along with the Markov Chain [14]. To get a generative model that can generate images from random Gaussian noises, one only needs to construct a Markov Chain whose stationary distribution is Gaussian distribution. This is the core idea of DDPM. In DDPM, given a data distribution \(\mathbf{x}_{0}\sim q(\mathbf{x}_{0})\), a forward noising process produces a series of latents \(\mathbf{x}_{1},...,\mathbf{x}_{T}\) of the same dimensionality as the data \(\mathbf{x}_{0}\) by adding Gaussian noise with variance \(\beta_{t}\in(0,1)\) at time \(t\):

\[q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\sqrt{1-\beta_{t}}\mathbf{x}_{ t-1},\beta_{t}\mathbf{I}),s.t.\ 0<\beta_{t}<1, \tag{1}\]

where \(\beta_{t}\) is the diffusion rate. Then the distribution \(q(\mathbf{x}_{t}|\mathbf{x}_{0})\) at any time \(t\) is:

\[q(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}(\sqrt{\bar{\alpha}_{t}},(1-\bar{ \alpha}_{t})\mathbf{I}),\ \mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{ \alpha}_{t}}\epsilon \tag{2}\]

where \(\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{t})\), \(\epsilon\sim\mathcal{N}(0,\mathbf{I})\). It can be proved that \(\lim_{t\to\infty}q(\mathbf{x}_{t})=\mathcal{N}(0,\mathbf{I})\). In other words, we can map the original data distribution into a Gaussian distribution with enough iterations. Such a stochastic forward process is named as diffusion process since what the process \(q(\mathbf{x}_{t}|\mathbf{x}_{t-1})\) does is adding noise to \(\mathbf{x}_{t-1}\).

To draw a fresh sample from the distribution \(q(\mathbf{x}_{0})\), the Markov process is reversed. That is, beginning from a Gaussian noise sample \(\mathbf{x}_{T}\sim\mathcal{N}(0,\mathbf{I})\), a reverse sequence is constructed by sampling the posteriors \(q(\mathbf{x}_{t-1}|\mathbf{x}_{t})\). To approximate the unknown function \(q(\mathbf{x}_{t-1}|\mathbf{x}_{t})\), in DDPMs, a deep model \(p_{\theta}\) is trained to predict the mean and the covariance of \(\mathbf{x}_{t-1}\) given \(\mathbf{x}_{t}\) instead. Then the \(\mathbf{x}_{t-1}\) can be sampled from the normal distribution defined as:

\[p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mu_{\theta}(\mathbf{ x}_{t},t),\Sigma_{\theta}(\mathbf{x}_{t},t)). \tag{3}\]

In stead of inferring \(\mu_{\theta}(\mathbf{x}_{t},t)\) directly, [24] propose to predict the noise \(\epsilon_{\theta}(\mathbf{x}_{t},t)\) which was added to \(\mathbf{x}_{0}\) to get \(\mathbf{x}_{t}\) with Eq. (2). Then \(\mu_{\theta}(\mathbf{x}_{t},t)\) is:

\[\mu_{\theta}(\mathbf{x}_{t},t)=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left(\mathbf{ x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(\mathbf{x}_{t},t )\right). \tag{4}\]

[24] keep the value of \(\Sigma_{\theta}(\mathbf{x}_{t},t)\) to be constant. As a result, given a sample \(\mathbf{x}_{t}\) at time \(t\), with a trained model that can predict the noise \(\epsilon_{\theta}(\mathbf{x}_{t},t)\), we can get \(\mu_{\theta}(\mathbf{x}_{t},t)\) according to Eq. (4) to reach the \(\mathbf{x}_{t-1}\) with Equation (3) and eventually we can get to \(\mathbf{x}_{0}\).

Previous studies have shown that diffusion models can achieve superior image generation quality compared to the current state-of-the-art generative models [1]. Besides, there have been plenty of works on utilizing the DDPMs to generate samples with desired properties, such as semantic image translation [36], high fidelity data generation from low-density regions [44], _etc_. In this paper, we also choose the DDPM adopted in [1] as our generator.

## 4 Attribute Editing with Diffusion Models and ImageNet-E

Most previous robustness-related work has focused on the important challenges of robustness on adversarial examples [6], common corruptions [21]. They have found that higher clean accuracy usually leads to better robustness.

Figure 2: Attribute editing with DDPMs. Give an input image and its corresponding object mask, the object is firstly removed with inpainting operation to get the pure background image. Then, we leverage the diffusion process to edit the background image \(\mathbf{x}_{0}\) and object image coherently. \(\odot\) denotes the element-wise blending of these two images using the object mask. For background editing, the background complexity objective function is added during the diffusion process (Alg. 1, line \(5\)). For other object attributes editing, the object image needs to be transformed first (Alg. 2, line \(1\)).
Therefore, instead of exploring a new corruption that models may encounter in reality, we pay attention to the model debugging in terms of object attributes, hoping to provide new insights to clean accuracy improvement. In the following, we describe our object attribute editing tool and the generated ImageNet-E dataset in detail. The whole pipeline can be found in Fig. 2.

### Object Attribute Editing with Diffusion Models

**Background editing.** Most existing corruptions conduct manipulations on the whole image, as shown in Fig. 1. Compared to adding global corruptions that may hinder the visual quality, a more likely-to-happen way in reality is to manipulate the backgrounds to fool the model. Besides, it is shown that there exists a spurious correlation between labels and image backgrounds [12]. From this point, a background corruption benchmark is needed to evaluate the model's robustness. However, the existing background challenge dataset achieves background editing with copy-paste operation, resulting an obvious artifacts in generated images [50]. This may leave some doubts about whether the evaluation is precise since the dataset's distribution may have changed. To alleviate this concern, we adopt DDPM approach to incorporate background editing by adding a guiding loss that can lead to backgrounds with desired properties to make the generated images stay in/close to the original distribution. Specifically, we choose to manipulate the background in terms of texture complexity due to the hypothesis that an object should be observed more easily from simple backgrounds than from complicated ones. In general, the texture complexity can be evaluated with the gray-level co-occurrence matrix (GLCM) [16], which calculates the gray-level histogram to show the texture characteristic. However, the calculation of GLCM is non-differentiable, thus it cannot serve as the conditional guidance of image generation. We hypothesize that a complex image should contain more frequency components in its spectrum and higher amplitude indicates greater complexity. Thus, we define the objective of complexity as:

\[\mathcal{L}_{c}=\sum\left|\mathcal{A}(\mathcal{F}(\mathbf{x}))\right|, \tag{5}\]

where \(\mathcal{F}\) is the Fourier transform [45], \(\mathcal{A}\) extracts the amplitude of the input spectrum. \(\mathbf{x}\) is the evaluated image. Since minimizing this loss helps us generate an image with desired properties and should be conducted on the \(\mathbf{x}_{0}\), we need a way of estimating a clean image \(\mathbf{x}_{0}\) from each noisy latent representation \(\mathbf{x}_{t}\) during the denoising diffusion process. Recall that the process estimates at each step the noise \(\epsilon_{\theta}(\mathbf{x}_{t},t)\) added to \(\mathbf{x}_{0}\) to obtain \(\mathbf{x}_{t}\). Thus, \(\hat{\mathbf{x}}_{0}\) can be estimated via Equation (6) [1]. The whole optimization procedure is shown in Algorithm 1.

\[\hat{\mathbf{x}}_{0}=\frac{\mathbf{x}_{t}}{\sqrt{\hat{\alpha}_{t}}}-\frac{ \sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}(\mathbf{x}_{t},t)}{\sqrt{\bar{ \alpha}_{t}}}. \tag{6}\]

As shown in Fig. 3(a), with the proposed method, when we guide the generation procedure with the proposed objective towards the complex direction, it will return images with visually complex backgrounds. We also provide the GLCM dissimilarity and contrast of each image to make a quantitative analysis of the generated images. A higher dissimilarity/contrast score indicates a more complex image background [16]. It can be observed that the complexity is consistent with that calculated with GLCM, indicating the effectiveness of the proposed method.

**Controlling object size, position and direction.** In general, the human vision system is robust to position, direction and small size changes. Whether the deep models are also robust to these object attribute changes is still unknown to researchers. Therefore, we conduct the image editing with controls of object sizes, positions and directions to find the answer. For a valid evaluation on different attributes, all other variables should remain unchanged, especially the background. Therefore, we first disentangle the object and background with the in-painting strategy provided by [54]. Specifically, we mask the object area in input image \(\mathbf{x}\). Then we conduct in-painting to remove the object and get the pure background image \(\mathbf{x}^{b}\), as shown in Fig. 3(b) column \(3\). To realize the aforementioned object attribute controlling, we adopt the orthogonal transformation. Denote \(P\) as the pixel locations of object in image \(\mathbf{x}\) where \(P\in\mathbb{R}^{3\times N_{o}}\). \(N_{o}\) is the number of pixels belong to object and \(p_{i}=[x_{i},y_{i},1]^{T}\) is the position of object's \(i\)-th pixel. \(h^{\prime}\in[0,H-h],w^{\prime}\in[0,W-w]\) where \([x,y,w,h]\) stand for the enclosing rectangle of the object with mask \(M\). Then the newly edited \(\mathbf{x}[T_{\text{attribute}}\cdot P]=\mathbf{x}[P]\) and \(M[T_{\text{attribute}}\cdot P]=M[P]\), where

\[T_{\text{size}}=\left[\begin{smallmatrix}8&0&\Delta x\\ 0&s&\Delta y\\ 0&0&1\end{smallmatrix}\right],T_{\text{poi}}=\left[\begin{smallmatrix}1&0&w^{ \prime}\\ 0&1&h^{\prime}\\ 0&0&1\end{smallmatrix}\right],T_{\text{direction}}=\left[\begin{smallmatrix}\cos \theta&\sin\theta&0\\ -\sin\theta&\cos\theta&0\\ 0&0&1\end{smallmatrix}\right] \tag{7}\]

where \(s\) is the resize scale. \(\theta\) is the rotation angle. \(\Delta x=(1-s)\cdot(x+w/2),\Delta y=(1-s)\cdot(y+h/2)\).

With the background image \(\mathbf{x}^{b}\) and edited object \(\mathbf{x}^{o}\), a naive way is to place the object in the original image to the corresponding area of background image \(\mathbf{x}^{b}\) as \(M\odot\mathbf{x}^{o}+(1-M)\odot\mathbf{x}^{b}\). However, the result generated in this manner may look disharmonic, lacking a delicate adjustment to blending them together. Besides, as shown in Fig. 3(b) column \(3\), the object-removing operation may leave some artifacts behind, failing to produce a coherent and seamless result. To deal with this problem, we leverage DDPM models to blend them at different noise levels along the diffusion process. Denote the image with desired object attribute as \(\mathbf{x}^{o}\). Starting from the pure background image \(\mathbf{x}^{b}\) at time \(t_{0}\), at each stage, we perform a guided diffusion step with a latent \(\mathbf{x}_{t}\) to obtain the \(\mathbf{x}_{t-1}\) and at the same time, obtain a noised version of object image
\(\mathbf{x}_{t-1}^{o}\). Then the two latents are blended with the mask \(M\) as \(\mathbf{x}_{t-1}=M\odot\mathbf{x}_{t-1}^{o}+(1-M)\odot\mathbf{x}_{t-1}\). The DDPM denoising procedure may change the background. Thus a proper initial timing is required to maintain a high resemblance to the original background. We set the iteration steps \(t_{0}\) as 50 and 25 in Algorithm 1 and 2 respectively.

```
input : source image \(\mathbf{x}\), mask \(M\), diffusion model \((\mu_{\theta}(\mathbf{x}_{t}),\Sigma_{\theta}(\mathbf{x}_{t}))\), \(\bar{\alpha}_{t}\), \(\lambda\), iteration steps \(t_{0}\) output : edited image \(\mathbf{x}_{0}\)
1\(\mathbf{x}_{t_{0}}\sim\mathcal{N}(\sqrt{\bar{\alpha}_{t_{0}}}\mathbf{x},(1- \bar{\alpha}_{t_{0}})\mathbf{I})\);
2for\(t\gets t_{0}\)todo
3\(\hat{\mathbf{x}}_{0}\leftarrow\frac{\mathbf{x}_{t}}{\sqrt{\bar{\alpha}_{t_{0 }}}}-\frac{\sqrt{1-\bar{\alpha}_{t}}\bar{\alpha}_{t_{0}}(\mathbf{x}_{t},t)}{ \sqrt{\bar{\alpha}_{t}}}\);
4\(\nabla_{bg}\leftarrow\nabla_{bg}\mathcal{N}_{\theta_{\text{c}}}\mathcal{L}_{ \text{c}}(\hat{\mathbf{x}}_{0})\);
5\(\mathbf{x}_{t-1}^{b}\sim\mathcal{N}(\mu_{\theta}(\mathbf{x}_{t})+\lambda \Sigma_{\theta}(\mathbf{x}_{t})\nabla_{bg},\Sigma_{\theta}(\mathbf{x}_{t}))\);
6\(\mathbf{x}^{o}\sim\mathcal{N}(\sqrt{\bar{\alpha}_{t}}\mathbf{x},(1-\bar{ \alpha}_{t})\mathbf{I})\);
7\(\mathbf{x}_{t-1}\gets M\odot\mathbf{x}^{o}+(1-M)\odot\mathbf{x}_{t-1}^{b}\);
8
9 end while
```

**Algorithm 1**Background editing

### ImageNet-E dataset

With the tool above, we conduct object attribute editing including background, size, direction and position changes based on the large-scale ImageNet dataset [41] and ImageNet-S [11], which provides the mask annotation. To guarantee the dataset quality, we choose the animal classes from ImageNet classes such as dogs, fishes and birds, since they appear more in nature without messy backgrounds. Classes such as stove and mortarboard are removed. Finally, our dataset consists of \(47872\) images with \(373\) classes based on the initial 4352 images, each of which is applied 11 transforms. Detailed information can be found in Appendix A. For background editing, we choose three levels of the complexity, including \(\lambda=-20,\lambda=20\) and \(\lambda=20\)-adv with adversarial guidance (see Sec.B for details) instead of complexity. Larger \(\lambda\) indicates stronger guidance towards high complexity. For the object size, we design four levels of sizes in terms of the object pixel rates (\(=\text{sum}(M>0.5)/\text{sum}(M\geq 0)\)): \([\text{Full},0.1,0.08,0.05]\) where 'Full' indicates making the object as large as possible while maintaining its whole body inside the image. Smaller rates indicate smaller objects. For object position, we find that some objects hold a high object pixel rate in the whole image, resulting in a small \(H-h\). Take the first picture in Fig. 3 for example, the dog is big and it will make little visual differences after position changing. Thus, we adopt the data whose pixel rate is 0.05 as the initial images for the position-changing operation.

In contrast to benchmarks like ImageNet-C [21] giving images from different domains so that the model robustness in these situations may be assessed, our effort aims to give an editable image tool that can conduct model debugging with in-distribution (ID) data, in order to identify specific shortcomings of different models and provide some insights for clean accuracy improving. Thus, the data distribution should not differ much from the original ImageNet. We choose the out-of-distribution (OOD) detection methods Energy [33] and GradNorm [26] to evaluate whether our editing tool will move the edited image out of its original distribution. These OOD detection methods aim to distinguish the OOD examples from the ID exam

Figure 3: (a) Images generated with the proposed background complexity editing method. (b) Edited images with size changing. The Fréchet inception distance (FID) for pasting is 50.64 while it is 32.59 for ours, indicating the effectiveness of the leveraging of DDPMs.
ples. The results are shown in Fig. 4. \(x\)-axis is the ID score in terms of the quantities in Energy and GradNorm and \(y\)-axis is the frequency of each ID score. A high ID score indicates the detection method takes the input sample as the ID data. Compared to other datasets, our method barely changes the data distribution under both Energy (the 1st row) and GradNorm (the 2nd row) evaluation methods. Besides, the Frechet inception distance (FID) [23] for our ImageNet-E is 15.57 under the random background setting, while it is 34.99 for ImageNet-9 (background challenge). These all imply that our editing tool can ensure the proximity to the original ImageNet, thus can give a controlled evaluation on object attribute changes. To find out whether the DDPM will induce some degradation to our evaluation, we have conducted experiment in Tab. 1 with the setting \(\lambda=0\) during background editing. This operation will first add noises to the original and then denoise them. It can be found in "Inver" column that the degradation is negligible compared to degradation induced by attribute changes.

## 5 Experiments

We conduct evaluation experiments on various architectures including both CNNs (ResNet (RN) [19], DenseNet [25], EfficientNet (EF) [47], ResNest [53], ConvNeXt [35]) and transformer-based models (Vision-Transformer (ViT) [9], Swin-Transformer (Swin) [34]). Other state-of-the-art models that trained with extra data such as CLIP [39], EfficientNet-L2-Noisy-Student [51] are also evaluated in the Appendix. Apart from different sizes of these models, we have also evaluated their adversarially trained versions for comprehensive studies. We report the drop of top-1 accuracy as metric based on the idea that the attribute changes should induce little influence to a robust trained model. More experimental details and results of top-1 accuracy can be found in the Appendix.

### Robustness evaluation

**Normally trained models.** To find out whether the widely used models in computer vision have gained robustness against changes on different object attributes, we conduct extensive experiments on different models. As shown in Tab. 1, when only the background is edited towards high complexity, the average drop in top-1 accuracy is 9.23% (\(\lambda=20\)). This indicates that most models are sensitive to object background changes. Other attribute changes such as size and position can also lead to model performance degradation. For example, when changing the object pixel rate to \(0.05\), as shown in Fig. 1 row \(4\) in the'size' column, while we can still recognize the image correctly, the performance drop is 18.34% on average. We also find that the robustness under different object attributes is improved along with improvements in terms of clean accuracy (Original) on different models. Accordingly, a switch from an RN50 (92.69% top-1 accuracy) to a Swin-S (96.21%) leads to the drop in accuracy decrease from 15.72% to 10.20% on average. By this measure, models have become more and more capable of generalizing to different backgrounds, which implies that they indeed learn some robust features. This shows that object attribute robustness can be a good way to measure future progress in representation learning. We also observe that larger networks possess better robustness on the attribute editing. For example, swapping a Swin-S (96.21% top-1 accuracy) with the larger Swin-B (95.96% top-1 accuracy) leads to the decrease of the dropped accuracy from 10.20% to 8.99% when \(\lambda=20\). In a similar fashion, a ConvNeXt-T (9.32% drop) is less robust than the giant ConvNeXt-B (7.26%). Consequently, models with even more depth, width, and feature aggregation may attain further attribute robustness. Previous studies [30] have shown that zero-shot CLIP exhibits better out-of-distribution robustness than the finetuned CLIP, which is opposite to our ImageNet-E as shown in Tab. 1. This may serve as the evidence that our ImageNet-E has a good proximity to ImageNet. We also find that compared with fully

Figure 4: Distributions of ID score of different datasets in terms of the quantities in Energy (the first row) and GradNorm (the second row) for in-distribution (ImageNet) and other datasets. Higher overlap indicates greater proximity to ImageNet.
supervised trained model under the same backbone (ViT-B), the CLIP fails to show a better attribute robustness. We think this may be caused by that the CLIP has spared some capacity for OOD robustness.

**Adversarially trained models.** Adversarial training [42] is one of the state-of-the-art methods for improving the adversarial robustness of deep models and has been widely studied [2]. To find out whether they can boost the attribute robustness, we conduct extensive experiments in terms of different architectures and perturbation budgets (constraints of \(l_{2}\) norm bound). As shown in Fig. 5, the adversarially trained ones are not robust against attribute changes including both backgrounds and size-changing situations. The dropped accuracies are much greater compared to normally trained models. As the perturbation budget grows, the situation gets worse. This indicates that adversarial training can do harm to robustness against attributes.

### Robustness enhancements

Based on the above evaluations, we step further to discover ways to enhance the attribute robustness in terms of preprocessing, network design and training strategies. More details including training setting and numerical experimental results can be found in Appendix C.5.

**Preprocessing.** Given that an object can be inconspicuous due to its small size or subtle position, viewing an object at several different locations may lead to a more stable prediction. Having this intuition in mind, we perform the classical Ten-Crop strategy to find out if this operation can help to get a robustness boost. The Ten-Crop operation is executed by cropping all four corners and the center of the input image. We average the predictions of these crops together with their horizontal mirrors as the final result. We find this operation can contribute a 0.69% and 1.24% performance boost on top-1 accuracy in both background and size changes scenarios on average respectively.

**Network designs.** Intuitively, a robust model should tend to focus more on the object of interest instead of the background. Therefore, recent models begin to enhance the model by employing attention modules. Of these, the ResNet [53] can be a representative. The ResNet is a modularized architecture, which applies channel-wise attention on different network branches to leverage their success in capturing cross-feature interactions and learning diverse representations. As it has achieved a great boost in the ImageNet dataset, it also shows superiority on ImageNet-E compared to ResNet. For example, a switch from RN50 decreases the average dropped accuracy from 15.72% to 12.57%. This indicates that the channel-wise attention module can be a good choice to improve the attribute robustness. Another representative model can be the vision transformer, which consists of multiple self-attention modules. To study whether incorporating transformer's self-attention-like architecture into the model design can help attribute robustness generalization, we establish a hybrid architecture by directly feeding the output of res_3 block in RN50 into ViT-S as the input feature like [3]. The dropped accuracy decreases by 1.04% compared to the original RN50, indicating the effectiveness of the self-attention-like architectures.

**Training strategy.** a) _Robust trained._ There have been plenty of studies focusing on the robust training strategy to improve model robustness. To find out whether these works can boost the robustness on our dataset, we further evaluate these state-of-the-art models including SIN [13], Debiased-CNN [31], Augmix [22], ANT [40], DeepAugment [20] and model trained with lots of standard augmentations (RN50-T) [48]. As shown in Tab. 2, apart from the RN50-T, while the Augmix model shows the best performance against the background change scenario, the Debiased model holds the best in the object size change scenario. What we find unexpectedly is the SIN performance. The SIN method features the novel data augmentation scheme where ImageNet images are stylized with style transfer as the training data to force the model to rely less on textural cues for classification. Though the robustness boost is achieved on ImageNet-C (mCE 69.32%) compared to its vanilla model (mCE 76.7%), it fails to improve the robustness in both object background and size-changing scenarios. The drops of top-1 accuracy for vanilla RN50 and RN50-SIN are 21.26% and 24.23% respectively, when the object size rate is 0.05, though they share similar accuracy on original ImageNet. This indicates that existing benchmarks cannot reflect the real robustness in object attribute changing. Therefore, a dataset like ImageNet-E is necessary for comprehensive evaluations on deep models. b) _Masked image modeling._ Considering that masked image modeling has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches [4], it may be robust to the attribute changes. Therefore, we choose the Masked AutoEncoder (MAE) [17] as the training strategy since its objective is recovering images with only 25% patches. Specifically, we adopt the MAE training strategy with ViT-B backbone and then finetune it with ImageNet

Figure 5: Comparisons between vanilla models and adversarially trained models across different architectures in terms of size changes (left). Evaluation of adversarial models (RN50) trained with different perturbation budgets is provided in the right figure.
training data. We find that the robustness is improved. For example, the dropped accuracy decreases from 10.62% to 9.05% on average compared to vanilla ViT-B.

### Failure case analysis

To explore the reason why some robust trained models may fail, we leverage the LayerCAM [28] to generate the heat map for different models including vanilla RN50, RN50+SIN and RN50+Debiased for comprehensive studies. As shown in Fig. 6, the heat map of the Debiased model aligns better with the objects in the image than that of the original model. It is interesting to find that the SIN model sometimes makes wrong predictions even with its attention on the main object. We suspect that the SIN model relies too much on the shape. for example, the'sea urchin' looks like the 'acron' with the shadow. However, its texture clearly indicates that it is the'sea urchin'. In contrast, the Debiased model which is trained to focus on both the shape and texture can recognize it correctly. More studies can be found in Appendix C.4.

### Model repairing

To validate that the evaluation on ImageNet (IN)-E can help to provide some insights for model's applicability and enhancement, we conduct a toy example for model repairing. Previous evaluation shows that the ResNet50 is vulnerable to background changes. Based on this observation, we randomly replace the backgrounds of objects with others during training and get a validation accuracy boost from 77.48% to 79.00%. Note that the promotion is not small as only 8781 training images with mask annotations are available in ImageNet. We also step further to find out if the improved model can get a boost the OOD robustness, as shown in the Tab. 3. It can be observed that with the insights provided by the evaluation on ImageNet-E, one can explore the model's attribute vulnerabilities and manage to

\begin{table}
\begin{tabular}{c|c|c c c c c|c c c|c|c|c} \hline \multirow{2}{*}{Models} & \multirow{2}{*}{Original} & \multicolumn{6}{c|}{Background changes} & \multicolumn{6}{c|}{Size changes} & Position & Direction & \multirow{2}{*}{Avg.} \\ \cline{3-3} \cline{5-16}  & & Inver & \(\lambda=-20\) & \(\lambda=20\) & \(\lambda=20\)-adv & Random & Full & 0.1 & 0.08 & 0.05 & rp & rd \\ \hline \hline RN50 & 92.69\% & 1.97\% & 7.30\% & 13.35\% & 29.92\% & 13.34\% & 2.71\% & 7.25\% & 10.51\% & 21.26\% & 26.46\% & 25.12\% & 15.72\% \\ DenseNet121 & 92.10\% & 1.49\% & 6.29\% & 9.00\% & 29.20\% & 12.43\% & 3.50\% & 7.00\% & 10.68\% & 21.55\% & 26.53\% & 23.64\% & 14.98\% \\ EF-B0 & 92.85\% & 1.07\% & 7.10\% & 10.71\% & 34.88\% & 15.64\% & 3.03\% & 8.00\% & 11.57\% & 23.28\% & 27.91\% & 19.15\% & 16.12\% \\ ResNet50 & 93.83\% & 1.44\% & 6.33\% & 8.98\% & 26.62\% & 11.28\% & 2.53\% & 5.27\% & 8.01\% & 18.03\% & 21.37\% & 17.32\% & 12.57\% \\ ViT-S & 94.14\% & **0.82\%** & 6.42\% & 8.98\% & 31.12\% & 13.06\% & **0.80\%** & 5.37\% & 8.95\% & 17.37\% & 22.86\% & 17.13\% & 13.17\% \\ Swin-S & **96.21\%** & 1.13\% & 5.18\% & 7.33\% & 23.50\% & 9.31\% & 1.27\% & 4.21\% & 6.29\% & 14.16\% & 17.35\% & **13.42\%** & 10.20\% \\ ConvNeXt-T & 96.07\% & 1.43\% & **4.69\%** & **6.26\%** & **19.83\%** & **7.93\%** & 1.75\% & **3.28\%** & **5.18\%** & **12.76\%** & **15.71\%** & 15.78\% & **9.32\%** \\ \hline \hline RN101 & 94.00\% & 2.11\% & 7.05\% & 11.62\% & 29.47\% & 13.57\% & 2.57\% & 6.81\% & 10.12\% & 26.05\% & 25.85\% & 24.42\% & 15.21\% \\ DenseNet169 & 92.37\% & 1.12\% & 5.81\% & 8.43\% & 27.51\% & 11.61\% & 2.25\% & 6.90\% & 10.41\% & 20.59\% & 24.93\% & 20.68\% & 13.91\% \\ EF-B3 & 94.97\% & 1.87\% & 7.77\% & 8.40\% & 29.90\% & 12.92\% & 1.36\% & 6.80\% & 10.16\% & 21.36\% & 24.98\% & 17.24\% & 14.09\% \\ ResNet101 & 95.54\% & 1.10\% & 5.58\% & 6.65\% & 23.03\% & 10.40\% & 1.35\% & 3.97\% & 6.53\% & 15.44\% & 19.11\% & 14.31\% & 10.64\% \\ ViT-B & 95.38\% & 0.83\% & 5.32\% & 8.43\% & 26.60\% & 10.98\% & **0.62\%** & 4.00\% & 6.30\% & 14.51\% & 18.82\% & 14.95\% & 11.05\% \\ Swin-B & 95.96\% & 0.79\% & 4.46\% & 6.23\% & 21.44\% & 8.25\% & 0.99\% & 3.16\% & 5.04\% & 12.34\% & 13.35\% & **12.60\%** & **8.99\%** \\ ConvNeXt-B & **96.42\%** & **0.69\%** & **3.75\%** & **4.86\%** & **16.49\%** & **6.04\%** & 9.09\% & **2.25\%** & **3.36\%** & **9.47\%** & **12.40\%** & 13.01\% & **7.26\%** \\ \hline CLIP-zeroshot & 80.01\% & 4.88\% & 11.56\% & 15.28\% & 36.14\% & 20.09\% & 3.33\% & 12.67\% & 15.77\% & 25.31\% & 28.87\% & 21.57\% & 19.06\% \\ CLIP-finetuned & 93.68\% & 2.17\% & 9.82\% & 11.83\% & 38.33\% & 18.19\% & 9.06\% & 9.25\% & 12.67\% & 23.32\% & 28.56\% & 22.00\% & 18.30\% \\ \hline \end{tabular}
\end{table}
Table 1: Evaluations with different state-of-the-art models in terms of Top-1 accuracy and the corresponding droped accuracy under background changes, size changes, random position (rp) and random direction (rd).

\begin{table}
\begin{tabular}{c|c|c c c c|c c c c|c} \hline \multirow{2}{*}{Architectures} & \multirow{2}{*}{Ori} & \multicolumn{6}{c|}{Background changes} & \multicolumn{6}{c|}{Size changes} & Position & Direction & \multirow{2}{*}{Avg.} \\ \cline{3-3} \cline{5-16}  & & Inver & \(\lambda=-20\) & \(\lambda=20\) & \(\lambda=20\)-adv & Random & Full & 0.1 & 0.08 & 0.05 & rp & rd \\ \hline RN50 & 92.69\% & 1.97\% & 7.30\% & 13.35\% & 29.92\% & 13.34\% & 2.71\% & 7.25\% & 10.51\% & 21.26\% & 26.46\% & 25.12\% & 15.72\% \\ RN50-Adversarial & 81.96\% & **0.66\%** & **4.75\%** & 13.62\% & 37.87\% & 15.25\% & 4.87\% & 9.62\% & 13.94\% & 25.51\% & 31.96\% & 18.99\% \\ RN50-SIN & 91.57\% & 2.23\% & 7.61\% & 12.19\% & 33.16\% & 13.58\% & 1.68\% & 8.30\% & 12.60\% & 24.23\% & 29.16\% & 27.24\% & 16.98\% \\ RN50-Debiased & 93.34\% & 1.43\% & 6.09\% & 11.45\% & 27.99\% & 12.12\% & 1.98\% & 5.53\%
repair the model and get a performance boost accordingly.

## 6 Conclusion and Future work

In this paper, we put forward an image editing toolkit that can take control of object attributes smoothly. With this tool, we create a new dataset called ImageNet-E that can serve as a general dataset for benchmarking robustness against different object attributes. Extensive evaluations conducted on different state-of-the-art models show that most models are vulnerable to attribute changes, especially the adversarially trained ones. Meanwhile, other robust trained models can show worse results than vanilla models even when they have achieved a great robustness boost on other robustness benchmarks. We further discover ways for robustness enhancement from both preprocessing, network designing and training strategies.

**Limitations and future work.** This paper proposes to edit the object attributes in terms of backgrounds, sizes, positions and directions. Therefore, the annotated mask of the interest object is required, resulting in a limitation of our method. Besides, since our editing toolkit is developed based on diffusion models, the generalization ability is determined by DDPMs. For example, we find synthesizing high-quality person images is difficult for DDPMs. Under the consideration of both the annotated mask and data quality, our ImageNet-E is a compact test set. In our future work, we would like to explore how to leverage the edited data to enhance the model's performance, including both the validation accuracy and robustness.

\begin{table}
\begin{tabular}{c|c|c c c c c c} \hline \hline Models & IN & IN-v2 & IN-A & IN-C\(\downarrow\) & IN-R & IN-Sketch & IN-E \\ \hline RN50 & 77.5 & 65.7 & 6.5 & 68.6 & 39.6 & 27.5 & 83.7 \\ RN50-repaired & **79.0** & **67.2** & **9.4** & **65.8** & **40.7** & **29.4** & **85.0** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Model repairing results. Top-1 accuracy (%) is reported except for IN-C, which is mCE (mean Corruption Error). Higher top-1 accuracy and lower mCE indicate better performance. IN-E reports the average accuracy on ImageNet-E.