Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Critical to our paradigm is to obtain _models of natural variation_, which vary data over a range of natural conditions.

The review comment to be evaluated is:

The paper extends current adversarial learning approaches beyond imperceptible L_p norm perturbations. The proposed approach can handle many models of natural variation, such as a change in brightness. The main idea behind the approach is to use unsupervised approaches such as GANs to model the natural variation. Given this model of natural variation, the paper replaces the adversarial learning objective of finding the worst example in an L_p norm ball around a data point to finding the worst example based on the model of natural variation. This is expensive, so the paper also proposes more computationally efficient approaches based on data augmentation. The experimental results demonstrate that the proposed approach performs well on a variety of tasks.

Strengths:

1. The question of training robust neural networks is clearly timely and well-motivated. A valid raised criticism of the current adversarial examples literature is that it is too tied to imperceptible or L_p norm perturbations, and this paper tries to go beyond them.
2. There is a bit of a lack of relevant methods to compare against, but the experiments still show that the proposed approach does well. I also appreciated the comparisons to domain adaption, and the multiple experimental setups generally.

Weaknesses:

1. Though the paper suggests that the approach is a “paradigm shift” in robust deep learning, it appears to be more or less an extension of the current adversarial examples literature. The adversarial training approach has been quite successful in defending against a fixed adversary model, from various L_p norms to geometric transformations. This paper is essentially defining a new adversary model, given by natural variations. In fact it is simpler than this, since the goal is to only do well on "random variations" in the predefined test set, i.e. there is no adversary which will try and attack the network by using the class of variations. There is a novelty here though that the variation model is learnt rather than pre-specified. But given that learning transformations such as change in brightness etc. is quite straightforward for unsupervised models such as GANs, this is not too much to ask for. Therefore, I find it a bit unsurprising that the proposed method works well, given the success of adversarial training and GANs.
2. Related to the above, in my view one main drawback of the current adversarial learning literature is that it is not difficult to get robustness to one pre-specified model of perturbation/variation, but in reality the space of possible variations is quite large. It is good that the proposed approach does seem to work on two simultaneous shifts, but the approach still seems restricted and not a substantial change in the current understanding. Note that some of the compared approaches are different in this regard: AugMix does not tailor the model to a particular variation (it explicitly excludes operations which overlap with ImageNet-C corruptions) and domain adaptation can handle quite large changes in the data distribution (which are not just transformations of the original data points).
3. Perhaps less importantly, in the experiments the proposed MDA approach based on data augmentation does almost as well as the other ones, and data augmentation using unsupervised models is already studied in the literature.

Overall, this is not a bad paper and I am not completely opposed to acceptance, but I am not sure I can argue for it.

Other less important comments:

1. Since there is a lack of baselines to compare against, the paper could benefit from a few ablation studies. For example, is there any difference on using other unsupervised approaches?
2. I think it would be better to have a concise list of a few major contributions in Section 1, rather than the current rather long list.
3. It would be nice to have representative images of the models of variation, perhaps in an appendix.

------Updates after author response------

I thank the authors for the response and it helps clarify some points. However, I am still not unconvinced that the paper is a significant departure from current work on adversarial robustness (see weakness 1 and 2 above). I think it would be much more interesting if the approach could yield robustness against a wider/different class of shifts compared to what it was adversarially trained against. Therefore, I am keeping my score at 5 and will not advocate for acceptance, though I am not completely opposed to it.

The evidence to be evaluated is:In this paper, we consider a standard classification task in which training data \((x,y)\sim\mathcal{D}\) is distributed according to a joint distribution \(\mathcal{D}\) over instances \(x\in\mathbb{R}^{d}\) and labels \(y\in[k]:=\{0,1,\dots,k\}\). In this setting, given a finite training sample drawn i.i.d. from \(\mathcal{D}\), the goal of the learning problem is to obtain a classifier \(f_{w}\) parameterized by weights \(w\in\mathbb{R}^{p}\) such that \(f_{w}\) can correctly predict the labels \(y\) corresponding to new instances \(x\) drawn i.i.d. from \(\mathcal{D}\).

Notably, DL is known to be fragile to many forms of _natural variation_, which cannot be described by small perturbations \(x\mapsto x+\delta\). In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019; Hosseini and Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018; Karanakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\).

**Learning models of natural variation from data.** In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

In this paper, we assume that data can be transformed according to a model of natural variation \(G(x,\delta)\) by choosing different values of \(\delta\) from a given _nuisance space_\(\Delta\). The goal of the model-based approach is to train a classifier that achieves high accuracy both on a test set drawn i.i.d. from \(\mathcal{D}\) and on _more-challenging_ test data that has been subjected to the source of natural variation that \(G\) models.

We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

In MDA, rather than explicitly trying to solve the inner problem, we seek a _diversity_ of naturally-varying data rather than the "worst-case."

In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

We note that our experiments contain domains with both _natural_ and _artificially-generated_ variation; details concerning how we extracted non-artificial variation can be found in Appendix D

In many applications, one might have data corresponding to low levels of natural variation, such as a dusting of snow in images of street signs. However, it is often difficult to collect data corresponding to high levels of natural variation, such as images taken during a blizzard. In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the "snow" subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019).

To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation. For example, one might have access to two domains corresponding to the shift from images of European street signs taken during the day to images taken at night. However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs.

In each row, we use unlabeled data from both domains to train a model of natural variation.

Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In this paper, we proposed a novel _model-based robust training paradigm_ for deep learning that provides robustness with respect to models of natural variation.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that obtaining "models of natural variation" is critical to their paradigm appears well-stated and supported by the extensive experimental details and results provided. The evidence demonstrates:

1. Clear methodology for learning and applying models of natural variation
2. Experimental validation across multiple datasets (CURE-TSR, ImageNet)
3. Demonstrated ability to handle various types of natural variations (snow, haze, rain, etc.)
4. Quantitative results showing effectiveness (Table 1)
5. The reviewer's comments, while noting some limitations, acknowledge that the approach works well for modeling natural variations like brightness changes

The claim directly aligns with the paper's core contribution and is supported by comprehensive evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Critical to our paradigm is to obtain _models of natural variation_, which vary data over a range of natural conditions.

The review comment to be evaluated is:

This paper proposes a model-based framework for improving the robustness of image classifiers to average-case corruptions of varying severity. The proposed framework can be thought of as adversarial training where the perturbation is replaced by a function that transforms the image according to a specific corruption. A nuisance parameter controls the instantiation and severity of the corruption that is applied to the input. The paper compares baselines to different versions of this general model-based framework with experiments on several datasets.

Pros:
- Focusing on average-case corruptions is an important and underexplored problem
- Using image translation to learn the corruptions is an interesting proposal

Cons:
- There isn't much novelty in the proposed technique. Adversarial training with 'natural' corruptions has been done several times before.
- There isn't any discussion of how the nuisance parameter is used. I suspect it isn't used for ImageNet-C experiments, though I may be wrong. In this case, the method becomes data augmentation with image translation, which is not very novel either.
- It sounds like the ImageNet-C experiments use the corruptions to train the image translation network. Even though these experiments use held-out images, the corruption types themselves are not being held out, which goes against the recommended methodology for that dataset.

Typo: "ImageNet-c" should be "ImageNet-C"

___________________________________________________________________________________

Update after author feedback:

I thank the authors for improving my understanding of the paper. I feel better about it after reading it again. One of the most interesting findings--that a translation network trained on one domain/set of classes generalizes to another--needs more discussion. Using this phenomenon to improve robustness is a good idea, and the MRT/MDA/MAT methods explored in this work are nice choices for this investigation. However, I agree with the other reviewers that it would be nice to include more baselines from other work where appropriate.

The results in Table 1 are also very interesting and deserve more discussion--perhaps an analysis of whether a translation network trained on weak corruptions can generalize to create something akin to the ground-truth stronger corruptions, as the results in the table for MRT imply.

Overall, I think this paper has some strong experiments and investigates a good idea, but the claims of a "paradigm shift" are overly grandiose, and some of the most interesting experiments could use more analysis. Additionally, the writing clarity and presentation could be improved. My initial understanding of the paper was flawed in some places, so I'll raise my score to 5.

The evidence to be evaluated is:In this paper, we consider a standard classification task in which training data \((x,y)\sim\mathcal{D}\) is distributed according to a joint distribution \(\mathcal{D}\) over instances \(x\in\mathbb{R}^{d}\) and labels \(y\in[k]:=\{0,1,\dots,k\}\). In this setting, given a finite training sample drawn i.i.d. from \(\mathcal{D}\), the goal of the learning problem is to obtain a classifier \(f_{w}\) parameterized by weights \(w\in\mathbb{R}^{p}\) such that \(f_{w}\) can correctly predict the labels \(y\) corresponding to new instances \(x\) drawn i.i.d. from \(\mathcal{D}\).

Notably, DL is known to be fragile to many forms of _natural variation_, which cannot be described by small perturbations \(x\mapsto x+\delta\). In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019; Hosseini and Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018; Karanakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\).

**Learning models of natural variation from data.** In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

In this paper, we assume that data can be transformed according to a model of natural variation \(G(x,\delta)\) by choosing different values of \(\delta\) from a given _nuisance space_\(\Delta\). The goal of the model-based approach is to train a classifier that achieves high accuracy both on a test set drawn i.i.d. from \(\mathcal{D}\) and on _more-challenging_ test data that has been subjected to the source of natural variation that \(G\) models.

We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

In MDA, rather than explicitly trying to solve the inner problem, we seek a _diversity_ of naturally-varying data rather than the "worst-case."

In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

We note that our experiments contain domains with both _natural_ and _artificially-generated_ variation; details concerning how we extracted non-artificial variation can be found in Appendix D

In many applications, one might have data corresponding to low levels of natural variation, such as a dusting of snow in images of street signs. However, it is often difficult to collect data corresponding to high levels of natural variation, such as images taken during a blizzard. In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the "snow" subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019).

To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation. For example, one might have access to two domains corresponding to the shift from images of European street signs taken during the day to images taken at night. However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs.

In each row, we use unlabeled data from both domains to train a model of natural variation.

Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In this paper, we proposed a novel _model-based robust training paradigm_ for deep learning that provides robustness with respect to models of natural variation.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Critical to our paradigm is to obtain models of natural variation, which vary data over a range of natural conditions."

Summary: The evidence strongly supports this claim. The paper extensively demonstrates how models of natural variation are obtained and applied, including:
- Formal definition and mathematical framework for these models
- Practical examples showing variations like weather changes
- Experimental validation using CURE-TSR dataset with different severity levels
- Demonstration of model reusability across datasets
- Real results showing effectiveness on multiple natural variations (snow, haze, rain, etc.)

The claim is well-stated (score 1) because it:
- Accurately represents the core concept without exaggeration
- Is supported by comprehensive theoretical and experimental evidence
- Uses measured language ("critical" is justified given the central role these models play)
- The claims are backed by concrete results shown in Table 1

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Critical to our paradigm is to obtain _models of natural variation_, which vary data over a range of natural conditions.

The review comment to be evaluated is:

The paper looks at the idea of building models of natural variation of an input and then using these models to develop robust training algorithms that are less susceptible to outliers ( testable for worst-cases via adversarial attacks ).

Strength:

+ The paper addresses a very important topic of adversarial robustness of DNN models and is accompanied by diligent evaluation over different datasets.

Weakness:

- The paper adopts an approach which is very reminiscent of Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1712.09196, 2017. The key difference seems to be adoption of the idea of using auxiliary transformations (called natural perturbations) which is also very well-studied in literature, for e.g. see https://openaccess.thecvf.com/content_CVPR_2020/papers/Zhang_Auxiliary_Training_Towards_Accurate_and_Robust_Models_CVPR_2020_paper.pdf  So, the approach presented here is quite incremental for a premier venue such as ICLR.

- The attack in Jalal et. al. paper can be launched against this approach once the attacker also access to this generative/natural model (which would be easy to build for an attacker)

- The reviewer will strongly recommend reviewing the advices in https://arxiv.org/abs/1902.06705 for writing papers on defense and how to self-evaluate its robustness by suitably designing the attacker. If the defense approach uses some background/auxiliary knowledge, one must consider the attacker with this knowledge if it is accessible to the attacker.

Questions to authors:

- Can authors explain why an attacker can't build similar natural model to defeat the proposed defense?
- Any clarification on incremental novelty from  Jalal et. al. and Zhang et. al. (and references therein) would be also useful.

After author's rebuttal:

"The other paper that the review points to ([4]) addresses a similar setting to our paper, but the approach is completely different. ... Moreover, [4] was published within three months of the ICLR submission deadline, meaning that it is essentially concurrent with our work."

Yes, the reviewer concurs that a work published so close to deadline should be treated as concurrent and will raise the score.

The evidence to be evaluated is:In this paper, we consider a standard classification task in which training data \((x,y)\sim\mathcal{D}\) is distributed according to a joint distribution \(\mathcal{D}\) over instances \(x\in\mathbb{R}^{d}\) and labels \(y\in[k]:=\{0,1,\dots,k\}\). In this setting, given a finite training sample drawn i.i.d. from \(\mathcal{D}\), the goal of the learning problem is to obtain a classifier \(f_{w}\) parameterized by weights \(w\in\mathbb{R}^{p}\) such that \(f_{w}\) can correctly predict the labels \(y\) corresponding to new instances \(x\) drawn i.i.d. from \(\mathcal{D}\).

Notably, DL is known to be fragile to many forms of _natural variation_, which cannot be described by small perturbations \(x\mapsto x+\delta\). In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019; Hosseini and Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018; Karanakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\).

**Learning models of natural variation from data.** In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

In this paper, we assume that data can be transformed according to a model of natural variation \(G(x,\delta)\) by choosing different values of \(\delta\) from a given _nuisance space_\(\Delta\). The goal of the model-based approach is to train a classifier that achieves high accuracy both on a test set drawn i.i.d. from \(\mathcal{D}\) and on _more-challenging_ test data that has been subjected to the source of natural variation that \(G\) models.

We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

In MDA, rather than explicitly trying to solve the inner problem, we seek a _diversity_ of naturally-varying data rather than the "worst-case."

In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

We note that our experiments contain domains with both _natural_ and _artificially-generated_ variation; details concerning how we extracted non-artificial variation can be found in Appendix D

In many applications, one might have data corresponding to low levels of natural variation, such as a dusting of snow in images of street signs. However, it is often difficult to collect data corresponding to high levels of natural variation, such as images taken during a blizzard. In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the "snow" subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019).

To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation. For example, one might have access to two domains corresponding to the shift from images of European street signs taken during the day to images taken at night. However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs.

In each row, we use unlabeled data from both domains to train a model of natural variation.

Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In this paper, we proposed a novel _model-based robust training paradigm_ for deep learning that provides robustness with respect to models of natural variation.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence:

The claim "Critical to our paradigm is to obtain models of natural variation, which vary data over a range of natural conditions" appears to be well-stated and accurately reflects the evidence. The paper provides extensive evidence supporting this claim through:

1. Detailed theoretical framework for modeling natural variations
2. Experimental validation across multiple datasets (CURE-TSR, ImageNet)
3. Demonstrated applications with real-world variations (snow, rain, haze)
4. Table 1 showing empirical results on different natural variations
5. Clear visual examples in Figure 1 contrasting perturbation-based vs natural variation

The review comments don't dispute the centrality or implementation of natural variation models, focusing instead on novelty concerns. The claim avoids overstating capabilities while accurately representing the core approach.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Then by exploiting these models, we develop three novel model-based robust training algorithms that improve the robustness of DL with respect to natural variation.

The evidence to be evaluated is:However, neural networks trained using ERM are known to be susceptible to _adversarial attacks_. This means that given a datum \(x\) with a corresponding label \(y\), one can find another datum \(x^{\text{adv}}\) such that \(x\) is close to \(x^{\text{adv}}\) in a given Euclidean norm and \(x^{\text{adv}}\) is predicted by the learned classifier as belonging to a different class \(c\neq y\). If such a datum \(x^{\text{adv}}\) exists, it is called an _adversarial example_. This is illustrated in Figure (a)a although these pandas look identical, they were classified differently in Goodfellow et al. (2014). The dominant paradigm toward improving robustness against adversarial examples relies on a robust optimization perspective wherein neural networks are trained to correctly classify worst-case perturbations of data (Madry et al., 2017; Wong and Kolter, 2017).

Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation. Underlying the task of improving the robustness of neural networks against natural, out-of-distribution data are two fundamental challenges.

Thus, the first step toward building a robust training procedure must be to design a mechanism that accurately describes how data varies in such environments. Next, assuming a suitable model of natural variation, the second challenge is to formulate a training procedure that exploits this model toward improving robustness.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\). An example of such a model is shown in Figure 2 where an image \(x\) on the left (in this case, in any weather) can be naturally varied by \(\delta\) and consequently transformed into the image \(x^{\prime}\) on the right (in snowy weather). In the remainder of this subsection, we consider cases in which (1) a model \(G\) is known a priori, and (2) a model \(G\) is unknown and therefore must be learned offline from data. In this second case in which models of natural variation must be learned, we propose a method for obtaining these models.

For example, there is underlying structure that describes how data can be rotated, translated, or scaled; models for rotating an image can be characterized by \(G(x,\delta)=R(\delta)x\) where \(R(\delta)\) is a rotation matrix and \(\delta\in\Delta:=[0,2\pi)\). In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al., 2017)(Kamath et al., 2020).

In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

The model-based robust training paradigm that we propose retains the basic elements of adversarial training described in Section 2.

In this paper, we assume that data can be transformed according to a model of natural variation \(G(x,\delta)\) by choosing different values of \(\delta\) from a given _nuisance space_\(\Delta\). The goal of the model-based approach is to train a classifier that achieves high accuracy both on a test set drawn i.i.d. from \(\mathcal{D}\) and on _more-challenging_ test data that has been subjected to the source of natural variation that \(G\) models. This perspective can be captured by the following robust optimization problem: \[\min_{w}\mathbb{E}_{(x,y)\sim\mathcal{D}}\Big{[}\max_{\delta\in\Delta}\ell(G( x,\delta),y;w)\Big{]}. \tag{2}\] In the inner maximization problem of this formulation, given an instance-label pair \((x,y)\), we seek a vector \(\delta^{*}\in\Delta\) that produces a corresponding instance \(x^{\prime}:=G(x,\delta^{*})\) which gives rise to high loss values \(\ell(G(x,\delta^{*}),y;w)\) under the current weight \(w\). One can think of this vector \(\delta^{*}\) as characterizing the _worst-case_ nuisance that can be generated by the model \(G(x,\delta^{*})\) for the original instance \(x\). After solving this inner problem, we solve the outer minimization problem by finding weights \(w\) that minimize the risk against the challenging instance \(G(x,\delta^{*})\). By training the network to correctly classify this worst-case data, the goal is to become invariant to the model \(G(x,\delta)\) for any \(\delta\in\Delta\). We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

Specifically, we propose three algorithmic variants: _Model-based Adversarial Training_ (MAT), _Model-based Robust Training_ (MRT), and _Model-based Data Augmentation_ (MDA).

At a high level, each of these methods alternates between solving the outer minimization problem and the inner maximization problem. To this end, each of these algorithms uses SGD to solve the outer problem; the methods differ in how they seek solutions to the inner problem, and in what follows, we describe each of these procedures in more detail.

**Model-based Adversarial Training.** In MAT, we seek an exact solution to the inner problem by performing \(k\) steps of gradient ascent in \(\delta\) on the objective \(\ell(G(x,\delta),y;w)\). The resulting nuisance parameter \(\delta^{\star}\) is one that causes \(\ell(G(x,\delta^{\star}),y;w)\) to have high loss under the current weight \(w\). **Model-based Robust Training.** In MRT, we first randomly sample \(\delta_{i}\in\Delta\) for \(i\in[k]\). We then select the \(i^{\star}\in[k]\) such that \(\ell(G(x,\delta_{i^{\star}}),y;w)\) is maximized.

In MDA, rather than explicitly trying to solve the inner problem, we seek a _diversity_ of naturally-varying data rather than the "worst-case." In this way, MDA samples \(\delta_{i}\in\Delta\) for \(i\in[k]\) and then appends \(\{G(x,\delta_{i}),y\}_{i=1}^{k}\) to the training dataset.

First, in Sections 5.1, 5.2, we show that our algorithms are the first to consistently provide out-of-distribution robustness across a range of challenging corruptions, including shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels.

Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019).

To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation.

Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table 4 shows several experiments of this stripe in which a model \(G\) is learned on one dataset \(\mathcal{D}_{1}\) and then applied on another \(\mathcal{D}_{2}\); we improve robustness on unseen domains by up to 40 percentage points.

In each row, we use unlabeled data from both domains to train a model of natural variation.

In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points.

Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In this paper, we proposed a novel _model-based robust training paradigm_ for deep learning that provides robustness with respect to models of natural variation.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

* Dunn et al. (2019) Isaac Dunn, Tom Melham, and Daniel Kroening. Generating realistic unrestricted adversarial inputs using dual-objective gan training. _arXiv preprint arXiv:1905.02463_, 2019.

* Engstrom et al. (2017) Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. _arXiv preprint arXiv:1712.02779_, 2017.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input: Table 4: Transferability of model-based robustness. In each experiment, we train a model of natural variation on a given training dataset D1. Then, we use this model to perform model-based training on a new dataset D1 entirely unseen during the training of the model.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that while the authors did develop three novel training algorithms (MAT, MRT, and MDA) that show improvements in robustness against natural variations, the evidence substantiates their effectiveness but with some limitations. The experiments demonstrate significant improvements across multiple datasets and scenarios, often showing 10-20 percentage point gains. However, the claim could be more precise by acknowledging that the improvements are dataset/scenario dependent and not uniform across all cases, as shown in Tables 1 and 4 where performance varies considerably across different conditions.

<score>2</score>

Justification:
1. The evidence strongly supports that three novel algorithms were developed and implemented successfully
2. Tables 1 and 4 provide concrete evidence of improved robustness across multiple scenarios
3. The authors acknowledge limitations and specific conditions where the methods work best
4. The experimental results show variable performance rather than universal improvements
5. The algorithms require specific conditions (like having access to paired data) that could be more explicitly stated in the claim
6. The claim could be more precise about the scope and magnitude of improvements, but isn't substantially overstated given the evidence provided
7. Direct evidence citations show "significant out-of-distribution robustness" with improvements "often by 10-20 percentage points" and "up to 40 percentage points" in some cases

The claim is relatively well-stated but could benefit from minor qualification about the variability of improvements. Hence a score of 2 rather than 1.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Then by exploiting these models, we develop three novel model-based robust training algorithms that improve the robustness of DL with respect to natural variation.

The review comment to be evaluated is:

# Summary
This paper proposes a “paradigm shift” for augmenting datasets when training CNN-based image classifiers. On one side, traditional augmentations include blur, Gaussian noise, color distortions. On the other side, methods like adversarial training consider augmentations under norm bounds in the image space. The proposed method, instead, uses models of natural variation to augment the images. Increased out-of-domain accuracy is shown on ImageNet-C and several slices of the CURE-TSR dataset.

Central question: How to shift from perturbation-based robustness analysis to model-based robustness analysis?

# Strong and Weak points

## Strong points

  * This paper introduces a “paradigm shift” from perturbation based to model-based augmentation, when training CNN based image classifiers. The existing perturbations only reach so far, and using models of natural variation provides a new avenue for augmenting data sets.
  * The method is presented in clear pseudo code, and detailed in the 36 page counting appendix.

## Weak points

  * The paper claims a “paradigm shift”, but the proposed method of augmenting data has been used since 2012,  [1], and is used ever since [2]. Tables 2 and 3 provide key results for this method, but both were obtained with Model-based Data Augmentation, which is the common method before the "paradigm shift".
  * Results outlined in Section 5.2 and Table 2 concerns scenarios where the test-time perturbation is known at training time. Table 2 then compares MDA to ERM using this a-priori knowledge, and shows improved accuracy. However, the comparison is unbalanced, as ERM does not benefit from the prior knowledge. If Table 2 would be about comparing methods under the assumption that the test-time perturbation would be known at training time, then the ERM should be trained with the same data augmentation.
  * The paper misses comparisons against previous published work. Although Table 2 and 4 compare against another method, these results were not obtained by the respective authors, thus introducing a bias. The related work, Section 6, cites as many as 9 papers that addressed the same problem as this paper. The argument for the proposed method could be stronger when comparing to results from any of these papers.
Moreover, Table 3 reports results on ImageNet-C, which multiple other research papers have published results on. See for example, [3] and [4]. Likewise, Table 4 reports results on MNIST-C, which have been reported in [5] and [6]. Omitting those comparisons hampers the argument for the proposed method.

[1] Krizhevsky, 2012
[2] Szegedy, 2015
[3] https://paperswithcode.com/sota/domain-generalization-on-imagenet-c
[4] https://arxiv.org/pdf/1903.12261.pdf
[5] https://proceedings.icml.cc/paper/2020/file/20546457187cf3d52ea86538403e47cc-Paper.pdf
[6] https://arxiv.org/pdf/1906.02337.pdf

# Statement

Recommendation: Reject
Reason:
Although this paper provides four tables of extensive evaluation of the method, none of the numbers were obtained from previous literature. The related work section cites 9 papers on the same problem, but none were compared with in the results section. (See concrete details above)

# Clarifying questions

  * How to quantify the increase in computational complexity for the inner maximization in Equation (2)? As I understand, even evaluating one point requires a forward pass of the variation-model and the classifier-model. In case of iterative optimization, this would also require two backward passes(see line 5 in Algorithm (1)).
If the proposed method requires significantly more compute, the comparison should be made with a method that has access to the same amount of compute (for example a larger model, or longer trained model).
  * Line 6 of Algorithm (1). I miss the definition of set $\Delta$ and the projection operator in the text nor the appendix. Please explain?
  * Section 5.2 speaks of ImageNet “classes”. I am confused if these “classes” are referring to the ImageNet classes (1000 in total) or the severity levels in ImageNet-C (75 in total). Could you please clarify?
  Moreover, this argument implicitly assumes that the test-time perturbation is known at training-time. Could you name a scenario where this assumption is justified?
  * To what extent is the accuracy on the clean data preserved by the proposed method? Table 3, for example, shows an increase in top-1 accuracy on ImageNet data under perturbation. However, I wonder if this model preserves accuracy on the clean data. We know from existing literature [1] that a trade off exists for adversarial training between accuracy on perturbed and clean data. Providing the clean accuracy would provide evidence that this method compares favourably on this trade off.

[1] Zhang, Hongyang, et al. "Theoretically principled trade-off between robustness and accuracy." arXiv preprint arXiv:1901.08573 (2019).

# Minor feedback

This minor feedback is not part of the assessment.

  * “arguably more common” -> “arguably more important”
  * “x with a corresponding label y” -> “x with a corresponding prediction y”
  * Abbreviation “MBRDL” appears in Section 5 without any explanation or definition.
  * Abbreviation “CURE-TSR” appears in Section 5.0 without any explanation or definition.
  * The citation for CURE in Section 5.1 is wrong. The bibliography refers to “Temel, Dogancan, Min-Hung Chen, and Ghassan AlRegib. "Traffic sign detection under challenging conditions: A deeper look into performance variations and spectral characteristics." IEEE Transactions on Intelligent Transportation Systems (2019).”, but the CURE TSR dataset was published in “Temel, Dogancan, et al. "CURE-TSR: Challenging unreal and real environments for traffic sign recognition." arXiv preprint arXiv:1712.02463 (2017).”
  * Caption of Table 4: “ Then, we use this model to perform model-based training on a new dataset D1” -> “ Then, we use this model to perform model-based testing on a new dataset D2”?

The evidence to be evaluated is:However, neural networks trained using ERM are known to be susceptible to _adversarial attacks_. This means that given a datum \(x\) with a corresponding label \(y\), one can find another datum \(x^{\text{adv}}\) such that \(x\) is close to \(x^{\text{adv}}\) in a given Euclidean norm and \(x^{\text{adv}}\) is predicted by the learned classifier as belonging to a different class \(c\neq y\). If such a datum \(x^{\text{adv}}\) exists, it is called an _adversarial example_. This is illustrated in Figure (a)a although these pandas look identical, they were classified differently in Goodfellow et al. (2014). The dominant paradigm toward improving robustness against adversarial examples relies on a robust optimization perspective wherein neural networks are trained to correctly classify worst-case perturbations of data (Madry et al., 2017; Wong and Kolter, 2017).

Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation. Underlying the task of improving the robustness of neural networks against natural, out-of-distribution data are two fundamental challenges.

Thus, the first step toward building a robust training procedure must be to design a mechanism that accurately describes how data varies in such environments. Next, assuming a suitable model of natural variation, the second challenge is to formulate a training procedure that exploits this model toward improving robustness.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\). An example of such a model is shown in Figure 2 where an image \(x\) on the left (in this case, in any weather) can be naturally varied by \(\delta\) and consequently transformed into the image \(x^{\prime}\) on the right (in snowy weather). In the remainder of this subsection, we consider cases in which (1) a model \(G\) is known a priori, and (2) a model \(G\) is unknown and therefore must be learned offline from data. In this second case in which models of natural variation must be learned, we propose a method for obtaining these models.

For example, there is underlying structure that describes how data can be rotated, translated, or scaled; models for rotating an image can be characterized by \(G(x,\delta)=R(\delta)x\) where \(R(\delta)\) is a rotation matrix and \(\delta\in\Delta:=[0,2\pi)\). In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al., 2017)(Kamath et al., 2020).

In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

The model-based robust training paradigm that we propose retains the basic elements of adversarial training described in Section 2.

In this paper, we assume that data can be transformed according to a model of natural variation \(G(x,\delta)\) by choosing different values of \(\delta\) from a given _nuisance space_\(\Delta\). The goal of the model-based approach is to train a classifier that achieves high accuracy both on a test set drawn i.i.d. from \(\mathcal{D}\) and on _more-challenging_ test data that has been subjected to the source of natural variation that \(G\) models. This perspective can be captured by the following robust optimization problem: \[\min_{w}\mathbb{E}_{(x,y)\sim\mathcal{D}}\Big{[}\max_{\delta\in\Delta}\ell(G( x,\delta),y;w)\Big{]}. \tag{2}\] In the inner maximization problem of this formulation, given an instance-label pair \((x,y)\), we seek a vector \(\delta^{*}\in\Delta\) that produces a corresponding instance \(x^{\prime}:=G(x,\delta^{*})\) which gives rise to high loss values \(\ell(G(x,\delta^{*}),y;w)\) under the current weight \(w\). One can think of this vector \(\delta^{*}\) as characterizing the _worst-case_ nuisance that can be generated by the model \(G(x,\delta^{*})\) for the original instance \(x\). After solving this inner problem, we solve the outer minimization problem by finding weights \(w\) that minimize the risk against the challenging instance \(G(x,\delta^{*})\). By training the network to correctly classify this worst-case data, the goal is to become invariant to the model \(G(x,\delta)\) for any \(\delta\in\Delta\). We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

Specifically, we propose three algorithmic variants: _Model-based Adversarial Training_ (MAT), _Model-based Robust Training_ (MRT), and _Model-based Data Augmentation_ (MDA).

At a high level, each of these methods alternates between solving the outer minimization problem and the inner maximization problem. To this end, each of these algorithms uses SGD to solve the outer problem; the methods differ in how they seek solutions to the inner problem, and in what follows, we describe each of these procedures in more detail.

**Model-based Adversarial Training.** In MAT, we seek an exact solution to the inner problem by performing \(k\) steps of gradient ascent in \(\delta\) on the objective \(\ell(G(x,\delta),y;w)\). The resulting nuisance parameter \(\delta^{\star}\) is one that causes \(\ell(G(x,\delta^{\star}),y;w)\) to have high loss under the current weight \(w\). **Model-based Robust Training.** In MRT, we first randomly sample \(\delta_{i}\in\Delta\) for \(i\in[k]\). We then select the \(i^{\star}\in[k]\) such that \(\ell(G(x,\delta_{i^{\star}}),y;w)\) is maximized.

In MDA, rather than explicitly trying to solve the inner problem, we seek a _diversity_ of naturally-varying data rather than the "worst-case." In this way, MDA samples \(\delta_{i}\in\Delta\) for \(i\in[k]\) and then appends \(\{G(x,\delta_{i}),y\}_{i=1}^{k}\) to the training dataset.

First, in Sections 5.1, 5.2, we show that our algorithms are the first to consistently provide out-of-distribution robustness across a range of challenging corruptions, including shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels.

Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019).

To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation.

Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table 4 shows several experiments of this stripe in which a model \(G\) is learned on one dataset \(\mathcal{D}_{1}\) and then applied on another \(\mathcal{D}_{2}\); we improve robustness on unseen domains by up to 40 percentage points.

In each row, we use unlabeled data from both domains to train a model of natural variation.

In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points.

Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In this paper, we proposed a novel _model-based robust training paradigm_ for deep learning that provides robustness with respect to models of natural variation.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

* Dunn et al. (2019) Isaac Dunn, Tom Melham, and Daniel Kroening. Generating realistic unrestricted adversarial inputs using dual-objective gan training. _arXiv preprint arXiv:1905.02463_, 2019.

* Engstrom et al. (2017) Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. _arXiv preprint arXiv:1712.02779_, 2017.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input: Table 4: Transferability of model-based robustness. In each experiment, we train a model of natural variation on a given training dataset D1. Then, we use this model to perform model-based training on a new dataset D1 entirely unseen during the training of the model.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence and review comments, I'll evaluate the claim: "Then by exploiting these models, we develop three novel model-based robust training algorithms that improve the robustness of DL with respect to natural variation."

The paper proposes three model-based training algorithms (MAT, MRT, MDA) and demonstrates their effectiveness through experimental results on multiple datasets. However, according to the reviewer comments, there are significant concerns:

1. The method of augmenting data has been used since 2012, challenging the "novelty" claim
2. Key results were obtained using MDA, which is described as a "common method before the paradigm shift"
3. Missing comparisons against existing published work
4. Results lack validation against state-of-the-art methods

While the evidence shows improved performance, the claim of novelty appears overstated given the historical context and limited comparisons.

<score>4</score>