Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Further, we show that our methods provide robustness (1) against natural, out-of-distribution data, (2) against multiple simultaneous distributional shifts, and (3) to domains entirely unseen during training.

The review comment to be evaluated is:

The paper extends current adversarial learning approaches beyond imperceptible L_p norm perturbations. The proposed approach can handle many models of natural variation, such as a change in brightness. The main idea behind the approach is to use unsupervised approaches such as GANs to model the natural variation. Given this model of natural variation, the paper replaces the adversarial learning objective of finding the worst example in an L_p norm ball around a data point to finding the worst example based on the model of natural variation. This is expensive, so the paper also proposes more computationally efficient approaches based on data augmentation. The experimental results demonstrate that the proposed approach performs well on a variety of tasks.

Strengths:

1. The question of training robust neural networks is clearly timely and well-motivated. A valid raised criticism of the current adversarial examples literature is that it is too tied to imperceptible or L_p norm perturbations, and this paper tries to go beyond them.
2. There is a bit of a lack of relevant methods to compare against, but the experiments still show that the proposed approach does well. I also appreciated the comparisons to domain adaption, and the multiple experimental setups generally.

Weaknesses:

1. Though the paper suggests that the approach is a “paradigm shift” in robust deep learning, it appears to be more or less an extension of the current adversarial examples literature. The adversarial training approach has been quite successful in defending against a fixed adversary model, from various L_p norms to geometric transformations. This paper is essentially defining a new adversary model, given by natural variations. In fact it is simpler than this, since the goal is to only do well on "random variations" in the predefined test set, i.e. there is no adversary which will try and attack the network by using the class of variations. There is a novelty here though that the variation model is learnt rather than pre-specified. But given that learning transformations such as change in brightness etc. is quite straightforward for unsupervised models such as GANs, this is not too much to ask for. Therefore, I find it a bit unsurprising that the proposed method works well, given the success of adversarial training and GANs.
2. Related to the above, in my view one main drawback of the current adversarial learning literature is that it is not difficult to get robustness to one pre-specified model of perturbation/variation, but in reality the space of possible variations is quite large. It is good that the proposed approach does seem to work on two simultaneous shifts, but the approach still seems restricted and not a substantial change in the current understanding. Note that some of the compared approaches are different in this regard: AugMix does not tailor the model to a particular variation (it explicitly excludes operations which overlap with ImageNet-C corruptions) and domain adaptation can handle quite large changes in the data distribution (which are not just transformations of the original data points).
3. Perhaps less importantly, in the experiments the proposed MDA approach based on data augmentation does almost as well as the other ones, and data augmentation using unsupervised models is already studied in the literature.

Overall, this is not a bad paper and I am not completely opposed to acceptance, but I am not sure I can argue for it.

Other less important comments:

1. Since there is a lack of baselines to compare against, the paper could benefit from a few ablation studies. For example, is there any difference on using other unsupervised approaches?
2. I think it would be better to have a concise list of a few major contributions in Section 1, rather than the current rather long list.
3. It would be nice to have representative images of the models of variation, perhaps in an appendix.

------Updates after author response------

I thank the authors for the response and it helps clarify some points. However, I am still not unconvinced that the paper is a significant departure from current work on adversarial robustness (see weakness 1 and 2 above). I think it would be much more interesting if the approach could yield robustness against a wider/different class of shifts compared to what it was adversarially trained against. Therefore, I am keeping my score at 5 and will not advocate for acceptance, though I am not completely opposed to it.

The evidence to be evaluated is:Notably, DL is known to be fragile to many forms of _natural variation_, which cannot be described by small perturbations \(x\mapsto x+\delta\). In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019; Hosseini and Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018; Karanakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation. Underlying the task of improving the robustness of neural networks against natural, out-of-distribution data are two fundamental challenges. Firstly, unlike in the adversarial robustness community, in real-world, safety-critical environments, data can vary in unknown and highly nonlinear ways. Thus, the first step toward building a robust training procedure must be to design a mechanism that accurately describes how data varies in such environments.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\).

In the remainder of this subsection, we consider cases in which (1) a model \(G\) is known a priori, and (2) a model \(G\) is unknown and therefore must be learned offline from data.

In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al., 2017)(Kamath et al., 2020).

In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

By training the network to correctly classify this worst-case data, the goal is to become invariant to the model \(G(x,\delta)\) for any \(\delta\in\Delta\). We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

Specifically, we propose three algorithmic variants: _Model-based Adversarial Training_ (MAT), _Model-based Robust Training_ (MRT), and _Model-based Data Augmentation_ (MDA).

First, in Sections 5.1, 5.2, we show that our algorithms are the first to consistently provide out-of-distribution robustness across a range of challenging corruptions, including shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the "snow" subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019). In practice, it is common to encounter multiple simultaneous distributional shifts. For example, in image classification, there may be shifts in both brightness and contrast; yet while there may be examples corresponding to shifts in either brightness or contrast in the training data, there may not be any examples of both shifts occurring simultaneously. To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation.

However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table 4 shows several experiments of this stripe in which a model \(G\) is learned on one dataset \(\mathcal{D}_{1}\) and then applied on another \(\mathcal{D}_{2}\); we improve robustness on unseen domains by up to 40 percentage points.

In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points.

Aside from the algorithms we introduced in Section2 we are not aware of any other algorithms that can be used to address out-of-distribution robustness across the diverse array of tasks presented in the previous section.

In the domain adaptation literature, various methods have been proposed which rely on the restrictive assumption that unlabeled data corresponding to a fixed distributional shift is available during training (Tzeng et al., 2017; Ajakan et al., 2014; Ganin and Lempitsky, 2015). Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In our framework we improve robustness against more challenging, natural shifts by learning from _unpaired_ datasets and we do not rely on class-conditioning to generate realistic images.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

On robustness and transferability of convolutional neural networks.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input: Table 4: Transferability of model-based robustness. In each experiment, we train a model of natural variation on a given training dataset D1. Then, we use this model to perform model-based training on a new dataset D1 entirely unseen during the training of the model.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my analysis of the claim against the evidence and review comments:

The claim appears moderately overstated. While the evidence demonstrates successful robustness against some natural variations and multiple simultaneous shifts (shown in Tables 1 and 4), the reviewer correctly points out that the approach is still restricted to specific pre-specified variations rather than truly handling "arbitrary" or "entirely unseen" domains as claimed. The results show improvements over baselines but not universal robustness. The method builds incrementally on existing adversarial training approaches rather than representing a fundamental paradigm shift.

Key limiting factors:
- Works mainly on pre-specified/learned variations rather than truly arbitrary shifts
- Success depends on having related training data
- Results show improvement but not universal robustness across all scenarios
- Builds on existing methods rather than representing a fundamental breakthrough

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Further, we show that our methods provide robustness (1) against natural, out-of-distribution data, (2) against multiple simultaneous distributional shifts, and (3) to domains entirely unseen during training.

The review comment to be evaluated is:

This paper proposes a model-based framework for improving the robustness of image classifiers to average-case corruptions of varying severity. The proposed framework can be thought of as adversarial training where the perturbation is replaced by a function that transforms the image according to a specific corruption. A nuisance parameter controls the instantiation and severity of the corruption that is applied to the input. The paper compares baselines to different versions of this general model-based framework with experiments on several datasets.

Pros:
- Focusing on average-case corruptions is an important and underexplored problem
- Using image translation to learn the corruptions is an interesting proposal

Cons:
- There isn't much novelty in the proposed technique. Adversarial training with 'natural' corruptions has been done several times before.
- There isn't any discussion of how the nuisance parameter is used. I suspect it isn't used for ImageNet-C experiments, though I may be wrong. In this case, the method becomes data augmentation with image translation, which is not very novel either.
- It sounds like the ImageNet-C experiments use the corruptions to train the image translation network. Even though these experiments use held-out images, the corruption types themselves are not being held out, which goes against the recommended methodology for that dataset.

Typo: "ImageNet-c" should be "ImageNet-C"

___________________________________________________________________________________

Update after author feedback:

I thank the authors for improving my understanding of the paper. I feel better about it after reading it again. One of the most interesting findings--that a translation network trained on one domain/set of classes generalizes to another--needs more discussion. Using this phenomenon to improve robustness is a good idea, and the MRT/MDA/MAT methods explored in this work are nice choices for this investigation. However, I agree with the other reviewers that it would be nice to include more baselines from other work where appropriate.

The results in Table 1 are also very interesting and deserve more discussion--perhaps an analysis of whether a translation network trained on weak corruptions can generalize to create something akin to the ground-truth stronger corruptions, as the results in the table for MRT imply.

Overall, I think this paper has some strong experiments and investigates a good idea, but the claims of a "paradigm shift" are overly grandiose, and some of the most interesting experiments could use more analysis. Additionally, the writing clarity and presentation could be improved. My initial understanding of the paper was flawed in some places, so I'll raise my score to 5.

The evidence to be evaluated is:Notably, DL is known to be fragile to many forms of _natural variation_, which cannot be described by small perturbations \(x\mapsto x+\delta\). In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019; Hosseini and Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018; Karanakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation. Underlying the task of improving the robustness of neural networks against natural, out-of-distribution data are two fundamental challenges. Firstly, unlike in the adversarial robustness community, in real-world, safety-critical environments, data can vary in unknown and highly nonlinear ways. Thus, the first step toward building a robust training procedure must be to design a mechanism that accurately describes how data varies in such environments.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\).

In the remainder of this subsection, we consider cases in which (1) a model \(G\) is known a priori, and (2) a model \(G\) is unknown and therefore must be learned offline from data.

In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al., 2017)(Kamath et al., 2020).

In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

By training the network to correctly classify this worst-case data, the goal is to become invariant to the model \(G(x,\delta)\) for any \(\delta\in\Delta\). We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

Specifically, we propose three algorithmic variants: _Model-based Adversarial Training_ (MAT), _Model-based Robust Training_ (MRT), and _Model-based Data Augmentation_ (MDA).

First, in Sections 5.1, 5.2, we show that our algorithms are the first to consistently provide out-of-distribution robustness across a range of challenging corruptions, including shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the "snow" subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019). In practice, it is common to encounter multiple simultaneous distributional shifts. For example, in image classification, there may be shifts in both brightness and contrast; yet while there may be examples corresponding to shifts in either brightness or contrast in the training data, there may not be any examples of both shifts occurring simultaneously. To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation.

However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table 4 shows several experiments of this stripe in which a model \(G\) is learned on one dataset \(\mathcal{D}_{1}\) and then applied on another \(\mathcal{D}_{2}\); we improve robustness on unseen domains by up to 40 percentage points.

In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points.

Aside from the algorithms we introduced in Section2 we are not aware of any other algorithms that can be used to address out-of-distribution robustness across the diverse array of tasks presented in the previous section.

In the domain adaptation literature, various methods have been proposed which rely on the restrictive assumption that unlabeled data corresponding to a fixed distributional shift is available during training (Tzeng et al., 2017; Ajakan et al., 2014; Ganin and Lempitsky, 2015). Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In our framework we improve robustness against more challenging, natural shifts by learning from _unpaired_ datasets and we do not rely on class-conditioning to generate realistic images.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

On robustness and transferability of convolutional neural networks.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input: Table 4: Transferability of model-based robustness. In each experiment, we train a model of natural variation on a given training dataset D1. Then, we use this model to perform model-based training on a new dataset D1 entirely unseen during the training of the model.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, I find the claim to be moderately overstated. While the evidence shows impressive results for robustness against natural variations and some transfer learning capabilities, there are key limitations:

1. For "out-of-distribution" robustness: The ImageNet-C experiments used corruption types that weren't fully held out (as noted in review), weakening claims about generalization.

2. For "multiple simultaneous shifts": Limited evidence is presented beyond mentioning the capability.

3. For "domains entirely unseen": Table 4 shows mixed results - while some transfers work well (MNIST variants), others show modest improvements (ImageNet to CURE) or minimal gains (brightness shifts).

The claim would be better stated with more measured language acknowledging these limitations and varying degrees of success across different scenarios.

<score>4</score>

Justification:

1. The reviewer points out that ImageNet-C experiments used corruption types for training, contradicting true out-of-distribution testing.

2. Limited empirical evidence is provided for simultaneous shifts despite being a key claim.

3. Table 4 shows variable transfer performance:
- Strong results (80-98%) for MNIST variants
- Moderate gains for ImageNet→CURE (62.2% vs 52-53% baselines)
- Minimal gains for some brightness shifts (47.5% vs ~41%)

4. Claims of being "the first to consistently provide out-of-distribution robustness" are strong given:
- Reviewer noting similar prior work ("Adversarial training with 'natural' corruptions has been done several times before")
- Mixed empirical results across scenarios

The evidence supports positive but varying degrees of robustness improvement, making the unqualified claim of robustness across all three categories an overstatement.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Further, we show that our methods provide robustness (1) against natural, out-of-distribution data, (2) against multiple simultaneous distributional shifts, and (3) to domains entirely unseen during training.

The review comment to be evaluated is:

The paper looks at the idea of building models of natural variation of an input and then using these models to develop robust training algorithms that are less susceptible to outliers ( testable for worst-cases via adversarial attacks ).

Strength:

+ The paper addresses a very important topic of adversarial robustness of DNN models and is accompanied by diligent evaluation over different datasets.

Weakness:

- The paper adopts an approach which is very reminiscent of Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1712.09196, 2017. The key difference seems to be adoption of the idea of using auxiliary transformations (called natural perturbations) which is also very well-studied in literature, for e.g. see https://openaccess.thecvf.com/content_CVPR_2020/papers/Zhang_Auxiliary_Training_Towards_Accurate_and_Robust_Models_CVPR_2020_paper.pdf  So, the approach presented here is quite incremental for a premier venue such as ICLR.

- The attack in Jalal et. al. paper can be launched against this approach once the attacker also access to this generative/natural model (which would be easy to build for an attacker)

- The reviewer will strongly recommend reviewing the advices in https://arxiv.org/abs/1902.06705 for writing papers on defense and how to self-evaluate its robustness by suitably designing the attacker. If the defense approach uses some background/auxiliary knowledge, one must consider the attacker with this knowledge if it is accessible to the attacker.

Questions to authors:

- Can authors explain why an attacker can't build similar natural model to defeat the proposed defense?
- Any clarification on incremental novelty from  Jalal et. al. and Zhang et. al. (and references therein) would be also useful.

After author's rebuttal:

"The other paper that the review points to ([4]) addresses a similar setting to our paper, but the approach is completely different. ... Moreover, [4] was published within three months of the ICLR submission deadline, meaning that it is essentially concurrent with our work."

Yes, the reviewer concurs that a work published so close to deadline should be treated as concurrent and will raise the score.

The evidence to be evaluated is:Notably, DL is known to be fragile to many forms of _natural variation_, which cannot be described by small perturbations \(x\mapsto x+\delta\). In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019; Hosseini and Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018; Karanakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation. Underlying the task of improving the robustness of neural networks against natural, out-of-distribution data are two fundamental challenges. Firstly, unlike in the adversarial robustness community, in real-world, safety-critical environments, data can vary in unknown and highly nonlinear ways. Thus, the first step toward building a robust training procedure must be to design a mechanism that accurately describes how data varies in such environments.

In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define _models of natural variation_. Concretely, a model of natural variation \(G(x,\delta)\) is a map that describes how an input datum \(x\) can be naturally varied by a _nuisance parameter_\(\delta\) resulting in a new datum \(x^{\prime}:=G(x,\delta)\). Ideally, for a fixed datum \(x\), varying the nuisance parameter \(\delta\) should vary the severity of the natural conditions in the generated datum \(x^{\prime}\).

In the remainder of this subsection, we consider cases in which (1) a model \(G\) is known a priori, and (2) a model \(G\) is unknown and therefore must be learned offline from data.

In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al., 2017)(Kamath et al., 2020).

In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 1 in which a model \(G(x,\delta)\) takes an image \(x\) of a street sign in sunny weather and maps it to an image \(x^{\prime}:=G(x,\delta)\) in snowy weather. Even though there is a relationship between the two images, obtaining a model \(G\) relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for _learning_ the model \(G\) from data. To do so, we assume that we have access to two unpaired domains \(A\) and \(B\) that are drawn from a common distribution. Domain \(A\) contains the original data, such as the images with sunny weather, and domain \(B\) contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain \(A\) to corresponding images with different levels of natural variation captured by the images of domain \(B\).

By training the network to correctly classify this worst-case data, the goal is to become invariant to the model \(G(x,\delta)\) for any \(\delta\in\Delta\). We now assume that we have access to a suitable model of natural variation \(G(x,\delta)\) and shift our attention toward exploiting \(G\) in the development of novel robust training algorithms.

Specifically, we propose three algorithmic variants: _Model-based Adversarial Training_ (MAT), _Model-based Robust Training_ (MRT), and _Model-based Data Augmentation_ (MDA).

First, in Sections 5.1, 5.2, we show that our algorithms are the first to consistently provide out-of-distribution robustness across a range of challenging corruptions, including shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3 we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training.

In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of natural variation. To do so, we use data from the CURE-TSR dataset (Hemel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the "snow" subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1 we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR.

To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks and Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019). In practice, it is common to encounter multiple simultaneous distributional shifts. For example, in image classification, there may be shifts in both brightness and contrast; yet while there may be examples corresponding to shifts in either brightness or contrast in the training data, there may not be any examples of both shifts occurring simultaneously. To address this robustness challenge, Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are _entirely unseen_ while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation.

However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table 4 shows several experiments of this stripe in which a model \(G\) is learned on one dataset \(\mathcal{D}_{1}\) and then applied on another \(\mathcal{D}_{2}\); we improve robustness on unseen domains by up to 40 percentage points.

In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points.

Aside from the algorithms we introduced in Section2 we are not aware of any other algorithms that can be used to address out-of-distribution robustness across the diverse array of tasks presented in the previous section.

In the domain adaptation literature, various methods have been proposed which rely on the restrictive assumption that unlabeled data corresponding to a fixed distributional shift is available during training (Tzeng et al., 2017; Ajakan et al., 2014; Ganin and Lempitsky, 2015). Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training.

On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.

In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets.

In our framework we improve robustness against more challenging, natural shifts by learning from _unpaired_ datasets and we do not rely on class-conditioning to generate realistic images.

In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.

On robustness and transferability of convolutional neural networks.

Input: Table 1: Out-of-distribution robustness. In each experiment, we train a model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.

Input: Figure 1: A new notion of robustness. Past work has focused on perturbation-based adversarial examples, such as Figure 1a. In this paper, we focus on robustness with respect to natural variation, shown in Figure 1b, which often does not obey perceptual or norm-bounded constraints.

Input: Table 4: Transferability of model-based robustness. In each experiment, we train a model of natural variation on a given training dataset D1. Then, we use this model to perform model-based training on a new dataset D1 entirely unseen during the training of the model.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence:

The claim states their methods provide robustness against (1) natural out-of-distribution data, (2) multiple simultaneous shifts, and (3) unseen domains. The evidence shows strong experimental results across multiple datasets (CURE-TSR, ImageNet, ImageNet-C) demonstrating improved robustness against natural variations like snow, fog, and brightness changes. Table 4 supports transfer learning to unseen domains with significant accuracy improvements. However, the review points out potential vulnerability to attackers who could build similar natural models, which isn't acknowledged as a limitation. The results, while positive, may not guarantee robustness in all scenarios.

<score>2</score>

Justification:
1. The claim is largely supported by comprehensive empirical evidence showing improved performance across various natural variations and datasets
2. Results show consistent outperformance of baselines, often by 10-20 percentage points
3. Transfer learning capabilities are demonstrated with up to 40% improvements on unseen domains
4. However, the reviewer raises valid concerns about potential vulnerabilities that aren't addressed in the claim
5. The claim would be better stated with some qualification about potential limitations or specific conditions under which robustness is achieved

The claim is slightly overstated but generally well-supported by the evidence, earning a score of 2.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To this end, we propose to learn a sampler that _learants_ to sample the most important samples for a given task, by training it jointly with the target task and additionally meta-learn a sampler over a distribution of datasets for instance selection in the classification task.

The evidence to be evaluated is:Furthermore, our method can be applied both at the instance and dataset level.

Balin et al. (2019) learns a global feature selection model for reconstruction of the input data from selected features via unsupervised learning. Chen et al. (2018) learns instancewise feature selection with the goal of model interpretation by extracting subset of features most informative for a given sample. Our method also falls in this category.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_.

When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\).

In order to select \(D_{s}\), we need to model the interactions among the elements of \(D\) and construct \(D_{s}\) based on said interactions.

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Since _Ber_ is non-differentiable, we use the continuous relaxations of the Bernoulli distribution introduced in (Maddison et al., 2016; Jang et al., 2016; Gal et al., 2017). Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\). We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

3:procedureStochastic Subset Selection(\(k,q,D\)) 4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

8:for\(i=1,\dots,k/q\)do\(\triangleright\) AutoRegressive Subset Selection 9:\(D_{s}\gets D_{s}\cup\text{AutoSelect}(q,D_{s},D_{c})\)\(\triangleright\) Select \(q\) elements 10:return\(D_{s}\) 11:procedureAutoSelect(\(q,D_{s},D_{c}\)) 12:\(C=\{w_{1},w_{2},\dots,w_{m}\}\gets D_{c}\setminus D_{s}\) 13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\). We adopt the idea of Information Bottleneck and constrain the distribution of \(Z\) for \(D_{c}\).

Given \(D_{s}\) and a network \(p_{\theta}(Y|X,D_{s})\) parameterized by \(\theta\), the task is to reconstruct \(D=(X,Y)\). The objective function for this task is given as: \[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(Y|X,D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{5}\] Minimizing this objective ensures that we learn a compact subset(\(D_{s}\)) most representative of \(D\) and \(D_{s}\) can then be used for other tasks.

An ANP takes as input a context(\(D_{s}\) in this case) and predicts a distribution of the elements in the original set \(D\).

We can also opt to train the network to predict a single target \(y_{D}\) for the set \(D\). For instance, the target could be the class of an image(classification) or the statistics of the set(regression problem).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(y_{D}|D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{6}\] **Dataset Distillation: Instance Selection**

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\).

Additionally, to ensure that \(c\) is properly learnt, we add an informativity loss by reconstructing \(c\) from the generated samples from the given dataset.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

The goal is to learn to select a subset for a target task such as classification or regression.

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task. Note that LTS cannot be applied to this experimental setup because during training, the generated virtual points cannot be converted back to an image in matrix form (due to the virtual coordinate), thus we cannot train the LTS model with CNN-based classification on the target task.

**Instance Selection** We present results on the instance selection task applied to a whole dataset. In this task, we use the CelebA dataset since it has an imbalance both in terms of gender and race. A dataset is constructed by sampling 200 random images from the full dataset. In this experiment, we seek to select only a few(5-30) representative images from these generated datasets. On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

The results for this experiment is shown in Table 3, where it can be observed that SSS can learn to select more representative prototypes compared to the other methods especially in the few-shot problems where the choice of prototypes matters more.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

We then show some visualization of our method for feature selection (in both 1D function and CelebA dataset) and report the results with multiple runs of the instance selection experiment, as well as its visualization. Qualitative results for Instance Selection as applied to the few-shot classification task are provided together with model specifications.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples.

5: a target task with loss function \(\ell(\cdot,\cdot)\)

9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\) 10:\(D^{(j)}_{\theta}\sim p(D^{(j)}_{\theta}|D^{(j)})\) for \(j=1\ldots m\) 11:\(i\sim\) random sample from \((0,\ldots,k-q)\) 12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

In Figure 11 we provide visualizations for the instance selection problem as applied to the few-shot classification task. Here, we go from a 20-shot to a 1-shot classification problem where the prototype is selected from the support using SSS. SSS consists of \(r(D)\), \(\rho(d_{i},r(D))\) and \(f(d,D_{c},D_{s}^{(i-1)}\). We describe the models in this section.

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 4: Models’ performance on set reconstruction and classification task (lighter-color areas present the standard deviations). (a) 1D Function reconstruction. (b) CelebA (reconstruction). (c) CelebA (classification)

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 1: Concept: Our Stochastic Subset Selection method is generic and can be applied to many type of sets. (a) Selecting a subset of features (pixels) out of an image. This reduces communication cost in data transfer between devices and allows faster training or inference on resource-constrained devices due to the reduced computational cost. (b) Selecting a subset of instances out of a dataset. This helps resource-constrained system to train faster, or to make non-parametric inference more scalable.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 7: Instance Selection Samples from a dataset of size 200.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, this claim appears moderately overstated. While the evidence demonstrates that the authors developed a sampler that can select important samples for classification tasks, the claim's wording about "learants" (likely a typo for "learns") and the broad statement about meta-learning "over a distribution of datasets" isn't fully supported by the evidence shown. The paper does present a method for subset selection that works at both instance and dataset levels, but the meta-learning aspect isn't as prominently featured in the results as the claim suggests. The evidence mainly focuses on the two-stage selection process (candidate selection and autoregressive subset selection) and shows applications in specific tasks rather than broad meta-learning capabilities.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To this end, we propose to learn a sampler that _learants_ to sample the most important samples for a given task, by training it jointly with the target task and additionally meta-learn a sampler over a distribution of datasets for instance selection in the classification task.

The review comment to be evaluated is:

**Final recommendation**
I support accepting this paper. While I am a bit disappointed the authors did not add in the paper the results they discussed during the rebuttal, I think the paper is interesting and clearer than at submission.

**Summary**
stochastic subset selection (SSS) is a method to learn to compress a set $D$ by selecting a subset $D_s$ such that the loss of a task performed on $D_s$ is as close as possible to the loss if the task had been performed on the original $D$. The paper show how this general method can be applied to several tasks. For example, $D$ can contain the pixels of an image and the task can be to reconstruct the original image from the subset. This corresponds to learning to compress. As another example, with the same $D$ but a label prediction task, SSS learns to dynamically select sparse features for a classification task. The other tasks considered are dataset distillation (compressing a data set) and selecting prototypes for few-shot classification. $D_s$ is constructed in two steps. A candidate set is first constructed by considering elements of $D$ independently and in a second step elements of $D_c$ are included in $D_s$ iteratively by considering other elements in $D_c$ and in $D_s$. Experiments are performed for one example of each task previously discussed and SSS is shown to outperform baselines. SSS is motivated by the need to reduce bandwidth, computation and memory footprint of deep learning in edge devices.

---
**Strong points**

1) Code is provided (I did not try to run it).

2) The proposed approach is applied to four different tasks. This convinced me of its applicability.

3) Empirical experiments seem well conducted (with some caveat, see weak point 3) and results are good.

4) The problems tackled are interesting.

**Weak points**

1) The paper is very dense, which makes it hard to follow. Related to that, I think that some parts of the paper are too concise to understand them well. The supplementary material partly compensates that.

2) While the experiments show that the proposed approach works well, I think that some experiments should compare full data pipelines. As an example, one experiment shows that using SSS to dynamically select pixels of an image and using this sparse representation as input leads to good classification accuracy. The paper implicitely suggests that this reduces inference cost if classification is performed on the device and bandwidth if inference is performed on another device. However, the cost of the pixel selection by SSS is never analyzed. I think comparing the memory footprint, the number of operations, bandwidth used and accuracy for all steps of the following pipelines would give a much clearer picture of the interest of the approach for deep learning on mobile devices:
    - classification is done directly on the device, using an optimized network.
    - pixels are randomly selected and classification is performed on the device.
    - pixels are randomly selected, transmitted and classification is performed on another device.
    - pixels are selected by SSS, which involves computing multiple embeddings, and classification is performed on the device.
    - pixels are selected by SSS, transmitted and classification is performed on another device.

---
**Recommendation**
Overall, I vote for accepting. I am on the fence with this paper. The current results are good and I think the proposed approach has some potential and will be interesting for the community. On the other hand the paper is hard to read and the experiments, while good to show the strength of the approach, do not cover usefullness for the mobile use case. I think the former can be adressed in time and that the latter, while important, does not prevent the paper to be published.

---
**Details**

I like that each task is first formally described and then evaluated in a specific case with the experiments. However I only understood most tasks when reading the experiment sections. I would suggest to either provide a short illustration of an actual instance of the task when describing the task formally, to directly provide the experimental result after describing the task or to direct the reader to the relevant experimental section(s) for illustration.

The paper states that model descriptions are available in appendix A, but I could not find them.

For the *Set Classification/Prediction* task, I would suggest clarifying that for this task a set typically contains the features of a single example, so $D=x$. I only understood this when reading the experimental section.

The *Dataset Distillation: Instance Selection* task was not clear to me. Here are a few points that puzzled me.
- The text suggest the proposed approach is applied to each $D_i$, but figure 3(c) has no $D_i$ and $i$ seems to be used as an index of $x$ rather than $D$. Furthermore, in figure 3(c) a single D_s is constructed whereas I was expecting one per $D_i$.
- The paper states that *It is important to note that $c$ is computed using only $D_i$ for every element in $D$*. However $c$ also depends on *a query $\alpha$ which is computed using $d_i$*, which, if I understood correctly, is the element of $D$ and can be outside $D_i$. So it seems that $c$ is not computed using only $D_i$.

For the *Dataset Distillation: Classification* task, in figure 3d, the labels are not observed. How can the model learn?

1) For this task, if the label is observed, how is this different from the first task?

2) For the Image Reconstruction experiment, why is $x$ 2 dimensional? Are the inputs black and white images?

I think that the image reconstruction experiment should include a comparison to compression algorithms. I agree with the paper that the proposed approach is applicable to many other types of data than images, but it would be very interesting to see how the approach compares to expert defined methods. Furthermore, implementation in relation to mobile devices is the motivation of this work and as far as I know classical compression algorithms are still used on mobile devices.

---
**Questions**

Questions 1 and 2 in *details* above.

I would also be interested to hear authors' opinions on weak point 2 and a comparison to compression.

For classification, pixels not selected are set to 0. Wouldn't the inference cost be the same when using convolutional networks?

---
**Minor details**

For the *set reconstruction* task, would it also be interesting to learn a model on $D_s$ and use it to predict $Y$ from $X$?

*typos*
foro

The evidence to be evaluated is:Furthermore, our method can be applied both at the instance and dataset level.

Balin et al. (2019) learns a global feature selection model for reconstruction of the input data from selected features via unsupervised learning. Chen et al. (2018) learns instancewise feature selection with the goal of model interpretation by extracting subset of features most informative for a given sample. Our method also falls in this category.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_.

When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\).

In order to select \(D_{s}\), we need to model the interactions among the elements of \(D\) and construct \(D_{s}\) based on said interactions.

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Since _Ber_ is non-differentiable, we use the continuous relaxations of the Bernoulli distribution introduced in (Maddison et al., 2016; Jang et al., 2016; Gal et al., 2017). Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\). We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

3:procedureStochastic Subset Selection(\(k,q,D\)) 4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

8:for\(i=1,\dots,k/q\)do\(\triangleright\) AutoRegressive Subset Selection 9:\(D_{s}\gets D_{s}\cup\text{AutoSelect}(q,D_{s},D_{c})\)\(\triangleright\) Select \(q\) elements 10:return\(D_{s}\) 11:procedureAutoSelect(\(q,D_{s},D_{c}\)) 12:\(C=\{w_{1},w_{2},\dots,w_{m}\}\gets D_{c}\setminus D_{s}\) 13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\). We adopt the idea of Information Bottleneck and constrain the distribution of \(Z\) for \(D_{c}\).

Given \(D_{s}\) and a network \(p_{\theta}(Y|X,D_{s})\) parameterized by \(\theta\), the task is to reconstruct \(D=(X,Y)\). The objective function for this task is given as: \[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(Y|X,D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{5}\] Minimizing this objective ensures that we learn a compact subset(\(D_{s}\)) most representative of \(D\) and \(D_{s}\) can then be used for other tasks.

An ANP takes as input a context(\(D_{s}\) in this case) and predicts a distribution of the elements in the original set \(D\).

We can also opt to train the network to predict a single target \(y_{D}\) for the set \(D\). For instance, the target could be the class of an image(classification) or the statistics of the set(regression problem).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(y_{D}|D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{6}\] **Dataset Distillation: Instance Selection**

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\).

Additionally, to ensure that \(c\) is properly learnt, we add an informativity loss by reconstructing \(c\) from the generated samples from the given dataset.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

The goal is to learn to select a subset for a target task such as classification or regression.

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task. Note that LTS cannot be applied to this experimental setup because during training, the generated virtual points cannot be converted back to an image in matrix form (due to the virtual coordinate), thus we cannot train the LTS model with CNN-based classification on the target task.

**Instance Selection** We present results on the instance selection task applied to a whole dataset. In this task, we use the CelebA dataset since it has an imbalance both in terms of gender and race. A dataset is constructed by sampling 200 random images from the full dataset. In this experiment, we seek to select only a few(5-30) representative images from these generated datasets. On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

The results for this experiment is shown in Table 3, where it can be observed that SSS can learn to select more representative prototypes compared to the other methods especially in the few-shot problems where the choice of prototypes matters more.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

We then show some visualization of our method for feature selection (in both 1D function and CelebA dataset) and report the results with multiple runs of the instance selection experiment, as well as its visualization. Qualitative results for Instance Selection as applied to the few-shot classification task are provided together with model specifications.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples.

5: a target task with loss function \(\ell(\cdot,\cdot)\)

9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\) 10:\(D^{(j)}_{\theta}\sim p(D^{(j)}_{\theta}|D^{(j)})\) for \(j=1\ldots m\) 11:\(i\sim\) random sample from \((0,\ldots,k-q)\) 12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

In Figure 11 we provide visualizations for the instance selection problem as applied to the few-shot classification task. Here, we go from a 20-shot to a 1-shot classification problem where the prototype is selected from the support using SSS. SSS consists of \(r(D)\), \(\rho(d_{i},r(D))\) and \(f(d,D_{c},D_{s}^{(i-1)}\). We describe the models in this section.

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 4: Models’ performance on set reconstruction and classification task (lighter-color areas present the standard deviations). (a) 1D Function reconstruction. (b) CelebA (reconstruction). (c) CelebA (classification)

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 1: Concept: Our Stochastic Subset Selection method is generic and can be applied to many type of sets. (a) Selecting a subset of features (pixels) out of an image. This reduces communication cost in data transfer between devices and allows faster training or inference on resource-constrained devices due to the reduced computational cost. (b) Selecting a subset of instances out of a dataset. This helps resource-constrained system to train faster, or to make non-parametric inference more scalable.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 7: Instance Selection Samples from a dataset of size 200.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, the claim appears somewhat overstated. While the evidence shows the method can effectively learn to sample important elements for various tasks (reconstruction, classification, instance selection), the term "learants" is a typo and unclear. Additionally, while the method does work across different datasets, the claim of "meta-learning a sampler over a distribution of datasets" is only briefly mentioned in the evidence without substantial experimental validation of this specific capability. The experiments demonstrate good performance on individual tasks but don't extensively validate the meta-learning aspect.

<score>3</score>