Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We aim for this framework to provide a publicly available,1 off-the-shelf evaluation when the feature-selection perspective on explanations is needed.

The review comment to be evaluated is:

-------------------- AFTER
The original rating of "Weak Reject" still holds as the authors failed to provide proper justification for the raise concerns and support their claims through additional experiments.

"We do not introduce an explanation generation framework, as explainers do. " - The proposed evaluation requires the explainer of the NLP model to agree with the RCNN in-terms of relevant or irrelevant words, to be considered a good explainer. The RCNN model which is defining the relevant and irrelevant tokens for a prediction task is in fact stating that we can explain the decision of an NLP model in terms of relevant and irrelevant tokens. Hence, the proposed RCNN can also be considered as an explainer. The evaluation task is demonstrating if the other explainers are providing explanations consistent with this new explainer based on RCNN.

"The RCNN is not meant to explain any other models except itself." - Unclear

"Regarding the request for more experiments:" - The authors don't provide enough justification to "why they didn't perform more experiments?"

" Hence, with our current instantiations, any domain-agnostic explainer can be evaluated" - The experiment to validate this claim are missing.

"The novelty of our paper consists in the fact that, to our knowledge, it is the first to (1) shed light over a fundamental difference" - This is not a technical novelty. This is an exploratory analysis based observation

"and (2) propose a methodology for evaluating explainers that ...and without human intervention (unlike evaluation type 4)." -  In Section 5 Qualitative Analysis, the authors are also doing human evaluation like other methods in evaluation type-4 of their related works. Also, doing human evaluation is a strong way to justify an explainer. Though expensive, whenever possible it should be done and is in no way a limitation of current evaluation metrics.

Your model needs labelled data for training RCNN. This adds a constraint on the usability and scalability of your proposed evaluation method. Since RCNN is also black-box, one will required another explainer to explain the RCNN.

In the worst-case scenario, if RCNN is trained with data such that it considers all relevant words as irrelevant, the evaluation made by RCNN will be incorrect. Hence, " Success depends on the ability of the RCNN to extract correct subsets of tokens."

-------------------  BEFORE
The paper proposed a verification framework to evaluate the performance of different explanatory methods in interpreting a given target model. Specifically, the authors evaluated three explanatory methods namely, LIME, SHAP and L2X for a target model trained to perform sentiment analysis on text data. Authors assume for each input text, there is a subset of tokens that are most relevant and that are completely irrelevant to the final prediction task.  The proposed framework uses a recurrent convolutional neural network (RCNN) to find these subsets. The performance of an explainer is evaluated in terms of overlap between the RCNN most relevant tokens and the most relevant tokens provided by the explainer as an explanation.

Major
•	The paper lack technical novelty.
•	The proposed architecture uses a RCNN to find the most relevant subset of tokens. Firstly, RCNN is also a black box that provides no intuition behind its selection decision. Secondly, in the absence of the ground truth labels for true relevance and irrelevance of a token in input sentence, this explainer method can also suffer from “assuming a reasonable behavior” assumption. The method assumes that the RCNN is performing reasonably in identifying relevant subsets.
•	The success of the method depends on the ability of the RCNN to extract correct subsets of tokens. The data used for training the RCNN, might have some underlying bias. In that case, the evaluation is not accurate.
•	In related work, for “Interpretable target models” the authors mentioned LIME as an example of explainer functions that explains target models that are “very simple models may not be representative for the large and intricate neural networks used in practice”. LIME locally explains the decision of a complex function for a given data point using simpler models like linear regression. But LIME itself can be used for generating explanation for prediction of complex neural network like Inception Net.
•	The example used to explain the difference between feature additive and feature selection-based explainer methods, is confusing. Its not clear how in health diagnostics, one will prefer feature-selection perspective. Although the most relevant features used for the instance are important to understand the decision, but in clinical settings sometimes low rank features can also be useful to understand the target model.
•	For text, the relevant features are the individual tokens of the input sentence. Similarly, for images relevance can be important regions of the image. The authors did not have any experiments on images or tabular data.
•	In the experiment section, the comparison is made with only 3 explainer models and for just one task. The experiments are inadequate.
•	In Figure 4, the colormap is not readable.

The evidence to be evaluated is:Another popular evaluation setup is to create synthetic tasks where the set of important features is controlled. For example, L2X (Chen et al., 2018) was evaluated on four synthetic tasks: 2-dim XOR, orange skin, nonlinear additive model, and switch feature.

In this setup, one identifies certain intuitive heuristics that a high-performing target model is assumed to follow. For example, in sentiment analysis, the model is supposed to rely on adjectives and adverbs in agreement with the predicted sentiment.

In contrast to the above, our evaluation is fully automatic, the target model is a non-trivial neural network trained on a real-world task and for which we provide guarantees on its inner-workings. Our framework is similar in scope with the sanity check introduced by Adebayo et al. (2018).

As mentioned before, current explanatory methods adhere to two major perspectives of explanations: _Perspective 1 (Feature-additivity)_: For a model \(f\) and an instance \(x\), the explanation of the prediction \(f(x)\) consists of a set of contributions \(\{w_{i}^{x}(f)\}_{i}\) for each feature \(i\) of \(x\) such that the sum of the contributions of the features in \(x\) approximates \(f(x)\), i.e., \(\sum_{i}w_{i}^{x}(f)\approx f(x)\). Many explanatory methods adhere to this perspective (Arras et al., 2017; Shrikumar et al., 2017; Ribeiro et al., 2016). For example, LIME (Ribeiro et al., 2016) learns the weights via a linear regression on the neighborhood (explained below) of the instance. Lundberg and Lee (2017) unified this class of methods by showing that the only set of feature-additive contributions that verify three desired constraints (local accuracy, missingness, and consistency--we refer to their paper for details) are given by the Shapley values from game theory: \[w_{i}^{x}(f)=\sum_{x^{\prime}\in x\setminus\{i\}}\frac{|x^{\prime}|![(|x|-|x^{ \prime}|-1)!}{|x|!}[f(x^{\prime}\cup\{i\})-f(x^{\prime})]\,, \tag{1}\] where the sum enumerates over all subsets \(x^{\prime}\) of features in \(x\) that do not include the feature \(i\), and \(|\cdot|\) denotes the number of features of its argument.

_Perspective 2 (Feature-selection)_: For a model \(f\) and an instance \(x\), the explanation of \(f(x)\) consists of a sufficient (ideally small) subset \(S(x)\) of (potentially ranked) features that alone lead to (almost) the same prediction as the original one, i.e., \(f(S(x))\approx f(x)\). Chen et al. (2018); Carter et al. (2018), and Ribeiro et al. (2018) adhere to this perspective. For example, L2X (Chen et al., 2018) learns \(S(x)\) by maximizing the mutual information between \(S(x)\) and the prediction.

However, this can be the case for certain tasks, such as sentiment analysis.

In the rest of the paper, we propose a verification framework for the _feature-selection_ perspective of instance-wise explanations.

We further prune the original dataset on which the RCNN had been trained to ensure that, for each datapoint \(x\), there exists a set of tokens that have zero contribution (irrelevant features) and a set of tokens that have a significant contribution (clearly relevant features) to RCNN's prediction on \(x\). We further introduce a set of metrics that measure how explainers fail to rank the irrelevant tokens lower than the clearly revelant ones.

We evaluate explainers that provide a ranking over the features. We denote by \(r_{1}(x),r_{2}(x),\ldots,r_{n}(x)\) the ranking (in decreasing order of importance) given by an explainer on the \(n=|x|\) features in the instance \(x\). Under our two properties above, we define the following error metrics:* **Percentage of instances for which the most important token provided by the explainer is among the non-selected tokens:** \[\text{\textasci{1.5}{\%}}_{\text{first}}=\tfrac{1}{|\mathcal{D}_{a}|}\sum \limits_{x\in\mathcal{D}_{a}}\mathbb{1}_{\{r_{1}(x)\in\mathcal{N}_{x}\}}\,,\] (4) where \(\mathbb{1}\) is the indicator function.

* **Percentage of instances for which at least one non-selected token is ranked higher than a clearly relevant token:** \[\textasci{1.5}{\%}_{\text{mismrk}}=\tfrac{1}{|\mathcal{D}_{a}|}\sum\limits_{x \in\mathcal{D}_{a}}\mathbb{1}_{\{\exists i<j\text{ such that }r_{i}(x)\in\mathcal{N}_{x}\text{ and }r_{j}(x)\in\mathcal{S} \mathcal{R}_{x}\}}\,.\] (5)

* **Average number of non-selected tokens ranked higher than any clearly relevant token:** \[\textasci{1.5}{avg\_mismrk}=\tfrac{1}{|\mathcal{D}_{a}|}\sum\limits_{x\in \mathcal{D}_{a}}\sum\limits_{i<\text{last},\text{\textasci{1.5}{\%}}} \mathbb{1}_{\{r_{i}(x)\in\mathcal{N}_{x}\}}\,,\] (6) where \(\text{last\_si}=\text{argmax}_{j}\{r_{j}(x)\in\mathcal{S}\mathcal{R}_{x}\}\) is the lowest rank of the clearly relevant tokens. Metric (A) shows the most dramatic failure: the percentage of times when the explainer tells us that the most relevant token is one of the zero contribution ones. Metric (B) shows the percentage of instances for which there is at least an error in the explanation. Finally, metric (C) quantifies the number of zero-contribution features that were ranked higher than any clearly relevant feature.

We test three popular explainers: LIME (Ribeiro et al., 2016), SHAP (Lundberg and Lee, 2017), and L2X (Chen et al., 2018). We used the code of the explainers as provided in the original repositories,5 with their default settings for text explanations, with the exception that, for L2X, we set the dimension of the word embeddings to \(200\) (the same as in the RCNN), and we also allowed training for a maximum of \(30\) epochs instead of \(5\).

As mentioned in Section 3, LIME and SHAP adhere to the feature-additivity perspective, hence our evaluation is not directly targeting these explainers.

Secondly, we introduced an off-the-shelf evaluation test for post-hoc explanatory methods under the feature-selection perspective. To our knowledge, this is the first automatic verification framework offering guarantees on the behaviour of a non-trivial real-world neural network.

While instantiated on a natural language processing task, our methodology is generic and can be adapted to other tasks and other areas.

We also point out that the core algorithm in the majority of the current post-hoc explainers are also domain-agnostic. Therefore, we expect our evaluation to provide a representative view of the fundamental limitations of the explainers.

Input: Figure 3: Partition of the features in our instances.

Input: Figure 4: Explainers’ rankings (with top 5 features on the right-hand side) on an instance from the palate aspect in our evaluation dataset.

Input: Figure 5: Explainers’ rankings (with the top 5 features on the right-hand side) on an instance from the appearance aspect in our evaluation.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, I find this claim somewhat overstated. While the authors do provide a framework for evaluating feature-selection based explanations, there are several limitations:

1. The reviewer points out valid concerns about the RCNN's black-box nature and potential biases
2. The framework currently only demonstrates results on text data, despite claims of being domain-agnostic
3. Experiments are limited to just 3 explainers and one task
4. The framework requires labeled training data, impacting its "off-the-shelf" usability
5. The success depends heavily on RCNN's ability to correctly identify relevant features

While the framework shows promise, characterizing it as a ready "off-the-shelf evaluation" overstates its current capabilities and overlooks important limitations.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We aim for this framework to provide a publicly available,1 off-the-shelf evaluation when the feature-selection perspective on explanations is needed.

The review comment to be evaluated is:

Overview/Contribution:
====================
The authors present a explanation generation framework that help validate post-hoc explanations when the explanations are generated based on feature selection. They claim to demonstrate their method by showing failure modes of exiting explanation generation methods.

Overall, the paper is not ready to be accepted to the conference and I describe my rational with the following strengths and weaknesses.

Strength:
========
+ Explanations make models more transparent and easy to understand for end users of the decision made by complex models such as deep neural networks [1]. In that respect, having a verification mechanism for post-hoc explanations is interesting and useful.
+ The paper is easy to read and follow.
Weakness:
===========
- evaluating explanations generated for an opaque model with another opaque model (RCNN) is cyclical.
- Just like many literature in this nascent space, interpretation (which is measuring the contribution of features or subsets of features towards predicted output) is confused as explanation. Human level explanations don’t necessarily depend on the direct interaction or contribution of model derived features. Rather they describe ‘why’ the model come up with the decision produced.
- Explanation generation is gaining traction in the deep learning community especially for critical applications such as healthcare and security. However, the authors claim that post-hoc explanations currently are only evaluated for only simple non-neural model. That is misleading given the recent attention toward generating explanations for various deep learning models.
- As a generalized pos-hoc explanation generators verification framework, the experiments are seriously lacking and are not well designed to illicit broad applicability.

1) Bekele, E., Lawson, W. E., Horne, Z., & Khemlani, S. (2018). Implementing a Robust Explanatory Bias in a Person Re-identification Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 2165-2172).

The evidence to be evaluated is:Another popular evaluation setup is to create synthetic tasks where the set of important features is controlled. For example, L2X (Chen et al., 2018) was evaluated on four synthetic tasks: 2-dim XOR, orange skin, nonlinear additive model, and switch feature.

In this setup, one identifies certain intuitive heuristics that a high-performing target model is assumed to follow. For example, in sentiment analysis, the model is supposed to rely on adjectives and adverbs in agreement with the predicted sentiment.

In contrast to the above, our evaluation is fully automatic, the target model is a non-trivial neural network trained on a real-world task and for which we provide guarantees on its inner-workings. Our framework is similar in scope with the sanity check introduced by Adebayo et al. (2018).

As mentioned before, current explanatory methods adhere to two major perspectives of explanations: _Perspective 1 (Feature-additivity)_: For a model \(f\) and an instance \(x\), the explanation of the prediction \(f(x)\) consists of a set of contributions \(\{w_{i}^{x}(f)\}_{i}\) for each feature \(i\) of \(x\) such that the sum of the contributions of the features in \(x\) approximates \(f(x)\), i.e., \(\sum_{i}w_{i}^{x}(f)\approx f(x)\). Many explanatory methods adhere to this perspective (Arras et al., 2017; Shrikumar et al., 2017; Ribeiro et al., 2016). For example, LIME (Ribeiro et al., 2016) learns the weights via a linear regression on the neighborhood (explained below) of the instance. Lundberg and Lee (2017) unified this class of methods by showing that the only set of feature-additive contributions that verify three desired constraints (local accuracy, missingness, and consistency--we refer to their paper for details) are given by the Shapley values from game theory: \[w_{i}^{x}(f)=\sum_{x^{\prime}\in x\setminus\{i\}}\frac{|x^{\prime}|![(|x|-|x^{ \prime}|-1)!}{|x|!}[f(x^{\prime}\cup\{i\})-f(x^{\prime})]\,, \tag{1}\] where the sum enumerates over all subsets \(x^{\prime}\) of features in \(x\) that do not include the feature \(i\), and \(|\cdot|\) denotes the number of features of its argument.

_Perspective 2 (Feature-selection)_: For a model \(f\) and an instance \(x\), the explanation of \(f(x)\) consists of a sufficient (ideally small) subset \(S(x)\) of (potentially ranked) features that alone lead to (almost) the same prediction as the original one, i.e., \(f(S(x))\approx f(x)\). Chen et al. (2018); Carter et al. (2018), and Ribeiro et al. (2018) adhere to this perspective. For example, L2X (Chen et al., 2018) learns \(S(x)\) by maximizing the mutual information between \(S(x)\) and the prediction.

However, this can be the case for certain tasks, such as sentiment analysis.

In the rest of the paper, we propose a verification framework for the _feature-selection_ perspective of instance-wise explanations.

We further prune the original dataset on which the RCNN had been trained to ensure that, for each datapoint \(x\), there exists a set of tokens that have zero contribution (irrelevant features) and a set of tokens that have a significant contribution (clearly relevant features) to RCNN's prediction on \(x\). We further introduce a set of metrics that measure how explainers fail to rank the irrelevant tokens lower than the clearly revelant ones.

We evaluate explainers that provide a ranking over the features. We denote by \(r_{1}(x),r_{2}(x),\ldots,r_{n}(x)\) the ranking (in decreasing order of importance) given by an explainer on the \(n=|x|\) features in the instance \(x\). Under our two properties above, we define the following error metrics:* **Percentage of instances for which the most important token provided by the explainer is among the non-selected tokens:** \[\text{\textasci{1.5}{\%}}_{\text{first}}=\tfrac{1}{|\mathcal{D}_{a}|}\sum \limits_{x\in\mathcal{D}_{a}}\mathbb{1}_{\{r_{1}(x)\in\mathcal{N}_{x}\}}\,,\] (4) where \(\mathbb{1}\) is the indicator function.

* **Percentage of instances for which at least one non-selected token is ranked higher than a clearly relevant token:** \[\textasci{1.5}{\%}_{\text{mismrk}}=\tfrac{1}{|\mathcal{D}_{a}|}\sum\limits_{x \in\mathcal{D}_{a}}\mathbb{1}_{\{\exists i<j\text{ such that }r_{i}(x)\in\mathcal{N}_{x}\text{ and }r_{j}(x)\in\mathcal{S} \mathcal{R}_{x}\}}\,.\] (5)

* **Average number of non-selected tokens ranked higher than any clearly relevant token:** \[\textasci{1.5}{avg\_mismrk}=\tfrac{1}{|\mathcal{D}_{a}|}\sum\limits_{x\in \mathcal{D}_{a}}\sum\limits_{i<\text{last},\text{\textasci{1.5}{\%}}} \mathbb{1}_{\{r_{i}(x)\in\mathcal{N}_{x}\}}\,,\] (6) where \(\text{last\_si}=\text{argmax}_{j}\{r_{j}(x)\in\mathcal{S}\mathcal{R}_{x}\}\) is the lowest rank of the clearly relevant tokens. Metric (A) shows the most dramatic failure: the percentage of times when the explainer tells us that the most relevant token is one of the zero contribution ones. Metric (B) shows the percentage of instances for which there is at least an error in the explanation. Finally, metric (C) quantifies the number of zero-contribution features that were ranked higher than any clearly relevant feature.

We test three popular explainers: LIME (Ribeiro et al., 2016), SHAP (Lundberg and Lee, 2017), and L2X (Chen et al., 2018). We used the code of the explainers as provided in the original repositories,5 with their default settings for text explanations, with the exception that, for L2X, we set the dimension of the word embeddings to \(200\) (the same as in the RCNN), and we also allowed training for a maximum of \(30\) epochs instead of \(5\).

As mentioned in Section 3, LIME and SHAP adhere to the feature-additivity perspective, hence our evaluation is not directly targeting these explainers.

Secondly, we introduced an off-the-shelf evaluation test for post-hoc explanatory methods under the feature-selection perspective. To our knowledge, this is the first automatic verification framework offering guarantees on the behaviour of a non-trivial real-world neural network.

While instantiated on a natural language processing task, our methodology is generic and can be adapted to other tasks and other areas.

We also point out that the core algorithm in the majority of the current post-hoc explainers are also domain-agnostic. Therefore, we expect our evaluation to provide a representative view of the fundamental limitations of the explainers.

Input: Figure 3: Partition of the features in our instances.

Input: Figure 4: Explainers’ rankings (with top 5 features on the right-hand side) on an instance from the palate aspect in our evaluation dataset.

Input: Figure 5: Explainers’ rankings (with the top 5 features on the right-hand side) on an instance from the appearance aspect in our evaluation.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim of providing a "publicly available, off-the-shelf evaluation" for feature-selection explanations appears somewhat overstated. While the paper does present an evaluation framework, several key limitations emerge:

1. The framework is primarily demonstrated on a specific NLP task, despite claims of broad applicability
2. The reviewer points out that experiments are "seriously lacking" to demonstrate broad applicability
3. The evaluation relies on an RCNN model, which the reviewer notes creates a circular dependency of "evaluating explanations generated for an opaque model with another opaque model"

While the framework shows promise and is indeed made publicly available, the claim would be better stated with more qualifications about its current scope and limitations.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

At every level of sparsity, for both CIFAR-10 and ImageNet we identify a subset of PIE images. At \(10\%\) and \(90\%\) sparsity, we classify \(3.34\%\) and \(10.27\%\) of all ImageNet test-set images respectively as PIEs. For CIFAR-10, PIEs constitute \(0.97\%\) and \(2.16\%\) of the test set at \(10\%\) and \(90\%\) sparsity.

The evidence to be evaluated is:While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim against the provided evidence:

Summary: The claim states specific percentages of PIE images identified at different sparsity levels for both ImageNet and CIFAR-10. While the evidence (including Figure 10) clearly shows the existence of PIE images across sparsity levels and demonstrates their impact on model performance, it does not explicitly provide or validate the exact percentages mentioned in the claim (3.34%, 10.27%, 0.97%, and 2.16%). The evidence discusses PIE trends qualitatively but lacks the quantitative support for these precise figures.

<score>3</score>

Justification:
1. The claim makes very specific numerical assertions about PIE percentages that aren't directly supported in the provided evidence
2. While Figure 10 shows PIE exists at different sparsity levels and their impact on accuracy, it doesn't validate the exact percentages claimed
3. The qualitative findings about PIE's existence and impact across sparsity levels are supported by the evidence
4. The general trend of increasing PIE prevalence with higher sparsity aligns with the evidence, but the precise magnitude isn't verified
5. The claim would be better stated with less precise numbers or with explicit references to supporting data for these specific percentages


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

At every level of sparsity, for both CIFAR-10 and ImageNet we identify a subset of PIE images. At \(10\%\) and \(90\%\) sparsity, we classify \(3.34\%\) and \(10.27\%\) of all ImageNet test-set images respectively as PIEs. For CIFAR-10, PIEs constitute \(0.97\%\) and \(2.16\%\) of the test set at \(10\%\) and \(90\%\) sparsity.

The review comment to be evaluated is:

The paper claims that neural network pruning methods have different impact on accuracy in class-wise and sample-wise. To this end, the paper performs statistical tests to identify such classes (and samples) from a population of neural network models of different levels of pruning. In particular, the identified samples, called PIEs, are shown to have significantly lower test accuracy, i.e., the PIEs are much harder to classify, and a user study is performed to support this claim. The paper also demonstrates that pruned models tend to have lower accuracy against adversarial attacks or common corruptions.

In overall, the paper addresses an important problem of investigating the effects of pruning. The experiments performed here seems fairly extensive. One of my primary concerns, however, is that I am still not convinced whether the empirical findings presented here is indeed significant, as some reader might feel those results are not that surprising. I personally feel that it is much more likely that pruning gives heterogeneous effects over the classes, since current pruning schemes do not enforce the "uniform" brain damage, i.e., they simply aim to minimize total accuracy drops. Also, the accuracy differences in Table 1 do not appear to be that large in my opinion. The questions in what follows may help to resolve such concerns:

- Section 2.1: For ImageNet dataset, data imbalance in training set might be the reason of such disparate impact? As CIFAR-10 dataset is balanced, I wonder if CIFAR-10 results could be also presented in more details.
- What if the whole training and pruning is re-performed after excluding the classes which the accuracy is decreased more by pruning? Would it lead to better pruning results, or better overall accuracy?
- I feel there should be more explanation about the actual statistical implication of the use of Welch's t-test: Is it ok that S^c_t may not be independent to S^c_0? What is the key benefits of not just picking outlier classes among S^c_t with respect to c?
- What would happens if we exclude the PIEs in training set, and performs training & pruning from scratch? The overall claims may be strenghten if this could improve the pruning efficiency.
- Figure 7: I think the pure ImageNet top-1 accuracy should be also presented in each plot as a baseline for fair comparison, as this accuracy will be decreased with respect to the sparsity as well.

The evidence to be evaluated is:While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim states specific percentages of PIE images identified at 10% and 90% sparsity levels for both ImageNet and CIFAR-10 datasets. The evidence, including Figure 10, demonstrates the existence of PIEs at different sparsity levels and their impact on model performance. The numbers cited in the claim appear to be precisely stated and are supported by the experimental results shown in the graphs. While the reviewer raises some questions about the broader significance of the findings, their comments don't directly challenge the accuracy of these specific percentage claims.

The claim sticks to reporting the observed percentages without making broader generalizations or unsupported assertions about their implications. It is well-stated and accurately reflects the empirical findings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

At every level of sparsity, for both CIFAR-10 and ImageNet we identify a subset of PIE images. At \(10\%\) and \(90\%\) sparsity, we classify \(3.34\%\) and \(10.27\%\) of all ImageNet test-set images respectively as PIEs. For CIFAR-10, PIEs constitute \(0.97\%\) and \(2.16\%\) of the test set at \(10\%\) and \(90\%\) sparsity.

The review comment to be evaluated is:

This paper study how the pruning impacts the images/classes and observes which images/classes do not generalize in the pruned networks under ResNet-50 with ImageNet dataset. The authors first observe whether there exist classes exhibiting higher/lower accuracy in pruned networks compared to unpruned ones. Then, the authors define images (PIE) that generate inconsistent predictions for unpruned/pruned networks and observe the properties of PIE (e.g., whether corrupted, contains multiple labels, etc.). In particular, PIE images could be considered as `hard examples’ since even the unpruned networks misclassify them w.h.p. The authors also verify the properties of PIE and conclude the paper.

Overall, I think the paper contains some interesting observations but not enough for publication yet, due to the following reasons. First, it is well known that hard (noisy or corrupted) images are harder to be correctly classified in networks of smaller model power (Hendrycks & Dietterich, 2019). That is, results in Figure 4 and Figure 5 are not very surprising. Moreover, there is no further discussion on the hypothesis test after presenting Table 1 and Figure 3.

The paper is generally well written even it contains few typos and inconsistent presentations (e.g., legends in images are inconsistent. Both underbar and space are mixed.)

The evidence to be evaluated is:While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim makes specific statistical assertions about PIE percentages at different sparsity levels for both ImageNet and CIFAR-10 datasets. While Figure 10 shows results related to PIE images and their impact on accuracy, it doesn't directly validate the exact percentages claimed (3.34%, 10.27%, 0.97%, and 2.16%). The paper demonstrates that PIEs exist and their impact increases with sparsity, but without explicit evidence showing these precise percentages, I cannot fully verify the specific numbers stated in the claim.

The reviewer's comments don't specifically address these percentage claims, and the textual evidence doesn't provide direct support for these exact figures. While the general observation about PIEs existing at different sparsity levels appears supported, the precise quantification lacks clear substantiation in the provided evidence.

<score>3</score>

Justification:
1. The claim makes very specific numerical assertions without clear supporting evidence in the provided materials
2. The general principle (existence of PIEs at different sparsity levels) is supported by Figure 10 and text
3. The evidence demonstrates the concept but doesn't validate the precise percentages
4. While not necessarily incorrect, the claim's specificity exceeds what can be verified from the given evidence


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

At every level of sparsity, for both CIFAR-10 and ImageNet we identify a subset of PIE images. At \(10\%\) and \(90\%\) sparsity, we classify \(3.34\%\) and \(10.27\%\) of all ImageNet test-set images respectively as PIEs. For CIFAR-10, PIEs constitute \(0.97\%\) and \(2.16\%\) of the test set at \(10\%\) and \(90\%\) sparsity.

The review comment to be evaluated is:

This paper presents an empirical study on the effect of pruning to the model performance on each class and example, which leads to a novel finding that it has disparate effects to each sample. Specifically, the authors have found out that examples that are affected the most by pruning are more difficult to classify even for the non-pruned network, due to low image quality, mislabeling, or being atypical from the class prototype, and performed a further human study to analyze the source of difficulty. Moreover, the authors performed an additional experiment, which shows that the sparse models are brittle against natural adversarial examples.

Pros
- The paper provides a novel insight on the effect of pruning at the class and the example level, which could lead to a more effective pruning approach that exploit this findings.

Cons

- The paper only provides a novel finding but not the solution on how to tackle this problem, and thus the paper looks incomplete. After section 3.3, I was expecting to see some approaches to tackle this problem but the paper abruptly ended.

- The effect of pruning could largely differ from one method to another, but the authors do not experimentally compare the effects of different pruning methods. Also, it is highly likely that the findings discussed in the paper may be only true for input-independent pruning approaches, and may not generalize to input-dependent pruning method. The authors need to perform extensive study of both input-dependent and input-independent pruning approaches to validate their points.

In sum, the paper provides a novel insight on how pruning affects the performance at the example level, but does not provide a solution, and the current set of experiments is insufficient to validate that the empirical findings that the authors report generalize to other types of pruning approaches, such as input-dependent pruning. Thus I believe that the paper is not ready for publication yet, and vote for rejecting this paper in its current form.

The evidence to be evaluated is:While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim states specific percentages of PIE images identified at different sparsity levels for both ImageNet and CIFAR-10. The evidence, including Figure 10, demonstrates the existence of PIEs across sparsity levels and their impact on model performance. However, while the graphs show the relative performance on PIE vs non-PIE images, they don't directly validate the exact percentages claimed (3.34%, 10.27%, 0.97%, and 2.16%). Without explicit data supporting these precise numbers, the claim's specificity may be overstated, even though the general phenomenon it describes is supported by the evidence.

<score>2</score>

Justification:
1. The evidence strongly supports the general existence of PIEs at different sparsity levels
2. The graphs demonstrate clear performance differences between PIE and non-PIE images
3. The review comment acknowledges the validity of the findings about pruning's disparate effects
4. While specific percentages are cited, the evidence shown doesn't directly verify these exact numbers
5. The claim remains relatively conservative in its scope and aligns with the paper's overall findings


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs.

The evidence to be evaluated is:We also consider how pruning impacts robustness to natural adversarial examples and image corruptions. We note that recent work by (Hendrycks and Dietterich, 2019; Hendrycks et al., 2019) considers complimentary variant of this question by benchmark ImageNet-A and ImageNet-B robustness across a limited set of dense non-sparse architectures with different numbers of parameters (for example ResNet-50 vs ResNet-101). While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others.

We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard. Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the claim appears to be well-stated based on multiple pieces of supporting evidence:

1. Figure 7 clearly shows that both ImageNet-A and ImageNet-C accuracies decline as sparsity increases, demonstrating similar degradation patterns.

2. The PIE analysis in Figure 10 shows that sparse models are consistently more sensitive to challenging examples (PIEs), which aligns with their degraded performance on ImageNet-A and ImageNet-C.

3. The evidence directly demonstrates parallel patterns of degradation across these different robustness measures as sparsity increases, supporting the claim's assertion about mirrored sensitivity patterns.

The claim avoids overstatement by simply noting the parallel patterns without making stronger causal claims or overgeneralizing the findings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs.

The review comment to be evaluated is:

The paper claims that neural network pruning methods have different impact on accuracy in class-wise and sample-wise. To this end, the paper performs statistical tests to identify such classes (and samples) from a population of neural network models of different levels of pruning. In particular, the identified samples, called PIEs, are shown to have significantly lower test accuracy, i.e., the PIEs are much harder to classify, and a user study is performed to support this claim. The paper also demonstrates that pruned models tend to have lower accuracy against adversarial attacks or common corruptions.

In overall, the paper addresses an important problem of investigating the effects of pruning. The experiments performed here seems fairly extensive. One of my primary concerns, however, is that I am still not convinced whether the empirical findings presented here is indeed significant, as some reader might feel those results are not that surprising. I personally feel that it is much more likely that pruning gives heterogeneous effects over the classes, since current pruning schemes do not enforce the "uniform" brain damage, i.e., they simply aim to minimize total accuracy drops. Also, the accuracy differences in Table 1 do not appear to be that large in my opinion. The questions in what follows may help to resolve such concerns:

- Section 2.1: For ImageNet dataset, data imbalance in training set might be the reason of such disparate impact? As CIFAR-10 dataset is balanced, I wonder if CIFAR-10 results could be also presented in more details.
- What if the whole training and pruning is re-performed after excluding the classes which the accuracy is decreased more by pruning? Would it lead to better pruning results, or better overall accuracy?
- I feel there should be more explanation about the actual statistical implication of the use of Welch's t-test: Is it ok that S^c_t may not be independent to S^c_0? What is the key benefits of not just picking outlier classes among S^c_t with respect to c?
- What would happens if we exclude the PIEs in training set, and performs training & pruning from scratch? The overall claims may be strenghten if this could improve the pruning efficiency.
- Figure 7: I think the pure ImageNet top-1 accuracy should be also presented in each plot as a baseline for fair comparison, as this accuracy will be decreased with respect to the sparsity as well.

The evidence to be evaluated is:We also consider how pruning impacts robustness to natural adversarial examples and image corruptions. We note that recent work by (Hendrycks and Dietterich, 2019; Hendrycks et al., 2019) considers complimentary variant of this question by benchmark ImageNet-A and ImageNet-B robustness across a limited set of dense non-sparse architectures with different numbers of parameters (for example ResNet-50 vs ResNet-101). While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others.

We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard. Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs" appears to be well-supported but slightly overstated. While Figure 7 shows declining performance on both ImageNet-A and ImageNet-C with increased sparsity, and Figure 10 demonstrates similar degradation patterns for PIEs, the term "mirrors" implies a stronger correlation than what's definitively shown in the data. The evidence shows similar directional trends but doesn't establish precise mirroring of patterns across these different types of challenging cases.

<score>2</score>

Justification:
1. The evidence supports that sparse models show degraded performance across ImageNet-A, ImageNet-C, and PIEs as sparsity increases (Fig 7 and Fig 10)
2. However:
- The term "mirrors" suggests a stronger relationship than demonstrated
- The degradation patterns, while similar in direction, show different magnitudes and curves
- ImageNet-A shows more dramatic drops (down to ~0.5% top-1 accuracy) compared to ImageNet-C (~25-40% accuracy)
- The reviewer notes concerns about the significance of the findings and suggests additional experiments/analysis
3. While potentially slightly overstated, the claim is largely grounded in the presented evidence, hence a score of 2 rather than a higher score indicating more severe overstatement.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs.

The review comment to be evaluated is:

This paper study how the pruning impacts the images/classes and observes which images/classes do not generalize in the pruned networks under ResNet-50 with ImageNet dataset. The authors first observe whether there exist classes exhibiting higher/lower accuracy in pruned networks compared to unpruned ones. Then, the authors define images (PIE) that generate inconsistent predictions for unpruned/pruned networks and observe the properties of PIE (e.g., whether corrupted, contains multiple labels, etc.). In particular, PIE images could be considered as `hard examples’ since even the unpruned networks misclassify them w.h.p. The authors also verify the properties of PIE and conclude the paper.

Overall, I think the paper contains some interesting observations but not enough for publication yet, due to the following reasons. First, it is well known that hard (noisy or corrupted) images are harder to be correctly classified in networks of smaller model power (Hendrycks & Dietterich, 2019). That is, results in Figure 4 and Figure 5 are not very surprising. Moreover, there is no further discussion on the hypothesis test after presenting Table 1 and Figure 3.

The paper is generally well written even it contains few typos and inconsistent presentations (e.g., legends in images are inconsistent. Both underbar and space are mixed.)

The evidence to be evaluated is:We also consider how pruning impacts robustness to natural adversarial examples and image corruptions. We note that recent work by (Hendrycks and Dietterich, 2019; Hendrycks et al., 2019) considers complimentary variant of this question by benchmark ImageNet-A and ImageNet-B robustness across a limited set of dense non-sparse architectures with different numbers of parameters (for example ResNet-50 vs ResNet-101). While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others.

We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard. Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs."

1. Summary of evaluation:
The evidence shows similar degradation patterns across ImageNet-A, ImageNet-C, and PIEs as sparsity increases, but with some important nuances:
- Figure 7 demonstrates declining performance on both ImageNet-A and ImageNet-C with increased sparsity
- Figure 10 shows similar degradation patterns for PIE images
- However, the magnitudes and rates of degradation differ between the datasets
- The review points out that degradation on hard examples with reduced model capacity is expected

The claim is slightly overstated as it implies a stronger correlation between these patterns than the evidence fully supports, though the general trend is accurate.

<score>2</score>

Justification:
- The evidence does show parallel degradation patterns across all three types of challenging images as sparsity increases
- Figures 7 and 10 demonstrate this trend visually
- However, the word "mirrors" suggests a stronger equivalence than shown - the degradation rates and magnitudes vary
- The reviewer correctly notes that this behavior is somewhat expected for models with reduced capacity
- The claim would be better stated by noting "similar patterns of degradation" rather than implying exact mirroring
- The evidence supports the general observation but the strength of the correlation is overstated


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs.

The review comment to be evaluated is:

This paper presents an empirical study on the effect of pruning to the model performance on each class and example, which leads to a novel finding that it has disparate effects to each sample. Specifically, the authors have found out that examples that are affected the most by pruning are more difficult to classify even for the non-pruned network, due to low image quality, mislabeling, or being atypical from the class prototype, and performed a further human study to analyze the source of difficulty. Moreover, the authors performed an additional experiment, which shows that the sparse models are brittle against natural adversarial examples.

Pros
- The paper provides a novel insight on the effect of pruning at the class and the example level, which could lead to a more effective pruning approach that exploit this findings.

Cons

- The paper only provides a novel finding but not the solution on how to tackle this problem, and thus the paper looks incomplete. After section 3.3, I was expecting to see some approaches to tackle this problem but the paper abruptly ended.

- The effect of pruning could largely differ from one method to another, but the authors do not experimentally compare the effects of different pruning methods. Also, it is highly likely that the findings discussed in the paper may be only true for input-independent pruning approaches, and may not generalize to input-dependent pruning method. The authors need to perform extensive study of both input-dependent and input-independent pruning approaches to validate their points.

In sum, the paper provides a novel insight on how pruning affects the performance at the example level, but does not provide a solution, and the current set of experiments is insufficient to validate that the empirical findings that the authors report generalize to other types of pruning approaches, such as input-dependent pruning. Thus I believe that the paper is not ready for publication yet, and vote for rejecting this paper in its current form.

The evidence to be evaluated is:We also consider how pruning impacts robustness to natural adversarial examples and image corruptions. We note that recent work by (Hendrycks and Dietterich, 2019; Hendrycks et al., 2019) considers complimentary variant of this question by benchmark ImageNet-A and ImageNet-B robustness across a limited set of dense non-sparse architectures with different numbers of parameters (for example ResNet-50 vs ResNet-101). While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models.

We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others.

We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for \(PIE_{10}\) images which we suspect are the most difficult and performance is still poor but better for \(PIE_{t}\) for larger values of \(t\). In Figure 10 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard. Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets of PIEs."

After reviewing the evidence:

The evidence shows similar degradation patterns across ImageNet-A, ImageNet-C, and PIEs as sparsity increases:
- Figure 7 demonstrates declining performance on both ImageNet-A and ImageNet-C with increased sparsity
- Figure 10 shows PIE examples are consistently more challenging for sparse models
- The text explicitly states sparse models are "less robust to natural adversarial examples" and "more brittle to common image corruptions"

However, while the patterns are similar, the claim could be more precise about the degree of similarity and acknowledge potential variations in the degradation patterns. The evidence shows directional similarity but doesn't prove exact mirroring. A more qualified statement would be more accurate.

<score>2</score>

Justification:
The claim is largely supported by the evidence but slightly overstates the relationship by using "mirrors" without qualification. While the degradation patterns are similar, the magnitude and rate of degradation vary across the different test sets. The evidence shows general alignment in how sparse models become more sensitive to these challenging cases, but doesn't demonstrate exact mirroring of patterns.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples.

The evidence to be evaluated is:We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance. Pruning identified exemplars are a subset of exemplars where there is a high level of disagreement between sparse and non-sparse models. We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 2: Visualization of pruning identified exemplars (PIE30) for the CIFAR-10 dataset. This subset of impacted images is identified by considering a set of 30 non-sparse wide ResNet models and 30 models trained to 30% sparsity.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input: Table 1: Summary of class-level results for ImageNet. Only classes passing the significance test are included. The model accuracy difference column reports mean βMt − βM0 as the percentage point difference between the pruned and baseline model accuracies; a negative value means the pruned model’s average accuracy is lower than the baseline model’s.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim that "as sparsity increases, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples."

Based on the evidence:
- Figure 7 clearly shows declining performance on both ImageNet-A (natural adversarial examples) and ImageNet-C (corruptions) as sparsity increases
- For ImageNet-A, both top-1 and top-5 accuracy show consistent degradation with increased sparsity
- The decline is particularly steep at high sparsity levels (0.7-0.9)
- Table 1 confirms increasingly negative model accuracy differences at higher sparsity levels
- The claim carefully uses "suggesting" rather than making absolute statements

The claim appears to be well-stated (score of 1) as it:
1. Accurately reflects the demonstrated relationship between sparsity and accuracy
2. Uses appropriately cautious language ("suggesting")
3. Is directly supported by multiple pieces of quantitative evidence

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples.

The review comment to be evaluated is:

The paper claims that neural network pruning methods have different impact on accuracy in class-wise and sample-wise. To this end, the paper performs statistical tests to identify such classes (and samples) from a population of neural network models of different levels of pruning. In particular, the identified samples, called PIEs, are shown to have significantly lower test accuracy, i.e., the PIEs are much harder to classify, and a user study is performed to support this claim. The paper also demonstrates that pruned models tend to have lower accuracy against adversarial attacks or common corruptions.

In overall, the paper addresses an important problem of investigating the effects of pruning. The experiments performed here seems fairly extensive. One of my primary concerns, however, is that I am still not convinced whether the empirical findings presented here is indeed significant, as some reader might feel those results are not that surprising. I personally feel that it is much more likely that pruning gives heterogeneous effects over the classes, since current pruning schemes do not enforce the "uniform" brain damage, i.e., they simply aim to minimize total accuracy drops. Also, the accuracy differences in Table 1 do not appear to be that large in my opinion. The questions in what follows may help to resolve such concerns:

- Section 2.1: For ImageNet dataset, data imbalance in training set might be the reason of such disparate impact? As CIFAR-10 dataset is balanced, I wonder if CIFAR-10 results could be also presented in more details.
- What if the whole training and pruning is re-performed after excluding the classes which the accuracy is decreased more by pruning? Would it lead to better pruning results, or better overall accuracy?
- I feel there should be more explanation about the actual statistical implication of the use of Welch's t-test: Is it ok that S^c_t may not be independent to S^c_0? What is the key benefits of not just picking outlier classes among S^c_t with respect to c?
- What would happens if we exclude the PIEs in training set, and performs training & pruning from scratch? The overall claims may be strenghten if this could improve the pruning efficiency.
- Figure 7: I think the pure ImageNet top-1 accuracy should be also presented in each plot as a baseline for fair comparison, as this accuracy will be decreased with respect to the sparsity as well.

The evidence to be evaluated is:We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance. Pruning identified exemplars are a subset of exemplars where there is a high level of disagreement between sparse and non-sparse models. We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 2: Visualization of pruning identified exemplars (PIE30) for the CIFAR-10 dataset. This subset of impacted images is identified by considering a set of 30 non-sparse wide ResNet models and 30 models trained to 30% sparsity.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input: Table 1: Summary of class-level results for ImageNet. Only classes passing the significance test are included. The model accuracy difference column reports mean βMt − βM0 as the percentage point difference between the pruned and baseline model accuracies; a negative value means the pruned model’s average accuracy is lower than the baseline model’s.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim that "as sparsity increases, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples" is moderately overstated based on the evidence.

While Figure 7 does show declining accuracy for both ImageNet-A and ImageNet-C as sparsity increases, suggesting increased brittleness, the evidence primarily demonstrates correlation rather than causation. The graphs show gradual degradation rather than dramatic "erosion." Additionally, the reviewer points out that baseline ImageNet accuracy should be included for fair comparison since accuracy decreases with sparsity regardless of adversarial examples.

The claim could be better stated by acknowledging that sparse models show reduced performance on adversarial examples, while avoiding implying direct causation.

<score>3</score>

Justification:
1. Figure 7 supports general trend but "erode" overstates the gradual decline shown
2. Reviewer correctly notes missing baseline comparison
3. Evidence shows correlation but claim implies causation
4. Term "brittle" is somewhat dramatic given the moderate slopes in accuracy decline
5. No statistical analysis provided to support brittleness claim specifically


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples.

The review comment to be evaluated is:

This paper study how the pruning impacts the images/classes and observes which images/classes do not generalize in the pruned networks under ResNet-50 with ImageNet dataset. The authors first observe whether there exist classes exhibiting higher/lower accuracy in pruned networks compared to unpruned ones. Then, the authors define images (PIE) that generate inconsistent predictions for unpruned/pruned networks and observe the properties of PIE (e.g., whether corrupted, contains multiple labels, etc.). In particular, PIE images could be considered as `hard examples’ since even the unpruned networks misclassify them w.h.p. The authors also verify the properties of PIE and conclude the paper.

Overall, I think the paper contains some interesting observations but not enough for publication yet, due to the following reasons. First, it is well known that hard (noisy or corrupted) images are harder to be correctly classified in networks of smaller model power (Hendrycks & Dietterich, 2019). That is, results in Figure 4 and Figure 5 are not very surprising. Moreover, there is no further discussion on the hypothesis test after presenting Table 1 and Figure 3.

The paper is generally well written even it contains few typos and inconsistent presentations (e.g., legends in images are inconsistent. Both underbar and space are mixed.)

The evidence to be evaluated is:We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance. Pruning identified exemplars are a subset of exemplars where there is a high level of disagreement between sparse and non-sparse models. We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 2: Visualization of pruning identified exemplars (PIE30) for the CIFAR-10 dataset. This subset of impacted images is identified by considering a set of 30 non-sparse wide ResNet models and 30 models trained to 30% sparsity.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input: Table 1: Summary of class-level results for ImageNet. Only classes passing the significance test are included. The model accuracy difference column reports mean βMt − βM0 as the percentage point difference between the pruned and baseline model accuracies; a negative value means the pruned model’s average accuracy is lower than the baseline model’s.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me analyze the claim and evidence:

1. The claim "As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples" is partially supported by the evidence, but requires some nuancing.

Figure 7 shows clear degradation in performance on both ImageNet-A (natural adversarial examples) and ImageNet-C (corrupted images) as sparsity increases. However, the erosion pattern varies:
- ImageNet-A shows gradual decline in both top-1 and top-5 accuracy
- ImageNet-C shows relatively stable performance until high sparsity levels (0.7-0.9), then sharp drops
- The impact differs across corruption types

While the general trend supports decreasing accuracy with increased sparsity, characterizing this simply as "more brittle" oversimplifies the nuanced patterns shown in the data. The claim would be more accurate if it acknowledged these variations and specified the conditions under which brittleness increases significantly.

<score>3</score>