Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we study a number of different approaches that aim at recovering some of this layer distributional context by computing a connection importance value that is dependent on the weights of the other connections in the same layer.

The review comment to be evaluated is:

Strength:
* The LTH problem considered is popular and the focus on an intuitive/simple issue with typical approaches to LTH (i.e., global pruning leads and layer collapse). I believe this is an interesting direction that the authors motivate well.
* I believe the observation that many different structures of lottery tickets can be discovered within a network has already been made (see e.g., https://arxiv.org/abs/1912.05671), but the authors do provide some extra analysis that goes beyond previous work that is interesting/useful. I really enjoy this extra analysis, and I think the authors could emphasize this more!

Weakness:
* Because the paper is purely empirical, I believe that large-scale experiments on ImageNet are very necessary in this case. This is because, for the LTH, it is widely recognized that many observations made on LTH for smaller-scale experiments do not hold at scale (see https://arxiv.org/abs/1810.05270 and https://arxiv.org/abs/1902.09574). Thus, in this work, performing the larger-scale experiments (e.g., ImageNet) is necessary.
* Accuracy improvements of the proposed procedure are small and there is not a single procedure that performs best. The proposed methods are outperformed by magnitude pruning (i.e., the normal LTH procedure) in many of the considered cases.
* Several pruning works consider pruning in a layer-normalized fashion (see https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html and references, just search “layer wise vs. global pruning” in the paper). None of these techniques are compared against within the analysis. I believe comparing against a few of the SOTA layer-based pruning techniques would be useful.

Small Stuff:
* Figure 1 is difficult to parse. It might be better to use a different subplot for each layer, then plot the sparsity as a function of the pruning iteration.
* L1/L2 normalization of per-layer importance are basically identical (even in the results). It might be better to just use one or the other and state that they perform very similarly.
* In the introduction, your posing of LTH makes it seem like it solves the problem of the train-prune-finetune loop. It might be good to state that LTH still requires this loop (i.e., the network needs to be pre-trained to discover the winning ticket still).
* “Using importance metrics makes this bias both more explicit and stronger, dependent on which metric is used.”: I am not quite sure what this means here or why this would be the case. It might be nice to explain a bit more or maybe provide a citation.
* “The only computational overhead incurred in our method is the calculation of importance scores, which is negligible.”: Whether this computational cost is negligible depends on the importance score you choose/implement, right?

Questions:
* Is there any motivation for the importance metrics that you consider, or are you simply trying a bunch of different approaches and trying to determine the best strategy empirically? Are there other approaches beyond these defined importances that could be useful?

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure.

Due to the definition of Min-Max normalization, at least one connection in each layer will have an importance of 0 and at least one connection in each layer has an importance of 1.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized. This suggests that we should focus our attention on these specific connections to find LTs more efficiently.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate how well the claim is supported by the evidence:

Summary: The claim states that the paper studies approaches for recovering layer distributional context through connection importance values. The evidence strongly supports this, showing detailed analysis of multiple importance metrics (L1, L2, softmax, min-max normalization) that incorporate layer-level information. The figures demonstrate how these approaches affect layer-wise pruning patterns differently. While the claim is accurate about studying these approaches, it could be more precise in noting that the improvements were modest according to the accuracy curves in Figure 4 and reviewer comments. However, the claim itself is carefully worded as it simply states they "study" these approaches rather than claiming significant improvements.

<score>1</score>

The claim is well-stated (score of 1) because it:
- Accurately describes what was studied without overstating results
- Is supported by extensive empirical evidence showing the layer-wise analyses
- Uses appropriately cautious language ("aim at recovering")
- Aligns with both the methods and results presented in figures/text
- Makes no exaggerated claims about performance improvements


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we study a number of different approaches that aim at recovering some of this layer distributional context by computing a connection importance value that is dependent on the weights of the other connections in the same layer.

The review comment to be evaluated is:

### Strength:

1. This paper is well-written and easy to follow.
2. Both quantitative and qualitative results are reported.
3. Experiments are conducted across different datasets and architectures.

### Weakness:

1. Only small-scale datasets are considered in the experiments.
2. I'm a little bit confused about the main contribution of this work. Using a layerwise importance score can improve the performance of identified lottery tickets or some observations that suggest lottery tickets are not unique. With different mini-batch sequences during training, we may also obtain significantly different lottery tickets from the same initialization.
3. Using layerwise importance score, can the LTH procedure identifies the winning ticket with higher sparsity?

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure.

Due to the definition of Min-Max normalization, at least one connection in each layer will have an importance of 0 and at least one connection in each layer has an importance of 1.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized. This suggests that we should focus our attention on these specific connections to find LTs more efficiently.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim about studying different approaches to recover layer distributional context through connection importance values.

The evidence strongly supports this claim through:
1. Detailed mathematical formulations of multiple importance measures (L1, L2, Softmax, Min-Max)
2. Comprehensive experimental results showing how different importance measures affect layer sparsity (Figures 1, 8, 9, 10)
3. Analysis of how these measures impact the weight distributions and pruning behaviors
4. Empirical validation across multiple architectures and datasets (Figure 4)

The claim is carefully worded and accurately reflects what was studied without overstatement. The authors acknowledge limitations and present balanced findings about the effectiveness of different approaches.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then generalise the LTH to consider weight importance values rather than weight magnitudes.

The evidence to be evaluated is:5:\(\theta_{pooled}\gets Pool(abs(\theta_{i,m}))\)

6:\(p\leftarrow\) j-th percentile of \(\theta_{pooled}\)

A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer.

Intuitively, this should alleviate the issue with layer-collapse and should improve the performance of the LTR procedure. Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\) 6:\(score_{pooled}\gets Pool(score_{i})\) 7:\(p\leftarrow\) j-th percentile of \(score_{pooled}\) 8: Prune all connections with \(abs(score_{i})<p\)

Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

First, the magnitude, \(L_{1}\) and \(L_{2}\) criterions all produce LTs of similar performance, the only noticeable difference is in the VGG16 + CIFAR10 experiment.

We notice that the top-1 accuracy of both magnitude, and the \(L_{1}\) and \(L_{2}\) normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition. Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure. We elaborate further on this more in the rest of the paper, where we conduct a deeper study on some differences between the found lottery tickets.

We consider the LTs found by the following importance measures: _magnitude_, \(L_{1}\) and \(L_{2}\).

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure. While we did not note a significant difference in performance when using certain importance measures, we emphasize that the LTH is invariant in a certain sense to the layerwise pruning ratios, as long as the lowest magnitude weights are removed from each layer. Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Summary: The claim about generalizing LTH to consider weight importance values is moderately overstated based on the evidence. While the authors did implement various importance measures (L1, L2, softmax, min-max normalization), their results show that magnitude-based, L1, and L2 approaches produced very similar performance in most cases, contradicting their initial intuition. The authors themselves acknowledge this by stating "we did not note a significant difference in performance when using certain importance measures." The evidence suggests the modification did not substantially improve or generalize LTH as claimed.

<score>4</score>

Justification:
1. The performance results in Figure 4 show nearly identical accuracy curves for magnitude, L1 and L2 measures across different models and datasets, contradicting the claimed benefits of generalization.

2. Authors directly state: "We notice that the top-1 accuracy of both magnitude, and the L1 and L2 normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition."

3. The layer sparsity evolution figures (1, 8-10) demonstrate that different importance measures led to similar pruning patterns across layers.

4. While the approach represents a methodological extension, calling it a "generalization" overstates its practical impact given the equivalent performance to simpler magnitude-based pruning.

5. The authors' own conclusions acknowledge limitations: "we emphasize that the LTH is invariant in a certain sense to the layerwise pruning ratios, as long as the lowest magnitude weights are removed from each layer."


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then generalise the LTH to consider weight importance values rather than weight magnitudes.

The review comment to be evaluated is:

The authors propose to modify the LTH procedure by using a score that measures the layerwise importance in order to select the weights to be pruned. The authors propose a few possible scores and evaluate their performance on a set of experiments. The experiments show that in most of the cases the proposed method does not improve with respect to the standard LTH that uses magnitude in order to select the weights to be pruned. The authors also perform a comparison of lottery tickets obtained with the different scores showing that hey have a noticeable amount of overlapping connections and such connections seems to be more stable.

The evidence to be evaluated is:5:\(\theta_{pooled}\gets Pool(abs(\theta_{i,m}))\)

6:\(p\leftarrow\) j-th percentile of \(\theta_{pooled}\)

A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer.

Intuitively, this should alleviate the issue with layer-collapse and should improve the performance of the LTR procedure. Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\) 6:\(score_{pooled}\gets Pool(score_{i})\) 7:\(p\leftarrow\) j-th percentile of \(score_{pooled}\) 8: Prune all connections with \(abs(score_{i})<p\)

Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

First, the magnitude, \(L_{1}\) and \(L_{2}\) criterions all produce LTs of similar performance, the only noticeable difference is in the VGG16 + CIFAR10 experiment.

We notice that the top-1 accuracy of both magnitude, and the \(L_{1}\) and \(L_{2}\) normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition. Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure. We elaborate further on this more in the rest of the paper, where we conduct a deeper study on some differences between the found lottery tickets.

We consider the LTs found by the following importance measures: _magnitude_, \(L_{1}\) and \(L_{2}\).

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure. While we did not note a significant difference in performance when using certain importance measures, we emphasize that the LTH is invariant in a certain sense to the layerwise pruning ratios, as long as the lowest magnitude weights are removed from each layer. Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, this claim is moderately overstated. While the authors do introduce layerwise importance measures as a modification to LTH, the experimental results show that this generalization does not lead to significant performance improvements compared to using weight magnitudes alone. The evidence demonstrates that magnitude, L1, and L2 criteria produce very similar lottery tickets and accuracies across multiple experiments. The reviewer's comment also highlights that "in most cases the proposed method does not improve with respect to the standard LTH." The claim's framing as a "generalization" implies an advancement that isn't strongly supported by the empirical findings.

<score>4</score>

Justification:
1. The accuracy curves in Figure 4 show nearly identical performance between magnitude-based and importance-based approaches
2. The authors acknowledge: "the top-1 accuracy of both magnitude, and the L1 and L2 normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition"
3. Layer sparsity evolution plots (Figures 8-10) demonstrate similar pruning patterns across different importance measures
4. The review comment confirms the lack of improvement over standard LTH
5. The claim lacks appropriate qualification about the limited practical impact of this theoretical generalization


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then generalise the LTH to consider weight importance values rather than weight magnitudes.

The review comment to be evaluated is:

Strength:
* The LTH problem considered is popular and the focus on an intuitive/simple issue with typical approaches to LTH (i.e., global pruning leads and layer collapse). I believe this is an interesting direction that the authors motivate well.
* I believe the observation that many different structures of lottery tickets can be discovered within a network has already been made (see e.g., https://arxiv.org/abs/1912.05671), but the authors do provide some extra analysis that goes beyond previous work that is interesting/useful. I really enjoy this extra analysis, and I think the authors could emphasize this more!

Weakness:
* Because the paper is purely empirical, I believe that large-scale experiments on ImageNet are very necessary in this case. This is because, for the LTH, it is widely recognized that many observations made on LTH for smaller-scale experiments do not hold at scale (see https://arxiv.org/abs/1810.05270 and https://arxiv.org/abs/1902.09574). Thus, in this work, performing the larger-scale experiments (e.g., ImageNet) is necessary.
* Accuracy improvements of the proposed procedure are small and there is not a single procedure that performs best. The proposed methods are outperformed by magnitude pruning (i.e., the normal LTH procedure) in many of the considered cases.
* Several pruning works consider pruning in a layer-normalized fashion (see https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html and references, just search “layer wise vs. global pruning” in the paper). None of these techniques are compared against within the analysis. I believe comparing against a few of the SOTA layer-based pruning techniques would be useful.

Small Stuff:
* Figure 1 is difficult to parse. It might be better to use a different subplot for each layer, then plot the sparsity as a function of the pruning iteration.
* L1/L2 normalization of per-layer importance are basically identical (even in the results). It might be better to just use one or the other and state that they perform very similarly.
* In the introduction, your posing of LTH makes it seem like it solves the problem of the train-prune-finetune loop. It might be good to state that LTH still requires this loop (i.e., the network needs to be pre-trained to discover the winning ticket still).
* “Using importance metrics makes this bias both more explicit and stronger, dependent on which metric is used.”: I am not quite sure what this means here or why this would be the case. It might be nice to explain a bit more or maybe provide a citation.
* “The only computational overhead incurred in our method is the calculation of importance scores, which is negligible.”: Whether this computational cost is negligible depends on the importance score you choose/implement, right?

Questions:
* Is there any motivation for the importance metrics that you consider, or are you simply trying a bunch of different approaches and trying to determine the best strategy empirically? Are there other approaches beyond these defined importances that could be useful?

The evidence to be evaluated is:5:\(\theta_{pooled}\gets Pool(abs(\theta_{i,m}))\)

6:\(p\leftarrow\) j-th percentile of \(\theta_{pooled}\)

A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer.

Intuitively, this should alleviate the issue with layer-collapse and should improve the performance of the LTR procedure. Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\) 6:\(score_{pooled}\gets Pool(score_{i})\) 7:\(p\leftarrow\) j-th percentile of \(score_{pooled}\) 8: Prune all connections with \(abs(score_{i})<p\)

Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

First, the magnitude, \(L_{1}\) and \(L_{2}\) criterions all produce LTs of similar performance, the only noticeable difference is in the VGG16 + CIFAR10 experiment.

We notice that the top-1 accuracy of both magnitude, and the \(L_{1}\) and \(L_{2}\) normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition. Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure. We elaborate further on this more in the rest of the paper, where we conduct a deeper study on some differences between the found lottery tickets.

We consider the LTs found by the following importance measures: _magnitude_, \(L_{1}\) and \(L_{2}\).

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure. While we did not note a significant difference in performance when using certain importance measures, we emphasize that the LTH is invariant in a certain sense to the layerwise pruning ratios, as long as the lowest magnitude weights are removed from each layer. Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about generalizing LTH to consider weight importance values appears somewhat overstated. While the authors do propose and test various importance metrics (L1, L2, softmax, min-max), the results show these methods perform similarly to or worse than standard magnitude pruning in most cases. The evidence (particularly Figures 4 and 8) demonstrates that different importance measures don't provide significant advantages over magnitude-based pruning. The review comments also highlight that accuracy improvements are small and no single proposed method consistently outperforms standard LTH. The claim implies a meaningful generalization, but the evidence suggests limited practical benefits.

<score>4</score>

Justification:
- The results show importance measures perform similarly to magnitude pruning with "no significant difference in performance" and "pretty much identical" accuracy at early pruning ratios
- The authors acknowledge their findings "contradict our initial intuition" about the benefits of layerwise importance
- Reviewer notes "accuracy improvements of the proposed procedure are small" and methods are "outperformed by magnitude pruning in many cases"
- Claim overstates the value and impact of the generalization given the empirical evidence


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then generalise the LTH to consider weight importance values rather than weight magnitudes.

The review comment to be evaluated is:

### Strength:

1. This paper is well-written and easy to follow.
2. Both quantitative and qualitative results are reported.
3. Experiments are conducted across different datasets and architectures.

### Weakness:

1. Only small-scale datasets are considered in the experiments.
2. I'm a little bit confused about the main contribution of this work. Using a layerwise importance score can improve the performance of identified lottery tickets or some observations that suggest lottery tickets are not unique. With different mini-batch sequences during training, we may also obtain significantly different lottery tickets from the same initialization.
3. Using layerwise importance score, can the LTH procedure identifies the winning ticket with higher sparsity?

The evidence to be evaluated is:5:\(\theta_{pooled}\gets Pool(abs(\theta_{i,m}))\)

6:\(p\leftarrow\) j-th percentile of \(\theta_{pooled}\)

A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer.

Intuitively, this should alleviate the issue with layer-collapse and should improve the performance of the LTR procedure. Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\) 6:\(score_{pooled}\gets Pool(score_{i})\) 7:\(p\leftarrow\) j-th percentile of \(score_{pooled}\) 8: Prune all connections with \(abs(score_{i})<p\)

Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

First, the magnitude, \(L_{1}\) and \(L_{2}\) criterions all produce LTs of similar performance, the only noticeable difference is in the VGG16 + CIFAR10 experiment.

We notice that the top-1 accuracy of both magnitude, and the \(L_{1}\) and \(L_{2}\) normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition. Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure. We elaborate further on this more in the rest of the paper, where we conduct a deeper study on some differences between the found lottery tickets.

We consider the LTs found by the following importance measures: _magnitude_, \(L_{1}\) and \(L_{2}\).

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure. While we did not note a significant difference in performance when using certain importance measures, we emphasize that the LTH is invariant in a certain sense to the layerwise pruning ratios, as long as the lowest magnitude weights are removed from each layer. Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that the LTH can be generalized to consider weight importance values is partially supported but appears somewhat overstated. While the authors do introduce layerwise importance measures as an alternative to weight magnitudes, their own results show that this modification does not lead to significant performance improvements compared to the original magnitude-based approach. The evidence actually suggests that lottery tickets are less sensitive to the specific pruning criterion than initially thought, as noted by the authors themselves: "introducing the notion of importance does not seem to have a large impact on the LTH procedure."

<score>4</score>

Justification:
1. The experimental results (Fig 4) show similar performance between magnitude-based and importance-based methods
2. The authors acknowledge: "We notice that the top-1 accuracy of both magnitude, and the L1 and L2 normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition"
3. While the methodology is sound, describing this as a "generalization" overstates the contribution given the minimal practical impact shown in the results
4. The review comment correctly identifies confusion about the main contribution, given that the importance scores don't clearly improve lottery ticket identification

The claim would be better stated as exploring an alternative pruning criterion that produces comparable results to the original LTH approach.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we achieve state-of-the-art performance for membership inference in the black-box setting by using a new adversarial approach.

The evidence to be evaluated is:Our work hinges on the observation that an optimized _canary_ image \(x_{mal}\) can be a more effective litmus test for determining the membership of \(x^{*}\).

We now formally present our strategy to generate adversarial queries. For a target data point \((x^{*},y^{*})\), its IN shadow models \(S_{\text{in}}=\{\theta_{1}^{\text{in}},...,\theta_{m}^{\text{in}}\}\), and its OUT shadow models \(S_{\text{out}}=\{\theta_{1}^{\text{out}},...,\theta_{m}^{\text{out}}\}\), the attacker's goal is to find a data point \(x_{\text{mal}}\) such that IN models and OUT models have different behaviors (logits/confidence scores/losses). In the simplest case, the attacker can optimize \(x_{\text{mal}}\) so that IN shadow models have high losses on \(x_{\text{mal}}\) and OUT models to have low losses on \(x_{\text{mal}}\).

To solve this, instead of searching for \(x_{\text{mal}}\) on the whole feasible data domain, we initialize the adversarial query with the target image or the target image with a small noise. Meanwhile, we add an \(\varepsilon\) bound to the perturbation between \(x_{\text{mal}}\) and \(x^{*}\).

We also illustrate our reasoning in Figure 1, where we visualize how the adversarially trained queries might alter the original queries' decision boundaries and provide more confident and diverse predictions.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

During the experiment, for all objectives above, we can easily get very low losses at the end of the optimization, and create Canary queries that perfectly separate the training shadow models. Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample.

Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Figure 1: Query Decision Boundary in Model Parameter Space. We illustrate our idea by plotting the decision boundaries of two queries x1 and x2 in model parameter space. In this case, the target image is indeed in the training data of (green) target model. We sketch three cases. (a) Both augmented queries are unable to separate both distributions and the membership inference fails. (b) Two optimal adversarial queries are generated that are both diverse and, on average, separate both distributions and the attack succeeds. (c) Without the constraints, adversarial queries can overfit and again lead to attack failure.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears moderately overstated. While the research shows improvements over baseline LiRA (4.14% boost in online CIFAR-10, 3% AUC boost in offline scenarios), calling it "state-of-the-art performance" is too strong without comprehensive comparisons against all existing methods. The evidence demonstrates reliable improvements across different datasets and models, but focuses mainly on comparisons to LiRA. The paper shows promising results but would need broader comparisons to definitively claim SOTA status.

<score>3</score>

Justification:
1. The evidence shows clear improvements:
- 23% relative TPR improvement for online CIFAR-10
- >3% AUC score boost on all datasets in offline scenarios
- Consistent performance across multiple datasets and models

2. Limitations in the claim:
- Comparison is primarily against LiRA baseline, not comprehensive evaluation against all existing methods
- "State-of-the-art" claim requires broader benchmarking
- While results are strong, the term "state-of-the-art" suggests definitive superiority that isn't fully supported

3. The evidence supports that this is an effective approach with significant improvements, but the "state-of-the-art" designation appears overstated without more extensive comparisons.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we achieve state-of-the-art performance for membership inference in the black-box setting by using a new adversarial approach.

The review comment to be evaluated is:

Strengths:
1. The idea of using adversarial attacks to optimize the MIA queries is quite novel.
2. Following the recent research in MIAs, the paper uses both AUC and FPR to measure the privacy threats.
3. The Canary attack outperforms the state-of-the-art LiRA attack.
4. The paper investigated practical attack scenarios in online and offline threat models. The effectiveness of the proposed Canary attack has been demonstrated in both threat models.
5. The paper investigates the impact of several critical hyperparameters in Canary attacks.
6. The paper is well-written.

Weaknesses:
1. The optimization process of the query generation introduces additional computational overhead. The attack performance is also related to the number of queries. It would be great to compare the computational cost with LiRA attack. In addition, how do we select perturbation bound ϵ in practice? Does the selection require additional cost?
2. The evaluation only considers differential privacy for privacy protection. The paper could discuss the performance of Canary on other state-of-the-art defenses.

The evidence to be evaluated is:Our work hinges on the observation that an optimized _canary_ image \(x_{mal}\) can be a more effective litmus test for determining the membership of \(x^{*}\).

We now formally present our strategy to generate adversarial queries. For a target data point \((x^{*},y^{*})\), its IN shadow models \(S_{\text{in}}=\{\theta_{1}^{\text{in}},...,\theta_{m}^{\text{in}}\}\), and its OUT shadow models \(S_{\text{out}}=\{\theta_{1}^{\text{out}},...,\theta_{m}^{\text{out}}\}\), the attacker's goal is to find a data point \(x_{\text{mal}}\) such that IN models and OUT models have different behaviors (logits/confidence scores/losses). In the simplest case, the attacker can optimize \(x_{\text{mal}}\) so that IN shadow models have high losses on \(x_{\text{mal}}\) and OUT models to have low losses on \(x_{\text{mal}}\).

To solve this, instead of searching for \(x_{\text{mal}}\) on the whole feasible data domain, we initialize the adversarial query with the target image or the target image with a small noise. Meanwhile, we add an \(\varepsilon\) bound to the perturbation between \(x_{\text{mal}}\) and \(x^{*}\).

We also illustrate our reasoning in Figure 1, where we visualize how the adversarially trained queries might alter the original queries' decision boundaries and provide more confident and diverse predictions.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

During the experiment, for all objectives above, we can easily get very low losses at the end of the optimization, and create Canary queries that perfectly separate the training shadow models. Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample.

Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Figure 1: Query Decision Boundary in Model Parameter Space. We illustrate our idea by plotting the decision boundaries of two queries x1 and x2 in model parameter space. In this case, the target image is indeed in the training data of (green) target model. We sketch three cases. (a) Both augmented queries are unable to separate both distributions and the membership inference fails. (b) Two optimal adversarial queries are generated that are both diverse and, on average, separate both distributions and the attack succeeds. (c) Without the constraints, adversarial queries can overfit and again lead to attack failure.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim slightly overstated. While the evidence shows clear improvements over LiRA (e.g., 4.14% boost in online CIFAR-10, 3% boost in AUC scores for offline scenarios), claiming "state-of-the-art performance" requires more comprehensive comparisons against other existing methods beyond just LiRA. The review comments support the novelty and effectiveness but highlight limitations like computational overhead and limited defense evaluations. The paper demonstrates significant improvements but could be more precise in scope.

<score>3</score>

Justification:
1. Evidence of success:
- Clear improvements over LiRA across multiple datasets
- Consistent performance gains in both online/offline settings
- Reviewer acknowledges novelty and effectiveness

2. Limitations/Overstatements:
- Comparison limited primarily to LiRA
- Computational costs not fully addressed (noted in review)
- Limited evaluation against defensive measures (only differential privacy)
- No comprehensive comparison against full spectrum of existing methods

3. Supporting citations:
- "The difference is especially notable for online CIFAR-10, where we achieve a 4.14% boost over the baseline LiRA"
- "On average, Canary attacks are most powerful in the more realistic offline scenario"
- Review weakness: "The paper could discuss the performance of Canary on other state-of-the-art defenses"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we achieve state-of-the-art performance for membership inference in the black-box setting by using a new adversarial approach.

The review comment to be evaluated is:

Membership inference attacks are of great interest to the privacy and security community, and the paper proposes a method to improve LiRA by learning the query vectors. I find the proposed method interesting, and of potential interest to the community. Paper is clearly written and is easy to understand.

My only comment is that it would be interesting to see the metrics with more stringent privacy budget (for differential privacy), example: $\varepsilon \in [10,20,\cdots,100]$, it will also add another practical dimension to the paper, proving strong DP protections against membership inference.

The evidence to be evaluated is:Our work hinges on the observation that an optimized _canary_ image \(x_{mal}\) can be a more effective litmus test for determining the membership of \(x^{*}\).

We now formally present our strategy to generate adversarial queries. For a target data point \((x^{*},y^{*})\), its IN shadow models \(S_{\text{in}}=\{\theta_{1}^{\text{in}},...,\theta_{m}^{\text{in}}\}\), and its OUT shadow models \(S_{\text{out}}=\{\theta_{1}^{\text{out}},...,\theta_{m}^{\text{out}}\}\), the attacker's goal is to find a data point \(x_{\text{mal}}\) such that IN models and OUT models have different behaviors (logits/confidence scores/losses). In the simplest case, the attacker can optimize \(x_{\text{mal}}\) so that IN shadow models have high losses on \(x_{\text{mal}}\) and OUT models to have low losses on \(x_{\text{mal}}\).

To solve this, instead of searching for \(x_{\text{mal}}\) on the whole feasible data domain, we initialize the adversarial query with the target image or the target image with a small noise. Meanwhile, we add an \(\varepsilon\) bound to the perturbation between \(x_{\text{mal}}\) and \(x^{*}\).

We also illustrate our reasoning in Figure 1, where we visualize how the adversarially trained queries might alter the original queries' decision boundaries and provide more confident and diverse predictions.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

During the experiment, for all objectives above, we can easily get very low losses at the end of the optimization, and create Canary queries that perfectly separate the training shadow models. Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample.

Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Figure 1: Query Decision Boundary in Model Parameter Space. We illustrate our idea by plotting the decision boundaries of two queries x1 and x2 in model parameter space. In this case, the target image is indeed in the training data of (green) target model. We sketch three cases. (a) Both augmented queries are unable to separate both distributions and the membership inference fails. (b) Two optimal adversarial queries are generated that are both diverse and, on average, separate both distributions and the attack succeeds. (c) Without the constraints, adversarial queries can overfit and again lead to attack failure.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Looking at the evidence, I'll assess the claim: "In this paper, we achieve state-of-the-art performance for membership inference in the black-box setting by using a new adversarial approach."

The evidence shows concrete improvements over the baseline LiRA method:
- 4.14% boost in TPR@1%FPR for online CIFAR-10 (23% relative improvement)
- >3% boost in AUC scores across all datasets
- Consistent improvements across multiple datasets, models, and privacy settings

However, some limitations exist:
- The reviewer suggests testing more stringent privacy budgets
- Results are primarily compared to LiRA rather than all state-of-the-art methods
- Performance gains vary across scenarios

While the improvements are meaningful and well-documented, the claim could be more precise by specifying "improved upon LiRA" rather than broadly claiming "state-of-the-art performance."

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we achieve state-of-the-art performance for membership inference in the black-box setting by using a new adversarial approach.

The review comment to be evaluated is:

Strong point:

Perturbing the target data point to boost the membership inference precision is a novel and interesting idea. The core idea behind is to make use of the perturbed variant of the target data point to facilitate the attack. Even though the loss of the target data point may hardly vary if the target data point is considered to be an in- or out-class sample, the perturbed variants within the epsilon-neighbourhood of the target data point can potentially unveil more information to identify the membership.

Weakness:
1. In Table.2, the AUC and the TPR values (given the FPR level) of membership inference attack do not provide a monotonic decreasing as the reproductivity of the target model decreases (as claimed in the paper, the target models in Table.2 are given in a descending order of the model reproductivity.)

2. On page 8, why does the attack performance become worse if the bound epsilon becomes larger or smaller than 2? Any explanation to the observation?  Is there any potential trade-off related to the choice of epsilon ?

3. More theoretical study regarding the effectiveness of the adversarial query could further consolidate this work. The adversarial perturbation added to the target data point may help show the loss distribution within the epsilon-neighborhood of the target data point. Though the loss on the target data point may not be rich enough for a successful membership inference attack, the difference of the loss distribution may provide a better lens to differentiate two different membership classes. For the moment, the reasonability behind Eq.3 is still not very clear.

The evidence to be evaluated is:Our work hinges on the observation that an optimized _canary_ image \(x_{mal}\) can be a more effective litmus test for determining the membership of \(x^{*}\).

We now formally present our strategy to generate adversarial queries. For a target data point \((x^{*},y^{*})\), its IN shadow models \(S_{\text{in}}=\{\theta_{1}^{\text{in}},...,\theta_{m}^{\text{in}}\}\), and its OUT shadow models \(S_{\text{out}}=\{\theta_{1}^{\text{out}},...,\theta_{m}^{\text{out}}\}\), the attacker's goal is to find a data point \(x_{\text{mal}}\) such that IN models and OUT models have different behaviors (logits/confidence scores/losses). In the simplest case, the attacker can optimize \(x_{\text{mal}}\) so that IN shadow models have high losses on \(x_{\text{mal}}\) and OUT models to have low losses on \(x_{\text{mal}}\).

To solve this, instead of searching for \(x_{\text{mal}}\) on the whole feasible data domain, we initialize the adversarial query with the target image or the target image with a small noise. Meanwhile, we add an \(\varepsilon\) bound to the perturbation between \(x_{\text{mal}}\) and \(x^{*}\).

We also illustrate our reasoning in Figure 1, where we visualize how the adversarially trained queries might alter the original queries' decision boundaries and provide more confident and diverse predictions.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

During the experiment, for all objectives above, we can easily get very low losses at the end of the optimization, and create Canary queries that perfectly separate the training shadow models. Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample.

Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Figure 1: Query Decision Boundary in Model Parameter Space. We illustrate our idea by plotting the decision boundaries of two queries x1 and x2 in model parameter space. In this case, the target image is indeed in the training data of (green) target model. We sketch three cases. (a) Both augmented queries are unable to separate both distributions and the membership inference fails. (b) Two optimal adversarial queries are generated that are both diverse and, on average, separate both distributions and the attack succeeds. (c) Without the constraints, adversarial queries can overfit and again lead to attack failure.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence and reviewer comments, I'll evaluate whether the claim "we achieve state-of-the-art performance for membership inference in the black-box setting by using a new adversarial approach" is well-stated or overstated.

Summary:
The evidence shows clear improvements over the baseline LiRA method, with specific numerical improvements documented (e.g., 4.14% boost in TPR for online CIFAR-10, >3% boost in AUC scores). However, the claim of "state-of-the-art performance" needs to be tempered by several limitations noted in the reviewer comments, including:
- Non-monotonic performance with model reproductivity
- Lack of theoretical explanation for epsilon parameter behavior
- Limited theoretical justification for the effectiveness of adversarial queries

While the method shows consistent improvements, calling it "state-of-the-art" without addressing these limitations and providing more theoretical grounding makes the claim somewhat overstated.

<score>3</score>

Justification:
1. The evidence demonstrates clear empirical improvements with specific metrics, supporting part of the claim.
2. The reviewer identifies important limitations and theoretical gaps that aren't acknowledged in the claim.
3. The term "state-of-the-art" implies comprehensive superiority, but the results show varying effectiveness across different conditions.
4. The lack of theoretical understanding about why certain parameters work better (e.g., epsilon=2) suggests more uncertainty than the claim implies.