Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper we study a refinement on this criterion by adding a notion of layerwise importance, which we introduce in section 2.

The review comment to be evaluated is:

Strength:
* The LTH problem considered is popular and the focus on an intuitive/simple issue with typical approaches to LTH (i.e., global pruning leads and layer collapse). I believe this is an interesting direction that the authors motivate well.
* I believe the observation that many different structures of lottery tickets can be discovered within a network has already been made (see e.g., https://arxiv.org/abs/1912.05671), but the authors do provide some extra analysis that goes beyond previous work that is interesting/useful. I really enjoy this extra analysis, and I think the authors could emphasize this more!

Weakness:
* Because the paper is purely empirical, I believe that large-scale experiments on ImageNet are very necessary in this case. This is because, for the LTH, it is widely recognized that many observations made on LTH for smaller-scale experiments do not hold at scale (see https://arxiv.org/abs/1810.05270 and https://arxiv.org/abs/1902.09574). Thus, in this work, performing the larger-scale experiments (e.g., ImageNet) is necessary.
* Accuracy improvements of the proposed procedure are small and there is not a single procedure that performs best. The proposed methods are outperformed by magnitude pruning (i.e., the normal LTH procedure) in many of the considered cases.
* Several pruning works consider pruning in a layer-normalized fashion (see https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html and references, just search “layer wise vs. global pruning” in the paper). None of these techniques are compared against within the analysis. I believe comparing against a few of the SOTA layer-based pruning techniques would be useful.

Small Stuff:
* Figure 1 is difficult to parse. It might be better to use a different subplot for each layer, then plot the sparsity as a function of the pruning iteration.
* L1/L2 normalization of per-layer importance are basically identical (even in the results). It might be better to just use one or the other and state that they perform very similarly.
* In the introduction, your posing of LTH makes it seem like it solves the problem of the train-prune-finetune loop. It might be good to state that LTH still requires this loop (i.e., the network needs to be pre-trained to discover the winning ticket still).
* “Using importance metrics makes this bias both more explicit and stronger, dependent on which metric is used.”: I am not quite sure what this means here or why this would be the case. It might be nice to explain a bit more or maybe provide a citation.
* “The only computational overhead incurred in our method is the calculation of importance scores, which is negligible.”: Whether this computational cost is negligible depends on the importance score you choose/implement, right?

Questions:
* Is there any motivation for the importance metrics that you consider, or are you simply trying a bunch of different approaches and trying to determine the best strategy empirically? Are there other approaches beyond these defined importances that could be useful?

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\)

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

When studying the layerwise sparsities generated by the different pruning criteria (see Figure 8), we can notice a number of interesting observations.

In Appendix C, we showed that some layers are disproportionally pruned less by different pruning criteria. In this section, we will take a look at how the sparsity of these layers evolves throughout the pruning process and compare them to 'normal' layers in both VGG16 (see Figure 9) and ResNet-18 (see Figure 10).

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim stating "In this paper we study a refinement on this criterion by adding a notion of layerwise importance" appears to be well-stated and accurately reflects what is presented in the evidence. The paper clearly demonstrates:

1. Introduction of multiple layerwise importance metrics (L1/L2 normalization, softmax, min-max)
2. Detailed analysis through figures showing how different layers evolve during pruning
3. Clear comparison between different importance measures
4. Empirical validation across different architectures (ResNet, VGG)

The claim is modest and straightforward, avoiding overstatement of the contribution. The reviewer comments acknowledge this is an "interesting direction" while noting some limitations, but these don't contradict the basic claim being made.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper we study a refinement on this criterion by adding a notion of layerwise importance, which we introduce in section 2.

The review comment to be evaluated is:

### Strength:

1. This paper is well-written and easy to follow.
2. Both quantitative and qualitative results are reported.
3. Experiments are conducted across different datasets and architectures.

### Weakness:

1. Only small-scale datasets are considered in the experiments.
2. I'm a little bit confused about the main contribution of this work. Using a layerwise importance score can improve the performance of identified lottery tickets or some observations that suggest lottery tickets are not unique. With different mini-batch sequences during training, we may also obtain significantly different lottery tickets from the same initialization.
3. Using layerwise importance score, can the LTH procedure identifies the winning ticket with higher sparsity?

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\)

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

When studying the layerwise sparsities generated by the different pruning criteria (see Figure 8), we can notice a number of interesting observations.

In Appendix C, we showed that some layers are disproportionally pruned less by different pruning criteria. In this section, we will take a look at how the sparsity of these layers evolves throughout the pruning process and compare them to 'normal' layers in both VGG16 (see Figure 9) and ResNet-18 (see Figure 10).

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "In this paper we study a refinement on this criterion by adding a notion of layerwise importance, which we introduce in section 2" appears to be well-stated. The evidence thoroughly demonstrates the introduction and study of layerwise importance through:
- Detailed mathematical formulations of different layerwise importance measures
- Comprehensive visualizations showing the effects of layerwise pruning across different architectures
- Clear explanation of the motivation and methodology
- Extensive experimental results demonstrating how different layers are pruned differently
- Empirical validation across multiple network architectures (ResNet18, VGG16, ResNet20)

The claim modestly states what the paper does without overstating its contributions or implications.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The key observations of our study are that: i) given a fixed weight initialization, it is possible to extract different lottery tickets that have similar performance, but differ significantly in their structure, ii) these tickets have a noticeable amount of common connections which have low-variance across tickets, and iii) these stable common connections survive the LTH procedure even when the other weights in the model are reinitialized.

The evidence to be evaluated is:In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

Even though these large differences exist between the tickets generated by different importance measures, we established in the previous section that there is no significant difference in the top-1 accuracy of Magnitude, \(L_{1}\) and \(L_{2}\). This is an indication that different kinds of structures exist in a network, that when trained can still achieve a competitive accuracy. As such, we might conclude that there are multiple ways to find Lottery Tickets (on which we expand in section 5). In later iterations, the difference between structures is still noticeable, but much less pronounced, which is expected as the number of pruned connections vastly outnumbers the number of unpruned connections. Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections. In this experiment we determine whether a single initialization may lead to multiple Lottery Tickets.

We find that the resulting tickets share little amount of connections. Only 0.33% of the total (pruned and unpruned) connections are overlapping, while 9.34% of the unpruned connections are overlapping between different settings. We also find that some layers have a noticeably larger fraction of overlapping connections, more specifically the first few convolutional layers as well as the linear classification layer. This can be attributed to the intuition that these layers have less connections and as such are much less overparameterized. Looking at the overlapping connections in the first convolutional layer (Figure 3), which is 68.78% of the remaining connections, compared to the non-overlapping connections, we can notice that the overlapping connections on average have a larger difference between the initial and final weight value.

We also study whether these overlapping connections converge to the same weight in each pruning step up until the winning ticket is found.

We repeat this process for all connections in the ticket and find that on average the overlapping connections have a lower standard deviation than the non-overlapping connections.

The second metric we also apply is counting the number of sign flips. Once again considering the set of weight values a connection has during the pruning steps, we define a sign flip, if the sign of at least one instance in the set is differing from the others. Here too, we find that the overlapping connections have less sign flips than the non-overlapping connections.

As a final sanity check, we do a _partial reinitialization_ test, where rather than starting the LTH/LTR procedure from the initial parameters \(\theta_{0}\), we preserve the weights of \(\theta_{0}\) that correspond to the overlapping connections at the 96.84% sparsity level, while reinitializing the weights of the remaining connections using the same distribution (Kaiming Normal). We then restart the LTR procedure with the studied importance measures using this initialization with the same deterministic training procedure as the initial initialization. The goal is to determine whether these overlapping connections are still (mostly) unpruned given a different initialization, and determine as such whether the overlapping property is dependent on the total initialization, or whether these connections are capable by themselves to steer an initialization in a similar, performant lottery ticket. We find that on average, for three random runs, the amount of remaining connections is higher than the expected value, assuming that each connection has the same chance to be in the ticket. More specifically, for the initial few convolutional layers, we see increases over the expected amount ranging from +8.5% to +70.0%. In the fully connected layer, we can see even higher increases ranging from +440% to +480%. Finally, these effects were not nearly as noticeable in the \(1\times 1\) convolutions, with only -3.2% to +20.0%. These results suggest that these connections are, by grace of their individual initializations, more likely to be part of the ticket. This also indicates that there likely is a minimum amount of connections in some layers that are necessary for a performant ticket, with those mostly concentrated in the initial layers and the classification layer.

A first line of follow-up work was focussed on the properties of the LTH. Zhou et al. (2019) systematically explored different components of the LTH procedure.

Maene et al. (2021) introduces a theoretical proof for the effectiveness of the LTH, while Evci et al. (2022) studied the gradient flow in the LTH procedure. Given the characteristics of our study, our work is situated in this category.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used. These Lottery Tickets all have roughly the same top-1 accuracy, but differ significantly in their structure. Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized.

Input: Figure 3: The evolution of overlapping connections within the first convolutional layer

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim appears to be well-stated and accurately reflects the evidence provided. The evidence thoroughly supports all three key observations:

1) Different lottery tickets with similar performance but different structures are demonstrated through the comparison of tickets generated using different importance measures (L1, L2, Magnitude).

2) Common connections are documented with specific percentages (0.33% total overlap, 9.34% unpruned overlap) and shown to have lower variance and fewer sign flips in the graphs.

3) The partial reinitialization test directly supports the claim about stable connections surviving, showing increases of 8.5-70% in early layers and 440-480% in fully connected layers.

The evidence is comprehensive, quantitative, and directly aligned with each component of the claim without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The key observations of our study are that: i) given a fixed weight initialization, it is possible to extract different lottery tickets that have similar performance, but differ significantly in their structure, ii) these tickets have a noticeable amount of common connections which have low-variance across tickets, and iii) these stable common connections survive the LTH procedure even when the other weights in the model are reinitialized.

The review comment to be evaluated is:

The authors propose to modify the LTH procedure by using a score that measures the layerwise importance in order to select the weights to be pruned. The authors propose a few possible scores and evaluate their performance on a set of experiments. The experiments show that in most of the cases the proposed method does not improve with respect to the standard LTH that uses magnitude in order to select the weights to be pruned. The authors also perform a comparison of lottery tickets obtained with the different scores showing that hey have a noticeable amount of overlapping connections and such connections seems to be more stable.

The evidence to be evaluated is:In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

Even though these large differences exist between the tickets generated by different importance measures, we established in the previous section that there is no significant difference in the top-1 accuracy of Magnitude, \(L_{1}\) and \(L_{2}\). This is an indication that different kinds of structures exist in a network, that when trained can still achieve a competitive accuracy. As such, we might conclude that there are multiple ways to find Lottery Tickets (on which we expand in section 5). In later iterations, the difference between structures is still noticeable, but much less pronounced, which is expected as the number of pruned connections vastly outnumbers the number of unpruned connections. Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections. In this experiment we determine whether a single initialization may lead to multiple Lottery Tickets.

We find that the resulting tickets share little amount of connections. Only 0.33% of the total (pruned and unpruned) connections are overlapping, while 9.34% of the unpruned connections are overlapping between different settings. We also find that some layers have a noticeably larger fraction of overlapping connections, more specifically the first few convolutional layers as well as the linear classification layer. This can be attributed to the intuition that these layers have less connections and as such are much less overparameterized. Looking at the overlapping connections in the first convolutional layer (Figure 3), which is 68.78% of the remaining connections, compared to the non-overlapping connections, we can notice that the overlapping connections on average have a larger difference between the initial and final weight value.

We also study whether these overlapping connections converge to the same weight in each pruning step up until the winning ticket is found.

We repeat this process for all connections in the ticket and find that on average the overlapping connections have a lower standard deviation than the non-overlapping connections.

The second metric we also apply is counting the number of sign flips. Once again considering the set of weight values a connection has during the pruning steps, we define a sign flip, if the sign of at least one instance in the set is differing from the others. Here too, we find that the overlapping connections have less sign flips than the non-overlapping connections.

As a final sanity check, we do a _partial reinitialization_ test, where rather than starting the LTH/LTR procedure from the initial parameters \(\theta_{0}\), we preserve the weights of \(\theta_{0}\) that correspond to the overlapping connections at the 96.84% sparsity level, while reinitializing the weights of the remaining connections using the same distribution (Kaiming Normal). We then restart the LTR procedure with the studied importance measures using this initialization with the same deterministic training procedure as the initial initialization. The goal is to determine whether these overlapping connections are still (mostly) unpruned given a different initialization, and determine as such whether the overlapping property is dependent on the total initialization, or whether these connections are capable by themselves to steer an initialization in a similar, performant lottery ticket. We find that on average, for three random runs, the amount of remaining connections is higher than the expected value, assuming that each connection has the same chance to be in the ticket. More specifically, for the initial few convolutional layers, we see increases over the expected amount ranging from +8.5% to +70.0%. In the fully connected layer, we can see even higher increases ranging from +440% to +480%. Finally, these effects were not nearly as noticeable in the \(1\times 1\) convolutions, with only -3.2% to +20.0%. These results suggest that these connections are, by grace of their individual initializations, more likely to be part of the ticket. This also indicates that there likely is a minimum amount of connections in some layers that are necessary for a performant ticket, with those mostly concentrated in the initial layers and the classification layer.

A first line of follow-up work was focussed on the properties of the LTH. Zhou et al. (2019) systematically explored different components of the LTH procedure.

Maene et al. (2021) introduces a theoretical proof for the effectiveness of the LTH, while Evci et al. (2022) studied the gradient flow in the LTH procedure. Given the characteristics of our study, our work is situated in this category.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used. These Lottery Tickets all have roughly the same top-1 accuracy, but differ significantly in their structure. Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized.

Input: Figure 3: The evolution of overlapping connections within the first convolutional layer

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review of the claim and evidence, I find that it is generally well-stated, with strong experimental support for all three key observations. The evidence shows:

1. Different importance measures produce structurally different tickets with similar performance (supported by layer-wise sparsity plots and overlap analysis)
2. Common connections exist between tickets (9.34% overlap in unpruned connections, particularly in first layers and output layer)
3. Stability of common connections is demonstrated through:
- Lower standard deviation in overlapping connections
- Fewer sign flips
- Survival in partial reinitialization tests (8.5-70% higher retention in conv layers, 440-480% in FC layer)

The claim accurately reflects these findings without overstating their significance.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The key observations of our study are that: i) given a fixed weight initialization, it is possible to extract different lottery tickets that have similar performance, but differ significantly in their structure, ii) these tickets have a noticeable amount of common connections which have low-variance across tickets, and iii) these stable common connections survive the LTH procedure even when the other weights in the model are reinitialized.

The review comment to be evaluated is:

Strength:
* The LTH problem considered is popular and the focus on an intuitive/simple issue with typical approaches to LTH (i.e., global pruning leads and layer collapse). I believe this is an interesting direction that the authors motivate well.
* I believe the observation that many different structures of lottery tickets can be discovered within a network has already been made (see e.g., https://arxiv.org/abs/1912.05671), but the authors do provide some extra analysis that goes beyond previous work that is interesting/useful. I really enjoy this extra analysis, and I think the authors could emphasize this more!

Weakness:
* Because the paper is purely empirical, I believe that large-scale experiments on ImageNet are very necessary in this case. This is because, for the LTH, it is widely recognized that many observations made on LTH for smaller-scale experiments do not hold at scale (see https://arxiv.org/abs/1810.05270 and https://arxiv.org/abs/1902.09574). Thus, in this work, performing the larger-scale experiments (e.g., ImageNet) is necessary.
* Accuracy improvements of the proposed procedure are small and there is not a single procedure that performs best. The proposed methods are outperformed by magnitude pruning (i.e., the normal LTH procedure) in many of the considered cases.
* Several pruning works consider pruning in a layer-normalized fashion (see https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html and references, just search “layer wise vs. global pruning” in the paper). None of these techniques are compared against within the analysis. I believe comparing against a few of the SOTA layer-based pruning techniques would be useful.

Small Stuff:
* Figure 1 is difficult to parse. It might be better to use a different subplot for each layer, then plot the sparsity as a function of the pruning iteration.
* L1/L2 normalization of per-layer importance are basically identical (even in the results). It might be better to just use one or the other and state that they perform very similarly.
* In the introduction, your posing of LTH makes it seem like it solves the problem of the train-prune-finetune loop. It might be good to state that LTH still requires this loop (i.e., the network needs to be pre-trained to discover the winning ticket still).
* “Using importance metrics makes this bias both more explicit and stronger, dependent on which metric is used.”: I am not quite sure what this means here or why this would be the case. It might be nice to explain a bit more or maybe provide a citation.
* “The only computational overhead incurred in our method is the calculation of importance scores, which is negligible.”: Whether this computational cost is negligible depends on the importance score you choose/implement, right?

Questions:
* Is there any motivation for the importance metrics that you consider, or are you simply trying a bunch of different approaches and trying to determine the best strategy empirically? Are there other approaches beyond these defined importances that could be useful?

The evidence to be evaluated is:In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

Even though these large differences exist between the tickets generated by different importance measures, we established in the previous section that there is no significant difference in the top-1 accuracy of Magnitude, \(L_{1}\) and \(L_{2}\). This is an indication that different kinds of structures exist in a network, that when trained can still achieve a competitive accuracy. As such, we might conclude that there are multiple ways to find Lottery Tickets (on which we expand in section 5). In later iterations, the difference between structures is still noticeable, but much less pronounced, which is expected as the number of pruned connections vastly outnumbers the number of unpruned connections. Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections. In this experiment we determine whether a single initialization may lead to multiple Lottery Tickets.

We find that the resulting tickets share little amount of connections. Only 0.33% of the total (pruned and unpruned) connections are overlapping, while 9.34% of the unpruned connections are overlapping between different settings. We also find that some layers have a noticeably larger fraction of overlapping connections, more specifically the first few convolutional layers as well as the linear classification layer. This can be attributed to the intuition that these layers have less connections and as such are much less overparameterized. Looking at the overlapping connections in the first convolutional layer (Figure 3), which is 68.78% of the remaining connections, compared to the non-overlapping connections, we can notice that the overlapping connections on average have a larger difference between the initial and final weight value.

We also study whether these overlapping connections converge to the same weight in each pruning step up until the winning ticket is found.

We repeat this process for all connections in the ticket and find that on average the overlapping connections have a lower standard deviation than the non-overlapping connections.

The second metric we also apply is counting the number of sign flips. Once again considering the set of weight values a connection has during the pruning steps, we define a sign flip, if the sign of at least one instance in the set is differing from the others. Here too, we find that the overlapping connections have less sign flips than the non-overlapping connections.

As a final sanity check, we do a _partial reinitialization_ test, where rather than starting the LTH/LTR procedure from the initial parameters \(\theta_{0}\), we preserve the weights of \(\theta_{0}\) that correspond to the overlapping connections at the 96.84% sparsity level, while reinitializing the weights of the remaining connections using the same distribution (Kaiming Normal). We then restart the LTR procedure with the studied importance measures using this initialization with the same deterministic training procedure as the initial initialization. The goal is to determine whether these overlapping connections are still (mostly) unpruned given a different initialization, and determine as such whether the overlapping property is dependent on the total initialization, or whether these connections are capable by themselves to steer an initialization in a similar, performant lottery ticket. We find that on average, for three random runs, the amount of remaining connections is higher than the expected value, assuming that each connection has the same chance to be in the ticket. More specifically, for the initial few convolutional layers, we see increases over the expected amount ranging from +8.5% to +70.0%. In the fully connected layer, we can see even higher increases ranging from +440% to +480%. Finally, these effects were not nearly as noticeable in the \(1\times 1\) convolutions, with only -3.2% to +20.0%. These results suggest that these connections are, by grace of their individual initializations, more likely to be part of the ticket. This also indicates that there likely is a minimum amount of connections in some layers that are necessary for a performant ticket, with those mostly concentrated in the initial layers and the classification layer.

A first line of follow-up work was focussed on the properties of the LTH. Zhou et al. (2019) systematically explored different components of the LTH procedure.

Maene et al. (2021) introduces a theoretical proof for the effectiveness of the LTH, while Evci et al. (2022) studied the gradient flow in the LTH procedure. Given the characteristics of our study, our work is situated in this category.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used. These Lottery Tickets all have roughly the same top-1 accuracy, but differ significantly in their structure. Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized.

Input: Figure 3: The evolution of overlapping connections within the first convolutional layer

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim appears to be well-supported by the evidence presented. The evidence demonstrates:

1. Different lottery tickets can be extracted (shown by different layer sparsity patterns in Fig. 9)
2. Common connections exist between tickets (quantified as 9.34% overlap of unpruned connections)
3. These common connections have lower variance and fewer sign flips (shown in Fig. 2)
4. The partial reinitialization test confirms stability of common connections (with increases of 8.5-70% in early layers and 440-480% in FC layer)

The claim conservatively describes these findings without overstating their significance or implications. The statements align closely with the empirical results while acknowledging variability.

<score>1</score>

Justification:
- The claim directly maps to demonstrated experimental results
- Quantitative evidence supports each key point
- Language used is measured ("noticeable amount" rather than making stronger claims)
- The reviewer comments acknowledge the novelty and validity of these observations
- While noting some limitations (like lack of ImageNet experiments), the reviewer's concerns don't contradict the specific claims made about the observed phenomena
- The figures (particularly 2, 3, and 9) provide clear visualization of the described patterns
- Claims are appropriately scoped to the specific findings without overgeneralization


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The key observations of our study are that: i) given a fixed weight initialization, it is possible to extract different lottery tickets that have similar performance, but differ significantly in their structure, ii) these tickets have a noticeable amount of common connections which have low-variance across tickets, and iii) these stable common connections survive the LTH procedure even when the other weights in the model are reinitialized.

The review comment to be evaluated is:

### Strength:

1. This paper is well-written and easy to follow.
2. Both quantitative and qualitative results are reported.
3. Experiments are conducted across different datasets and architectures.

### Weakness:

1. Only small-scale datasets are considered in the experiments.
2. I'm a little bit confused about the main contribution of this work. Using a layerwise importance score can improve the performance of identified lottery tickets or some observations that suggest lottery tickets are not unique. With different mini-batch sequences during training, we may also obtain significantly different lottery tickets from the same initialization.
3. Using layerwise importance score, can the LTH procedure identifies the winning ticket with higher sparsity?

The evidence to be evaluated is:In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

Even though these large differences exist between the tickets generated by different importance measures, we established in the previous section that there is no significant difference in the top-1 accuracy of Magnitude, \(L_{1}\) and \(L_{2}\). This is an indication that different kinds of structures exist in a network, that when trained can still achieve a competitive accuracy. As such, we might conclude that there are multiple ways to find Lottery Tickets (on which we expand in section 5). In later iterations, the difference between structures is still noticeable, but much less pronounced, which is expected as the number of pruned connections vastly outnumbers the number of unpruned connections. Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections. In this experiment we determine whether a single initialization may lead to multiple Lottery Tickets.

We find that the resulting tickets share little amount of connections. Only 0.33% of the total (pruned and unpruned) connections are overlapping, while 9.34% of the unpruned connections are overlapping between different settings. We also find that some layers have a noticeably larger fraction of overlapping connections, more specifically the first few convolutional layers as well as the linear classification layer. This can be attributed to the intuition that these layers have less connections and as such are much less overparameterized. Looking at the overlapping connections in the first convolutional layer (Figure 3), which is 68.78% of the remaining connections, compared to the non-overlapping connections, we can notice that the overlapping connections on average have a larger difference between the initial and final weight value.

We also study whether these overlapping connections converge to the same weight in each pruning step up until the winning ticket is found.

We repeat this process for all connections in the ticket and find that on average the overlapping connections have a lower standard deviation than the non-overlapping connections.

The second metric we also apply is counting the number of sign flips. Once again considering the set of weight values a connection has during the pruning steps, we define a sign flip, if the sign of at least one instance in the set is differing from the others. Here too, we find that the overlapping connections have less sign flips than the non-overlapping connections.

As a final sanity check, we do a _partial reinitialization_ test, where rather than starting the LTH/LTR procedure from the initial parameters \(\theta_{0}\), we preserve the weights of \(\theta_{0}\) that correspond to the overlapping connections at the 96.84% sparsity level, while reinitializing the weights of the remaining connections using the same distribution (Kaiming Normal). We then restart the LTR procedure with the studied importance measures using this initialization with the same deterministic training procedure as the initial initialization. The goal is to determine whether these overlapping connections are still (mostly) unpruned given a different initialization, and determine as such whether the overlapping property is dependent on the total initialization, or whether these connections are capable by themselves to steer an initialization in a similar, performant lottery ticket. We find that on average, for three random runs, the amount of remaining connections is higher than the expected value, assuming that each connection has the same chance to be in the ticket. More specifically, for the initial few convolutional layers, we see increases over the expected amount ranging from +8.5% to +70.0%. In the fully connected layer, we can see even higher increases ranging from +440% to +480%. Finally, these effects were not nearly as noticeable in the \(1\times 1\) convolutions, with only -3.2% to +20.0%. These results suggest that these connections are, by grace of their individual initializations, more likely to be part of the ticket. This also indicates that there likely is a minimum amount of connections in some layers that are necessary for a performant ticket, with those mostly concentrated in the initial layers and the classification layer.

A first line of follow-up work was focussed on the properties of the LTH. Zhou et al. (2019) systematically explored different components of the LTH procedure.

Maene et al. (2021) introduces a theoretical proof for the effectiveness of the LTH, while Evci et al. (2022) studied the gradient flow in the LTH procedure. Given the characteristics of our study, our work is situated in this category.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used. These Lottery Tickets all have roughly the same top-1 accuracy, but differ significantly in their structure. Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized.

Input: Figure 3: The evolution of overlapping connections within the first convolutional layer

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the key observations are well-supported by detailed experimental data and analysis. The evidence demonstrates:

1. Different lottery tickets were successfully extracted using various importance measures (L1, L2, Magnitude) with similar performance but distinct structures, as shown in Figure 9.

2. Overlapping connections exist between tickets (9.34% of unpruned connections), particularly in first convolutional layers (68.78%) and output layer, supported by quantitative data.

3. The stability of common connections is demonstrated through:
- Lower standard deviations (Figure 2)
- Fewer sign flips in overlapping vs non-overlapping connections
- Survival after reinitialization (+8.5% to +70% in conv layers, +440-480% in FC layer)

The claim carefully reflects the evidence without exaggeration and acknowledges the specific conditions and limitations.

<score>1</score>

Justification:
1. The claim is precise and accurately reflects the experimental findings
2. Each statement is supported by concrete evidence and metrics
3. The language used is measured ("noticeable amount" rather than absolute claims)
4. Results are contextualized within specific experimental conditions
5. The evidence, including multiple figures and statistical analyses, strongly supports all three key observations
6. The review comments don't highlight any overstatement issues, mainly focusing on dataset scale limitations


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we study a number of different approaches that aim at recovering some of this layer distributional context by computing a connection importance value that is dependent on the weights of the other connections in the same layer.

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure.

Due to the definition of Min-Max normalization, at least one connection in each layer will have an importance of 0 and at least one connection in each layer has an importance of 1.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized. This suggests that we should focus our attention on these specific connections to find LTs more efficiently.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears well-stated. The paper thoroughly studies different layerwise importance measures (L1, L2, softmax, min-max normalization) to improve upon global pruning by incorporating layer distributional context. The evidence shows detailed mathematical formulations, extensive experimental results across multiple architectures, and clear visualizations demonstrating how different importance measures affect layer sparsity evolution. The claim accurately reflects what was studied without overstating the findings or conclusions. The experimental evidence strongly supports the core assertion about studying approaches for recovering layer distributional context through connection importance values.

<score>1</score>

Let me explain the key factors in my evaluation:

1. Detailed mathematical formulations are provided for calculating layerwise importance values
2. Comprehensive empirical analysis with multiple models and datasets
3. Clear visualization of results showing layer sparsity evolution
4. Measured discussion of findings without overreaching conclusions
5. Transparent discussion of limitations and observations
6. Results backed by extensive experimental data shown in figures
7. Claim language accurately matches the scope and depth of the investigation presented

The claim is precise and well-supported by the evidence without exaggeration.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we study a number of different approaches that aim at recovering some of this layer distributional context by computing a connection importance value that is dependent on the weights of the other connections in the same layer.

The review comment to be evaluated is:

The authors propose to modify the LTH procedure by using a score that measures the layerwise importance in order to select the weights to be pruned. The authors propose a few possible scores and evaluate their performance on a set of experiments. The experiments show that in most of the cases the proposed method does not improve with respect to the standard LTH that uses magnitude in order to select the weights to be pruned. The authors also perform a comparison of lottery tickets obtained with the different scores showing that hey have a noticeable amount of overlapping connections and such connections seems to be more stable.

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance.

To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

The first approach, as followed by \(L_{1}\), \(L_{2}\) and SoftMax is the sum-to-one principle, to rescale the importances of a layer such that the total importance of a layer is one. Then the importances themselves are still determined by (an operation on) the magnitude, but rescaled. The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.

In case of the softmax function, we notice that the importance distribution is very narrow and the lower bound is exactly \(1/|W|\), while for the other functions the distribution is more broad with a lower bound of \(0\). This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and a standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which \(e^{x}\approx 1\) if \(|x|\ll 1\). Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. \(L_{2}\) is another measure that transforms the relevance distribution to be more narrower than the weight distribution due to the quadratic function. The other measures, \(L_{1}\) and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.

Our hypothesis is that this lack of difference indicates that the approaches to finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have a large impact on the LTH procedure.

Due to the definition of Min-Max normalization, at least one connection in each layer will have an importance of 0 and at least one connection in each layer has an importance of 1.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

Furthermore, we noticed that given a fixed initialization, it is possible to extract different lottery tickets that are dependent on the importance measure that is used.

Looking at the remaining connections in the tickets, we noticed that there exist a noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have a lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized. This suggests that we should focus our attention on these specific connections to find LTs more efficiently.

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.

Input: Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for overlapping and non-overlapping connections at 96.48% sparsity for the selected importance measures.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input: Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input: Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the {L1, L2, Magnitude} importance measures

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-stated based on several key points:

1. The evidence clearly demonstrates the authors' study of different layerwise importance approaches (L1, L2, softmax, min-max normalization) and their impact on connection pruning.

2. The empirical results shown in the figures (particularly Figures 1, 4, and 8) validate that they examined how these approaches affect layer distribution context through sparsity patterns and model performance.

3. The reviewer comment supports that the authors thoroughly investigated different scoring methods, though noting the methods didn't necessarily improve upon standard LTH.

4. The claim modestly states they "study" approaches rather than making stronger assertions about improvements or superiority.

The claim accurately reflects what was investigated without overstating the findings or their significance.

<score>1</score>

Justification:
- The extensive experimental evidence in Figures 1, 4, 8-10 demonstrates systematic study of different layerwise importance measures
- Results comprehensively examine impact on layer distributions and pruning patterns
- Language used in claim is appropriately measured ("study", "aim at recovering") rather than making excessive claims
- Direct evidence shows implementation and analysis of the described approaches
- Review comment confirms thorough investigation of the approaches claimed