Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization.

The review comment to be evaluated is:

## Strengths
- Equivariant networks have more hyperparameters to tune due to the choice of symmetry group.  For example, in E(2)-CNN (https://arxiv.org/abs/1911.08251), the authors experiment with symmetry groups $C_n$ and $D_n$ for many values of n before finding n=12 to work best.  This paper proposes a method for automatically determining the correct level of symmetry.
- Moreover, on many problems end-to-end strict symmetry is not desirable if the symmetry in the domain is only approximate or if symmetry constraints interfere with optimization.  The proposed method adds to the relatively few methods which learn relaxed symmetry constraints and adapt to the level of symmetry in the data.
- I found the equivariance relaxation morphism to be a simple and effective idea for relaxing symmetry constraints in networks on a per-layer basis.
- The [G]-mixed equivariant layer also seems like a reasonable idea for selecting the degree of equivariance by differentiably choosing weights over kernels with different levels of equivariance.  In theory this allows the model to learn a specific equivariance or an approximate equivariance which is in-between.  It also allows the network to reduce the symmetry constraint in later layers, which is what is done effectively by CNNs in practice by downsampling with stride.
- Experimental results on the learned subgroup weights (Fig. 2) reveal interesting trends showing that the networks do select for higher equivariance in early layers and lower equivariance in later layers which matches what has found to be effective in practice for equivariant networks.

## Weaknesses / Questions
- I am not sure of the framing of the method.  I am not an expert on NAS and am open to correction, but it seems to me both proposed methods find a trained network, not an architecture, since they are training the network weights and selecting an architecture at the same time.  The final architecture may not have great performance if trained from scratch.  In this sense, it seems the method is more a special training procedure than NAS method.   EquiNAS_D, in particular, is simply an architecture in which some weights are optimized alternatingly.
- Given the evolutionary algorithm starts with a single network, how do we know to what extent its evolution is guided by architecture versus the specific weights in the mutated networks?  In experiments, I think both population level variance and initialization variance should be accounted for.
- One of the biggest issues is the fact that the relaxation morphism increases the number of parameters making it difficult to distinguish performance increases are from relaxing constrains or increasing parameter count.  The authors do discuss this issue and suggest their optimization strategy over both parameter count and accuracy helps.  It would be better, however, if the number of parameters could be preserved by the relaxation.
- The experimental results are not completely convincing of when this method would add significant value over simply doing hyperparameter search over equivariant networks.  In Table 1, an equivariant baseline is best on rotMNIST and ISIC, and a non-rotationally equivariant network with equivariant initialization is best for Galaxy10.  In table 2, EquiNAS_D does do best for rotMNIST, but the score is much lower than the best values in E(2)-CNN after full tuning of the best equivariant method.  In Galaxy10, EquiNAS_D does outperform, but with very high variance.
- I have some issues with the clarity of the description of 4.2.  The condition that all groups in $[G]$ be subgroups or supergroups of all other groups in $[G]$ does not appear to be met by the expermental example since $D_2$ is not a subgroup or supergroup of $C_4$.  It's also not clear to me why this condition is necessary.  It is a fairly strict condition, although I think it would include useful cases.  I am also a bit unclear on the form of the input signal $f$.  In order to be convolved with $\tilde{\Psi}$ and have $G$-equivariance for each $G$ in $[G]$, $f$ should be defined over a supergroup of all $G \in [G]$.  Is this the case?   I am correct that if $z_G$ is a one-hot vector on the group $G$, then the operation is $G$-equivariant?

## Questions
- If the evolution algorithm can only mutate in the direction of symmetry constrain relaxation, doesn't this bias the algorithm towards relaxing all the symmetry constraints?
- Does this work have a relation to diffstride (https://arxiv.org/abs/2202.01653) in that stride is typically a hyperparameter that corresonds to a choice of the subgroup of the translation group and can be shown to differentiably optimized?
- After equivariance is broken in a given layer, there isn't much point in principle to persevering it downstream.  Are mutations automatically applied to all following layers or did your method discover this?  If so, I think that is interesting enough to be considered a strength.

## Minor
- In Eqn 1, if you reverse the order of $f$ and $\Psi$ and drop the sum over c and indices $c,d$, the equation can be simplified using matrix multiplication.
- 4.1, para 2, "neutral element" -> "identity element"
- 4.1 may benefit from writing as a proposition
- 4.1, Para 3, Line 5,  $h \in G'$ -> $h' \in G'$
- Sec 5., Line 4, typo "optimizffe"

The evidence to be evaluated is:Neural architecture search (NAS) aims to optimize both the architecture and its parameters for a given task. Liu et al. (2018) approaches this difficult bi-level optimization by creating a large super-network containing all possible elements and continuously relaxing the discrete architectural parameters to enable search by gradient descent. Other NAS approaches include evolutionary algorithms (Real et al., 2017; Lu et al., 2019; Elsken et al., 2017) and reinforcement learning (Zoph & Le, 2017), which search over discretely represented architectures.

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

We present two NAS methods that utilize the presented mechanisms for discovering appropriate equivariance during training: Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) and Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)). Both methods optimize an architecture while learning weights, yielding a final trained network adapted to equivariances present in the training data.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm. The Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) algorithm, given in Algorithm 1, is similar to other evolutionary NAS methods such as Elsken et al. (2017) with pareto selection as in Falanti et al. (2022). A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Offspring are generated from each parent separately by mutation using the relaxation morphism. This preserves the weights of the parametrized equivariance during mutation, allowing for the continuous training of networks over evolution by inheritance from parent individuals. Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers. This constraint yields local equivariance properties for the network, as shown in Weiler and Cesa (2019) and Elsayed et al. (2020) to be empirically favorable in image classification tasks. The resulting individuals are each trained independently for a given training time, and then this process repeats. The second objective of minimizing parameter count is intended to advance efficient networks, such as those with large symmetry groups. Accuracy-based selection alone would necessarily prefer larger networks as mutation via the equivariance relaxation morphism results in two networks with identical performance but different size, the relaxed network having more parameters, until training; potentially short-term increases in validation accuracy after training would then result in the selection of individuals with more parameters. Thus, the proposed strategy of selecting both pareto-dominant and high-accuracy individuals is intended to maintain a diverse yet efficient population without succumbing to overly greedy selections too early. In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm. Our Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)) algorithm, defined in Algorithm 2, is inspired by DARTS (Liu et al., 2018) with significant changes detailed in the following paragraphs. EquiNAS\({}_{D}\) simplifies the bilevel optimization of the architecture weighting parameters \(Z\) and filter weights \(W\) into alternating independent updates, computing the gradient update for \(Z\) with the current, rather than optimal, \(\Psi\) for the current architecture encoded by \(Z\), to boost search efficiency with minimal performance loss compared to higher order approximations (Liu et al., 2018).

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance. We next study empirically the two EquiNAS methods on three datasets, one with known rotational symmetry and two with unknown but visually significant rotational and reflectional symmetry. We focus on the regular representation of groups and show experiments with reflectional and up to 4-fold rotational symmetry groups applied to image classification tasks. Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation.

As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance.

The advantages of equivariance search methods are most apparent in the Galaxy10 benchmark.

However, the \(D_{4}\) baseline fails at the Galaxy10 task, demonstrating that the same equivariant architecture can not always be naively applied. Both search methods, EquiNAS\({}_{E}\) and RPP, outperform all baseline models on Galaxy10, and by a large margin for EquiNAS\({}_{E}\).

the selected population maintains a fully equivariant network in every generation. The final selected population originates from two main lineages, one staying fully equivariant until the last generations and the other diverging from the fully equivariant network midway through, showing that training with dynamically constrained parametrizations can produce performant models.

EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

EquiNAS\({}_{D}\) achieves better test accuracy than the other comparable methods on RotMNIST and Galaxy10. Due to differences in training protocol, only comparisons of relative rankings with Table 1 are possible: baseline methods accuracies follow similar ranking patterns, suggesting the benefit of general \(C_{4}\) equivariance for RotMNIST and Galaxy10 and general \(D_{4}\) equivariance, including RPP, for ISIC.

Previous differentiable NAS works often used regularization of network size or even architecture weighting parameters themselves to encourage efficient architectures with a single highly weighted choice for each layer. However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer. The EquiNAS\({}_{D}\) algorithm avoids many known problems in differentiable NAS such as the discretization gap that occurs when searching over a continuous relaxation of a discrete architectural search space (Xie et al., 2021), such as that of EquiNAS\({}_{E}\). Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

EquiNAS\({}_{E}\) may particularly work well on tasks that benefit from local equivariance, determined by analyzing the architecture weighting parameters from first applying the more efficient EquiNAS\({}_{D}\), as well as for finding good discrete architectures within which to retrain weights, based on the ablation and random comparisons.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\). We investigate how dynamic equivariance achieved by these algorithms affects the training and performance of models across multiple image classification tasks of varying complexity and assumed symmetry, demonstrating that these techniques can search for performant architectures and weights even on noisy tasks.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization.

* Zoph & Le (2017) Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning.

Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\). For example, an image segmentation neural network should be \(T(2)\)-equivariant: shifting the input should result in the same shift in the output.

Evolutionary algorithms are optimization methods inspired by evolution in biology, where individuals in a population compete with their phenotypic traits in order to pass on their genotypic traits to offspring.

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

Group convolutional layers The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism.

The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

For both EquiNAS\({}_{E}\) and EquiNAS\({}_{D}\) experiments, we use the same backbone architecture, such that the static baselines have the same architecture across experiments. The architectures have a lifting layer followed by 7 group convolutional layers, for a total of 8 convolutional layers.

For EquiNAS\({}_{E}\), the parent selection size is 5, the training time per generation is 0.5 epochs, and the number of generations is 50 for all tasks.

To explore the symmetry discovery of EquiNAS\({}_{D}\), we apply it to six augmentations of the MNIST dataset (LeCun et al., 1998), where each augmentation applies the group actions of each group in \(\{C_{1},C_{2},C_{4},D_{1},D_{2},D_{4}\}\) respectively. The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

Shown in Figure 5, EquiNAS\({}_{E}\) outperforms EquiNAS\({}_{E}\) Random Select, showing the benefit of using informed selection to guide the relaxation of equivariance constraints over training. Additionally, EquiNAS\({}_{E}\) Retrain outperforms the Random Static baseline, showing that using compute in an informed search is more beneficial than just randomly searching the space of static architecture constraints. EquiNAS\({}_{D}\) finds competitive architectures on average and can find architectures which outperform baseline choices like architectures fully equivariant to \(C_{1}\) or \(D_{4}\). From comparing EquiNAS\({}_{D}\) Retrain to EquiNAS\({}_{D}\) results in Figure 8, retraining a resulting architecture is not consistently better or worse than using the final weights from EquiNAS\({}_{D}\), showing that there isn't a disadvantage to training weights during search and avoiding the additional cost of a post-search training step. The use of randomized loss information in Random Static in Figure 8 shows that an informed search for architecture hyperparameters is generally useful.

On this baseline, EquiNAS\({}_{D}\) greatly outperforms a search of random architectures, demonstrating that EquiNAS\({}_{D}\) can discover the appropriate equivariance for a specific dataset over fixed or randomly selected architectures.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space. Our methods enable searching for both architecture and parameters concurrently in a single training process. This approach is more efficient than NAS approaches that only search for architectures, requiring a retraining process within the resulting architecture for evaluation.

Input: Table 1: Test error (lower is better) in percent of incorrect classifications across tasks and approaches. Statistics are aggregated over the final selected population of 5 individuals for EquiNASE and across 5 random seeds for all other methods. The best and second best average errors per task are highlighted. See Figures 5-6 in Appendix D for individual results and additional experiments.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 8: Test errors for experiments of Section 6.2.

Input: Figure 5: Test errors for experiments of Section 6.1.

Input: Figure 2: Historical parameter counts of all selected individuals for EquiNASE on RotMNIST. The architectures of the final selected population are labeled. Each history is colored by the final test accuracy, measured of each individual upon removal. For other tasks, see Figure 7 in Appendix D.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 7: Historical parameter counts of each selected individual for EquiNASE (see Figure 2 for RotMNIST).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that the paper presents evolutionary and differentiable NAS algorithms for equivariance-aware optimization appears well-stated with only minor overstatement. The evidence shows detailed descriptions and implementations of both EquiNAS_E and EquiNAS_D algorithms, supported by extensive experimental results across multiple datasets. While the reviewer notes some limitations regarding whether these are truly NAS methods versus training procedures, the core claim about presenting these algorithms is accurate. The methods demonstrably perform equivariance-aware architectural optimization, even if there are some caveats about their exact classification as NAS algorithms.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization.

The review comment to be evaluated is:

### Strengths:
1. The problem of finding the correct equivariance that can exploit symmetries in the data is important for the development of general-purpose NAS methods.
2. The proposed approaches are simple and natural procedures.
3. The writing is fairly clear and the experimental results are interesting.
4. Code will be released upon acceptance.

### Weaknesses:
1. The experimental evaluation is somewhat limited. Given the fairly direct extension of prior work (morphisms and weight-sharing) to actually developing the search algorithms, I would expect more emphasis to be placed on experimental depth. This could either come in the form of evaluation on interesting benchmarks where we might expect finding symmetries to outperform regular CNNs (e.g. Tu et al. (2022)) and/or more detailed analysis on more involved search spaces of whether the proposed methods are recovering the “right” symmetry for the dataset, even if synthetic.
2. The authors do not consider the random search baseline for their method. More generally, it would be useful to follow the NAS reproducibility checklist here (Lindauer & Hutter, 2020).

### References:
- Lindauer, Hutter. Best practices for scientific research on neural architecture search. JMLR 2020.
- Tu, Roberts, Khodak, Shen, Sala, Talwalkar. NAS-Bench-360: Benchmarking neural architecture search on diverse tasks. NeurIPS 2022.

The evidence to be evaluated is:Neural architecture search (NAS) aims to optimize both the architecture and its parameters for a given task. Liu et al. (2018) approaches this difficult bi-level optimization by creating a large super-network containing all possible elements and continuously relaxing the discrete architectural parameters to enable search by gradient descent. Other NAS approaches include evolutionary algorithms (Real et al., 2017; Lu et al., 2019; Elsken et al., 2017) and reinforcement learning (Zoph & Le, 2017), which search over discretely represented architectures.

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

We present two NAS methods that utilize the presented mechanisms for discovering appropriate equivariance during training: Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) and Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)). Both methods optimize an architecture while learning weights, yielding a final trained network adapted to equivariances present in the training data.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm. The Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) algorithm, given in Algorithm 1, is similar to other evolutionary NAS methods such as Elsken et al. (2017) with pareto selection as in Falanti et al. (2022). A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Offspring are generated from each parent separately by mutation using the relaxation morphism. This preserves the weights of the parametrized equivariance during mutation, allowing for the continuous training of networks over evolution by inheritance from parent individuals. Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers. This constraint yields local equivariance properties for the network, as shown in Weiler and Cesa (2019) and Elsayed et al. (2020) to be empirically favorable in image classification tasks. The resulting individuals are each trained independently for a given training time, and then this process repeats. The second objective of minimizing parameter count is intended to advance efficient networks, such as those with large symmetry groups. Accuracy-based selection alone would necessarily prefer larger networks as mutation via the equivariance relaxation morphism results in two networks with identical performance but different size, the relaxed network having more parameters, until training; potentially short-term increases in validation accuracy after training would then result in the selection of individuals with more parameters. Thus, the proposed strategy of selecting both pareto-dominant and high-accuracy individuals is intended to maintain a diverse yet efficient population without succumbing to overly greedy selections too early. In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm. Our Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)) algorithm, defined in Algorithm 2, is inspired by DARTS (Liu et al., 2018) with significant changes detailed in the following paragraphs. EquiNAS\({}_{D}\) simplifies the bilevel optimization of the architecture weighting parameters \(Z\) and filter weights \(W\) into alternating independent updates, computing the gradient update for \(Z\) with the current, rather than optimal, \(\Psi\) for the current architecture encoded by \(Z\), to boost search efficiency with minimal performance loss compared to higher order approximations (Liu et al., 2018).

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance. We next study empirically the two EquiNAS methods on three datasets, one with known rotational symmetry and two with unknown but visually significant rotational and reflectional symmetry. We focus on the regular representation of groups and show experiments with reflectional and up to 4-fold rotational symmetry groups applied to image classification tasks. Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation.

As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance.

The advantages of equivariance search methods are most apparent in the Galaxy10 benchmark.

However, the \(D_{4}\) baseline fails at the Galaxy10 task, demonstrating that the same equivariant architecture can not always be naively applied. Both search methods, EquiNAS\({}_{E}\) and RPP, outperform all baseline models on Galaxy10, and by a large margin for EquiNAS\({}_{E}\).

the selected population maintains a fully equivariant network in every generation. The final selected population originates from two main lineages, one staying fully equivariant until the last generations and the other diverging from the fully equivariant network midway through, showing that training with dynamically constrained parametrizations can produce performant models.

EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

EquiNAS\({}_{D}\) achieves better test accuracy than the other comparable methods on RotMNIST and Galaxy10. Due to differences in training protocol, only comparisons of relative rankings with Table 1 are possible: baseline methods accuracies follow similar ranking patterns, suggesting the benefit of general \(C_{4}\) equivariance for RotMNIST and Galaxy10 and general \(D_{4}\) equivariance, including RPP, for ISIC.

Previous differentiable NAS works often used regularization of network size or even architecture weighting parameters themselves to encourage efficient architectures with a single highly weighted choice for each layer. However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer. The EquiNAS\({}_{D}\) algorithm avoids many known problems in differentiable NAS such as the discretization gap that occurs when searching over a continuous relaxation of a discrete architectural search space (Xie et al., 2021), such as that of EquiNAS\({}_{E}\). Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

EquiNAS\({}_{E}\) may particularly work well on tasks that benefit from local equivariance, determined by analyzing the architecture weighting parameters from first applying the more efficient EquiNAS\({}_{D}\), as well as for finding good discrete architectures within which to retrain weights, based on the ablation and random comparisons.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\). We investigate how dynamic equivariance achieved by these algorithms affects the training and performance of models across multiple image classification tasks of varying complexity and assumed symmetry, demonstrating that these techniques can search for performant architectures and weights even on noisy tasks.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization.

* Zoph & Le (2017) Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning.

Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\). For example, an image segmentation neural network should be \(T(2)\)-equivariant: shifting the input should result in the same shift in the output.

Evolutionary algorithms are optimization methods inspired by evolution in biology, where individuals in a population compete with their phenotypic traits in order to pass on their genotypic traits to offspring.

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

Group convolutional layers The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism.

The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

For both EquiNAS\({}_{E}\) and EquiNAS\({}_{D}\) experiments, we use the same backbone architecture, such that the static baselines have the same architecture across experiments. The architectures have a lifting layer followed by 7 group convolutional layers, for a total of 8 convolutional layers.

For EquiNAS\({}_{E}\), the parent selection size is 5, the training time per generation is 0.5 epochs, and the number of generations is 50 for all tasks.

To explore the symmetry discovery of EquiNAS\({}_{D}\), we apply it to six augmentations of the MNIST dataset (LeCun et al., 1998), where each augmentation applies the group actions of each group in \(\{C_{1},C_{2},C_{4},D_{1},D_{2},D_{4}\}\) respectively. The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

Shown in Figure 5, EquiNAS\({}_{E}\) outperforms EquiNAS\({}_{E}\) Random Select, showing the benefit of using informed selection to guide the relaxation of equivariance constraints over training. Additionally, EquiNAS\({}_{E}\) Retrain outperforms the Random Static baseline, showing that using compute in an informed search is more beneficial than just randomly searching the space of static architecture constraints. EquiNAS\({}_{D}\) finds competitive architectures on average and can find architectures which outperform baseline choices like architectures fully equivariant to \(C_{1}\) or \(D_{4}\). From comparing EquiNAS\({}_{D}\) Retrain to EquiNAS\({}_{D}\) results in Figure 8, retraining a resulting architecture is not consistently better or worse than using the final weights from EquiNAS\({}_{D}\), showing that there isn't a disadvantage to training weights during search and avoiding the additional cost of a post-search training step. The use of randomized loss information in Random Static in Figure 8 shows that an informed search for architecture hyperparameters is generally useful.

On this baseline, EquiNAS\({}_{D}\) greatly outperforms a search of random architectures, demonstrating that EquiNAS\({}_{D}\) can discover the appropriate equivariance for a specific dataset over fixed or randomly selected architectures.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space. Our methods enable searching for both architecture and parameters concurrently in a single training process. This approach is more efficient than NAS approaches that only search for architectures, requiring a retraining process within the resulting architecture for evaluation.

Input: Table 1: Test error (lower is better) in percent of incorrect classifications across tasks and approaches. Statistics are aggregated over the final selected population of 5 individuals for EquiNASE and across 5 random seeds for all other methods. The best and second best average errors per task are highlighted. See Figures 5-6 in Appendix D for individual results and additional experiments.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 8: Test errors for experiments of Section 6.2.

Input: Figure 5: Test errors for experiments of Section 6.1.

Input: Figure 2: Historical parameter counts of all selected individuals for EquiNASE on RotMNIST. The architectures of the final selected population are labeled. Each history is colored by the final test accuracy, measured of each individual upon removal. For other tasks, see Figure 7 in Appendix D.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 7: Historical parameter counts of each selected individual for EquiNASE (see Figure 2 for RotMNIST).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I evaluated the claim: "We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization."

The evidence strongly supports this claim, showing detailed implementations of both evolutionary (EquiNAS_E) and differentiable (EquiNAS_D) NAS algorithms that leverage the equivariance relaxation morphism and [G]-mixed equivariant layer mechanisms. The extensive experimental results, visualizations, and performance comparisons demonstrate these algorithms successfully optimize architectures while preserving equivariance properties. The claim accurately describes what was achieved without overstating capabilities or results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization.

The review comment to be evaluated is:

**Strengths:**
1. The formulation of partial/ approximate equivariances/ invariances $-$ as a mixture over subgroups is interesting and an important problem to tackle.
2. Strong experimental results for both their learning paradigms.
3. For a person with sufficient group theory background - the paper is easy to understand.

**Weaknesses and correspnding questions/ suggestions**:
1. Lack of comparison of the evolutionary paradigm with a similar approach [1] which searches over the subspace lattice  to identify te groups to be equivariant/ invariant to.
2. It is hard to see how this extends to non-finite groups which are not compact. Would suggest that the authors comment on this.
3. While the paper is understandable for someone who has the necessary group theory background - it is almost impossible for general audience - would recommend the authors add necessary background and preliminaries to the appendix.
4. Studies in out of distribution settings are missing (for e.g. train only contains images rotated by 0-15 degrees, but test has those between 60-180 degrees) - this is will give us more insight into the weights learned.

**Minor:**
1. Please make sure the title on openreview matches the title on the paper

**References**
1. Mouli, S. Chandra, and Bruno Ribeiro. "Neural Networks for Learning Counterfactual G-Invariances from Single Environments." ICLR 2021

The evidence to be evaluated is:Neural architecture search (NAS) aims to optimize both the architecture and its parameters for a given task. Liu et al. (2018) approaches this difficult bi-level optimization by creating a large super-network containing all possible elements and continuously relaxing the discrete architectural parameters to enable search by gradient descent. Other NAS approaches include evolutionary algorithms (Real et al., 2017; Lu et al., 2019; Elsken et al., 2017) and reinforcement learning (Zoph & Le, 2017), which search over discretely represented architectures.

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

We present two NAS methods that utilize the presented mechanisms for discovering appropriate equivariance during training: Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) and Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)). Both methods optimize an architecture while learning weights, yielding a final trained network adapted to equivariances present in the training data.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm. The Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) algorithm, given in Algorithm 1, is similar to other evolutionary NAS methods such as Elsken et al. (2017) with pareto selection as in Falanti et al. (2022). A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Offspring are generated from each parent separately by mutation using the relaxation morphism. This preserves the weights of the parametrized equivariance during mutation, allowing for the continuous training of networks over evolution by inheritance from parent individuals. Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers. This constraint yields local equivariance properties for the network, as shown in Weiler and Cesa (2019) and Elsayed et al. (2020) to be empirically favorable in image classification tasks. The resulting individuals are each trained independently for a given training time, and then this process repeats. The second objective of minimizing parameter count is intended to advance efficient networks, such as those with large symmetry groups. Accuracy-based selection alone would necessarily prefer larger networks as mutation via the equivariance relaxation morphism results in two networks with identical performance but different size, the relaxed network having more parameters, until training; potentially short-term increases in validation accuracy after training would then result in the selection of individuals with more parameters. Thus, the proposed strategy of selecting both pareto-dominant and high-accuracy individuals is intended to maintain a diverse yet efficient population without succumbing to overly greedy selections too early. In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm. Our Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)) algorithm, defined in Algorithm 2, is inspired by DARTS (Liu et al., 2018) with significant changes detailed in the following paragraphs. EquiNAS\({}_{D}\) simplifies the bilevel optimization of the architecture weighting parameters \(Z\) and filter weights \(W\) into alternating independent updates, computing the gradient update for \(Z\) with the current, rather than optimal, \(\Psi\) for the current architecture encoded by \(Z\), to boost search efficiency with minimal performance loss compared to higher order approximations (Liu et al., 2018).

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance. We next study empirically the two EquiNAS methods on three datasets, one with known rotational symmetry and two with unknown but visually significant rotational and reflectional symmetry. We focus on the regular representation of groups and show experiments with reflectional and up to 4-fold rotational symmetry groups applied to image classification tasks. Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation.

As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance.

The advantages of equivariance search methods are most apparent in the Galaxy10 benchmark.

However, the \(D_{4}\) baseline fails at the Galaxy10 task, demonstrating that the same equivariant architecture can not always be naively applied. Both search methods, EquiNAS\({}_{E}\) and RPP, outperform all baseline models on Galaxy10, and by a large margin for EquiNAS\({}_{E}\).

the selected population maintains a fully equivariant network in every generation. The final selected population originates from two main lineages, one staying fully equivariant until the last generations and the other diverging from the fully equivariant network midway through, showing that training with dynamically constrained parametrizations can produce performant models.

EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

EquiNAS\({}_{D}\) achieves better test accuracy than the other comparable methods on RotMNIST and Galaxy10. Due to differences in training protocol, only comparisons of relative rankings with Table 1 are possible: baseline methods accuracies follow similar ranking patterns, suggesting the benefit of general \(C_{4}\) equivariance for RotMNIST and Galaxy10 and general \(D_{4}\) equivariance, including RPP, for ISIC.

Previous differentiable NAS works often used regularization of network size or even architecture weighting parameters themselves to encourage efficient architectures with a single highly weighted choice for each layer. However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer. The EquiNAS\({}_{D}\) algorithm avoids many known problems in differentiable NAS such as the discretization gap that occurs when searching over a continuous relaxation of a discrete architectural search space (Xie et al., 2021), such as that of EquiNAS\({}_{E}\). Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

EquiNAS\({}_{E}\) may particularly work well on tasks that benefit from local equivariance, determined by analyzing the architecture weighting parameters from first applying the more efficient EquiNAS\({}_{D}\), as well as for finding good discrete architectures within which to retrain weights, based on the ablation and random comparisons.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\). We investigate how dynamic equivariance achieved by these algorithms affects the training and performance of models across multiple image classification tasks of varying complexity and assumed symmetry, demonstrating that these techniques can search for performant architectures and weights even on noisy tasks.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization.

* Zoph & Le (2017) Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning.

Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\). For example, an image segmentation neural network should be \(T(2)\)-equivariant: shifting the input should result in the same shift in the output.

Evolutionary algorithms are optimization methods inspired by evolution in biology, where individuals in a population compete with their phenotypic traits in order to pass on their genotypic traits to offspring.

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

Group convolutional layers The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism.

The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

For both EquiNAS\({}_{E}\) and EquiNAS\({}_{D}\) experiments, we use the same backbone architecture, such that the static baselines have the same architecture across experiments. The architectures have a lifting layer followed by 7 group convolutional layers, for a total of 8 convolutional layers.

For EquiNAS\({}_{E}\), the parent selection size is 5, the training time per generation is 0.5 epochs, and the number of generations is 50 for all tasks.

To explore the symmetry discovery of EquiNAS\({}_{D}\), we apply it to six augmentations of the MNIST dataset (LeCun et al., 1998), where each augmentation applies the group actions of each group in \(\{C_{1},C_{2},C_{4},D_{1},D_{2},D_{4}\}\) respectively. The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

Shown in Figure 5, EquiNAS\({}_{E}\) outperforms EquiNAS\({}_{E}\) Random Select, showing the benefit of using informed selection to guide the relaxation of equivariance constraints over training. Additionally, EquiNAS\({}_{E}\) Retrain outperforms the Random Static baseline, showing that using compute in an informed search is more beneficial than just randomly searching the space of static architecture constraints. EquiNAS\({}_{D}\) finds competitive architectures on average and can find architectures which outperform baseline choices like architectures fully equivariant to \(C_{1}\) or \(D_{4}\). From comparing EquiNAS\({}_{D}\) Retrain to EquiNAS\({}_{D}\) results in Figure 8, retraining a resulting architecture is not consistently better or worse than using the final weights from EquiNAS\({}_{D}\), showing that there isn't a disadvantage to training weights during search and avoiding the additional cost of a post-search training step. The use of randomized loss information in Random Static in Figure 8 shows that an informed search for architecture hyperparameters is generally useful.

On this baseline, EquiNAS\({}_{D}\) greatly outperforms a search of random architectures, demonstrating that EquiNAS\({}_{D}\) can discover the appropriate equivariance for a specific dataset over fixed or randomly selected architectures.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space. Our methods enable searching for both architecture and parameters concurrently in a single training process. This approach is more efficient than NAS approaches that only search for architectures, requiring a retraining process within the resulting architecture for evaluation.

Input: Table 1: Test error (lower is better) in percent of incorrect classifications across tasks and approaches. Statistics are aggregated over the final selected population of 5 individuals for EquiNASE and across 5 random seeds for all other methods. The best and second best average errors per task are highlighted. See Figures 5-6 in Appendix D for individual results and additional experiments.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 8: Test errors for experiments of Section 6.2.

Input: Figure 5: Test errors for experiments of Section 6.1.

Input: Figure 2: Historical parameter counts of all selected individuals for EquiNASE on RotMNIST. The architectures of the final selected population are labeled. Each history is colored by the final test accuracy, measured of each individual upon removal. For other tasks, see Figure 7 in Appendix D.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 7: Historical parameter counts of each selected individual for EquiNASE (see Figure 2 for RotMNIST).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Summary: The evidence strongly supports this claim. The paper presents two clear NAS mechanisms: (1) an equivariance relaxation morphism and (2) a [G]-mixed equivariant layer, which are then implemented in evolutionary (EquiNAS_E) and differentiable (EquiNAS_D) algorithms respectively. The paper provides detailed mathematical formulations, pseudocode, and extensive experimental results demonstrating how these mechanisms enable architectural optimization for equivariance. The review comment acknowledges "strong experimental results for both their learning paradigms." The claim accurately reflects what is demonstrated without overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper we study a refinement on this criterion by adding a notion of layerwise importance, which we introduce in section 2.

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\)

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

When studying the layerwise sparsities generated by the different pruning criteria (see Figure 8), we can notice a number of interesting observations.

In Appendix C, we showed that some layers are disproportionally pruned less by different pruning criteria. In this section, we will take a look at how the sparsity of these layers evolves throughout the pruning process and compare them to 'normal' layers in both VGG16 (see Figure 9) and ResNet-18 (see Figure 10).

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the authors accurately present their study of layerwise importance as a refinement criterion. The evidence thoroughly demonstrates this through:
- Clear explanation of the layerwise importance concept and mathematical formulations
- Extensive experimental results showing how different layers are pruned differently based on importance
- Multiple visualizations across different architectures (ResNet18, VGG16) demonstrating the effects of layerwise pruning
- Detailed analysis of how certain layers (input, output, skip connections) are pruned less due to their importance

The claim is modest and well-supported by the comprehensive evidence provided, making it well-stated rather than overstated.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper we study a refinement on this criterion by adding a notion of layerwise importance, which we introduce in section 2.

The review comment to be evaluated is:

The authors propose to modify the LTH procedure by using a score that measures the layerwise importance in order to select the weights to be pruned. The authors propose a few possible scores and evaluate their performance on a set of experiments. The experiments show that in most of the cases the proposed method does not improve with respect to the standard LTH that uses magnitude in order to select the weights to be pruned. The authors also perform a comparison of lottery tickets obtained with the different scores showing that hey have a noticeable amount of overlapping connections and such connections seems to be more stable.

The evidence to be evaluated is:A weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on a global scale, rather than within the layer.

Taking these issues into account, we propose a modification of the LTR procedure in the form of assigning an importance value to each connection representing its importance within the layer. This importance score is still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the same layer. In doing so, it injects an additional notion of the layer distribution in the pruning process.

Furthermore, it is possible to view this modification as a generalization of the LTR procedure, where the purely magnitude-based approach uses the identity mapping to calculate importance values.

5:\(score_{i}\leftarrow\) layerwise-importance\((abs(\theta_{i,m}))\)

In the same context of layerwise importance calculation, Lee et al. (2021) introduced a metric called the LAMP score, which follows a model-level distortion minimization perspective by calculating the \(L_{2}\) norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets at a very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure to focus more on the layers with more connections -- and more redundancy, and as such avoid removing connections from sparser layers, thus limiting the impact on model performance. Additionally, as also noticed in Lee et al. (2021), we can view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning. To calculate a layerwise importance value for a given connection at position \(j\) in layer \(i\), we need access to the weight matrix \(\mathbf{W}_{i,:}\). * \(L_{1}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}}{\sum\mathbf{W}_{i,:}}\) * \(L_{2}\) normalization : \(importance(i,j)=\dfrac{W_{i,j}^{2}}{\sum\mathbf{W}_{i,:}^{2}}\)(Lee et al., 2021)

* Softmax : \(importance(i,j)=\dfrac{exp(W_{i,j})}{\sum exp(\mathbf{W}_{i,:})}\)

* Min-Max normalization : \(importance(i,j)=\dfrac{W_{i,j}-min(\mathbf{W}_{i,:})}{max(\mathbf{W}_{i,:})-min(\mathbf{W}_{ i,:})}\) The goal of the considered importance measures is to make weight values from different layers comparable.

Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure 1 for the plots from ResNet18 trained on TinyImageNet), we can clearly see that during the initial pruning iterations there are large differences both between different layers and between importance measures.

We find that in each case, certain layers are pruned (significantly) less than most other layers in the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet-18, these layers are both the input and output layer of the network, as well as the \(1\times 1\) convolutions used in the skip connections.

We studied a modification to the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure.

When studying the layerwise sparsities generated by the different pruning criteria (see Figure 8), we can notice a number of interesting observations.

In Appendix C, we showed that some layers are disproportionally pruned less by different pruning criteria. In this section, we will take a look at how the sparsity of these layers evolves throughout the pruning process and compare them to 'normal' layers in both VGG16 (see Figure 9) and ResNet-18 (see Figure 10).

Input: Figure 1: How the layer sparsities evolve during the LTH procedure with the ResNet-18 model trained on TinyImageNet. Layers of interest are annotated on the X-axis.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio. The x-axis is (roughly) corresponding to model depth and layers of interest are indicated.

Input: Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.

Input: Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.

Input: Figure 7: Additional curves measuring the proportion of overlapping connections between tickets

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, this claim appears well-stated (1) since it accurately describes what is presented in the paper without exaggeration. The evidence shows:

1. Clear introduction and explanation of layerwise importance concept
2. Detailed mathematical formulations of different layerwise importance measures (L1, L2, Softmax, Min-Max)
3. Extensive experimental results shown in figures demonstrating how different layers are pruned differently
4. Visual evidence through Figures 1, 8, 9, and 10 showing the evolution of layer sparsity
5. The reviewer's comment confirms this focus on layerwise importance scores

The claim modestly states they "study a refinement" without overstating the novelty or impact, and the evidence thoroughly supports this characterization.

<score>1</score>