Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose the equivariance relaxation morphism, which preserves functionality while reparametrizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the \([G]\)-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization.

The review comment to be evaluated is:

**Strength**
1. The paper is well motivated. Most G-CNN literature assume that the symmetry group of the learning task is explicitly known and perfect. The paper instead uses a equivariance-aware NAS procedure to search for the optimal architecture for (potentially) unknown and imperfect group transformations.
2. The paper is mostly well-written.
3. The experiments are detailed and thorough.

**Weakness**
1. The equivariance relaxation morphism, which relies on reparameterizing the group convolution on a subgroup, seems to be limited to discrete group. Can this be extended to continuous groups?
2. Moreover, the relaxation morphism is also restrictive in the sense that the number of the unstructured channel (c_out) cannot be adjusted after changing to a subgroup. This inevitably increases the model size as group size decreases, which is indeed the case for EquiNAS_D.
3. The clarity of the paper in section 5.1 can benefit from more explanation on the technical terms such as population, pareto selection, pareto front, etc.
4. In MNIST_rot, the review would like to see what happens when $C_8$ and $D_8$ are included in the picture.
5. If I understand correctly, the reported EquiNAS_E result on Galaxy10 in Table 1 might be misleading. Figure 4(a) shows that only one NAS output in the end achieves significantly smaller test error compared to all the remaining results.
6. Are the models involved in EquiNAS_D significantly larger compared to the baseline? After all, a linear combination of layers equivariant to different groups is taken. Does that mean the model is slow to train and (especially) test?

The evidence to be evaluated is:The main contributions of Basu et al. (2021) and Agrawal & Ostrowski (2022) are similar to our proposed equivariant relaxation morphism. Basu et al. (2021) also utilizes subgroup decomposition but instead algorithmically builds up equivariances from smaller groups, while our work focuses on relaxing existing constraints.

The main contributions of Wang et al. (2022) and Finzi et al. (2021) are similar to our proposed \([G]\)-mixed equivariant layer. Wang et al. (2022) also uses a weighted sum of filters, but uses the same group for each filter and defines the weights over the domain of group elements. Finzi et al. (2021) uses an equivariant layer in parallel to a linear layer with weighted regularization, thus only using two layers in parallel and weighting them by regularization rather than parametrization.

In more diverse approaches, Zhou et al. (2020) and Yeh et al. (2022) represent symmetry-inducing weight sharing via learnable matrices. Romero & Lohit (2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances for each layer.

For discrete group \(G\), the \(l^{\text{th}}\)\(G\)-equivariant group convolutional layer (Cohen & Welling, 2016) of a group convolutional neural network (G-CNN) convolves1 the feature map \(f\colon G\to\mathbb{R}^{C_{l-1}}\) from the previous layer with a filter with kernel size \(k\) represented as learnable parameters \(\psi\colon G\to\mathbb{R}^{C_{l}\times C_{l-1}}\).

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

We define the _equivariance relaxation morphism_ from \(G\) to \(G^{\prime}\) as the reparametrization of \(\psi\) as \(\tilde{\psi}\) (Eq. 4) and reshaping of \(f\) as \(\tilde{f}\) (Eq. 3). We will show that the new layer, \([\tilde{f}\star_{G^{\prime}}\tilde{\psi}]_{(d,t)}(g^{\prime})\), is equivalent to \([f\star_{G}\psi]_{d}(g^{\prime}t)\) down to reshaping.

Thus, the convolution of \(\tilde{f}\) with \(\tilde{\psi}\) is equivariant to \(G\) but parametrized as a \(G^{\prime}\)-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing a \(G^{\prime}\)-equivariant layer with a pre-trained prior of equivariance to \(G\), maintaining any previous training.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm.

A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers.

In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm.

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance.

Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation. The \(p4\) group consists of discrete translations and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes C_{4}\). The \(p4m\) group consists of discrete translations, reflections, and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes D_{4}\). As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance. This is a notably different paradigm than other works that equate parameter counts across architectures with different equivariance properties.

These are implemented by initializing all layers to be constrained to the prior symmetry group, then using the equivariance relaxation morphism on each layer. EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer.

Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

Networks with more equivariance constraints tend to learn faster, but equivariance relaxation may yield large gradients for newly unconstrained parameters and thus fast increases in performance.

Beyond NAS, the equivariance relaxation morphism could be used in other applications such as fine-tuning and distillation.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\).

The proposed mechanisms and algorithms are extendable beyond vision tasks to any architecture with parametrized equivariance to any discrete group.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In _International Conference on Learning Representations_, 2020.

A _symmetry group_\(G\) is a set of such mappings along with a binary operation \(\cdot\): \(G\times G\to G\), known as the _group product_, that satisfies axioms for closure, associativity, the identity, and the inverse (Herstein, 2006). A group \(G\) acts on a set \(\mathcal{X}\) via the _group action_ \(\cdot:G\times\mathcal{X}\rightarrow\mathcal{X},\ (g,x)\mapsto g.x\) that satisfies axioms for identity and compatibility: \(\mathcal{X}\) is called a _\(G\)-space_. Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\).

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism. With the preliminaries given in Section 4.1 and the case of \(G^{\prime}=T(2),\tilde{f}\) and \(\tilde{\psi}\) are computed such that \(\tilde{f}_{(c,s)}(g^{\prime})\coloneqq f_{c}(g^{\prime}s)\) and \(\tilde{\psi}_{(d,t),(c,s)}(g^{\prime})\coloneqq\psi_{d,c}(t^{-1}g^{\prime}s)\) for each \(g^{\prime}\in T(2)\), \(c\in[C_{l-1}]\), \(s,t\in R\), and \(d\in[C_{l}]\). Let \(S_{G}:=|R|\). The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

Equivariance relaxation morphism To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I've carefully reviewed the claim and evidence. The claim appears to accurately describe the two key technical contributions - the equivariance relaxation morphism and [G]-mixed equivariant layer. The evidence provides extensive mathematical formulation, implementation details, and experimental validation demonstrating both mechanisms work as claimed. The review comment raises some valid limitations regarding discrete groups and model size increases, but these don't invalidate the core claims. The figures show clear visualization of how these mechanisms operate and their effects across different architectures. The claim appropriately describes the functionality without overstating capabilities or novelty.

<score>1</score>

Justification: The claim is well-stated because:
1. The mathematical formulations and implementation details directly support the described functionality
2. Experimental results validate that both mechanisms work as described
3. The language used is precise and technically accurate without exaggeration
4. Limitations are acknowledged in the broader work
5. The evidence thoroughly demonstrates both mechanisms achieving their stated purposes


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose the equivariance relaxation morphism, which preserves functionality while reparametrizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the \([G]\)-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization.

The review comment to be evaluated is:

## Strengths
- Equivariant networks have more hyperparameters to tune due to the choice of symmetry group.  For example, in E(2)-CNN (https://arxiv.org/abs/1911.08251), the authors experiment with symmetry groups $C_n$ and $D_n$ for many values of n before finding n=12 to work best.  This paper proposes a method for automatically determining the correct level of symmetry.
- Moreover, on many problems end-to-end strict symmetry is not desirable if the symmetry in the domain is only approximate or if symmetry constraints interfere with optimization.  The proposed method adds to the relatively few methods which learn relaxed symmetry constraints and adapt to the level of symmetry in the data.
- I found the equivariance relaxation morphism to be a simple and effective idea for relaxing symmetry constraints in networks on a per-layer basis.
- The [G]-mixed equivariant layer also seems like a reasonable idea for selecting the degree of equivariance by differentiably choosing weights over kernels with different levels of equivariance.  In theory this allows the model to learn a specific equivariance or an approximate equivariance which is in-between.  It also allows the network to reduce the symmetry constraint in later layers, which is what is done effectively by CNNs in practice by downsampling with stride.
- Experimental results on the learned subgroup weights (Fig. 2) reveal interesting trends showing that the networks do select for higher equivariance in early layers and lower equivariance in later layers which matches what has found to be effective in practice for equivariant networks.

## Weaknesses / Questions
- I am not sure of the framing of the method.  I am not an expert on NAS and am open to correction, but it seems to me both proposed methods find a trained network, not an architecture, since they are training the network weights and selecting an architecture at the same time.  The final architecture may not have great performance if trained from scratch.  In this sense, it seems the method is more a special training procedure than NAS method.   EquiNAS_D, in particular, is simply an architecture in which some weights are optimized alternatingly.
- Given the evolutionary algorithm starts with a single network, how do we know to what extent its evolution is guided by architecture versus the specific weights in the mutated networks?  In experiments, I think both population level variance and initialization variance should be accounted for.
- One of the biggest issues is the fact that the relaxation morphism increases the number of parameters making it difficult to distinguish performance increases are from relaxing constrains or increasing parameter count.  The authors do discuss this issue and suggest their optimization strategy over both parameter count and accuracy helps.  It would be better, however, if the number of parameters could be preserved by the relaxation.
- The experimental results are not completely convincing of when this method would add significant value over simply doing hyperparameter search over equivariant networks.  In Table 1, an equivariant baseline is best on rotMNIST and ISIC, and a non-rotationally equivariant network with equivariant initialization is best for Galaxy10.  In table 2, EquiNAS_D does do best for rotMNIST, but the score is much lower than the best values in E(2)-CNN after full tuning of the best equivariant method.  In Galaxy10, EquiNAS_D does outperform, but with very high variance.
- I have some issues with the clarity of the description of 4.2.  The condition that all groups in $[G]$ be subgroups or supergroups of all other groups in $[G]$ does not appear to be met by the expermental example since $D_2$ is not a subgroup or supergroup of $C_4$.  It's also not clear to me why this condition is necessary.  It is a fairly strict condition, although I think it would include useful cases.  I am also a bit unclear on the form of the input signal $f$.  In order to be convolved with $\tilde{\Psi}$ and have $G$-equivariance for each $G$ in $[G]$, $f$ should be defined over a supergroup of all $G \in [G]$.  Is this the case?   I am correct that if $z_G$ is a one-hot vector on the group $G$, then the operation is $G$-equivariant?

## Questions
- If the evolution algorithm can only mutate in the direction of symmetry constrain relaxation, doesn't this bias the algorithm towards relaxing all the symmetry constraints?
- Does this work have a relation to diffstride (https://arxiv.org/abs/2202.01653) in that stride is typically a hyperparameter that corresonds to a choice of the subgroup of the translation group and can be shown to differentiably optimized?
- After equivariance is broken in a given layer, there isn't much point in principle to persevering it downstream.  Are mutations automatically applied to all following layers or did your method discover this?  If so, I think that is interesting enough to be considered a strength.

## Minor
- In Eqn 1, if you reverse the order of $f$ and $\Psi$ and drop the sum over c and indices $c,d$, the equation can be simplified using matrix multiplication.
- 4.1, para 2, "neutral element" -> "identity element"
- 4.1 may benefit from writing as a proposition
- 4.1, Para 3, Line 5,  $h \in G'$ -> $h' \in G'$
- Sec 5., Line 4, typo "optimizffe"

The evidence to be evaluated is:The main contributions of Basu et al. (2021) and Agrawal & Ostrowski (2022) are similar to our proposed equivariant relaxation morphism. Basu et al. (2021) also utilizes subgroup decomposition but instead algorithmically builds up equivariances from smaller groups, while our work focuses on relaxing existing constraints.

The main contributions of Wang et al. (2022) and Finzi et al. (2021) are similar to our proposed \([G]\)-mixed equivariant layer. Wang et al. (2022) also uses a weighted sum of filters, but uses the same group for each filter and defines the weights over the domain of group elements. Finzi et al. (2021) uses an equivariant layer in parallel to a linear layer with weighted regularization, thus only using two layers in parallel and weighting them by regularization rather than parametrization.

In more diverse approaches, Zhou et al. (2020) and Yeh et al. (2022) represent symmetry-inducing weight sharing via learnable matrices. Romero & Lohit (2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances for each layer.

For discrete group \(G\), the \(l^{\text{th}}\)\(G\)-equivariant group convolutional layer (Cohen & Welling, 2016) of a group convolutional neural network (G-CNN) convolves1 the feature map \(f\colon G\to\mathbb{R}^{C_{l-1}}\) from the previous layer with a filter with kernel size \(k\) represented as learnable parameters \(\psi\colon G\to\mathbb{R}^{C_{l}\times C_{l-1}}\).

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

We define the _equivariance relaxation morphism_ from \(G\) to \(G^{\prime}\) as the reparametrization of \(\psi\) as \(\tilde{\psi}\) (Eq. 4) and reshaping of \(f\) as \(\tilde{f}\) (Eq. 3). We will show that the new layer, \([\tilde{f}\star_{G^{\prime}}\tilde{\psi}]_{(d,t)}(g^{\prime})\), is equivalent to \([f\star_{G}\psi]_{d}(g^{\prime}t)\) down to reshaping.

Thus, the convolution of \(\tilde{f}\) with \(\tilde{\psi}\) is equivariant to \(G\) but parametrized as a \(G^{\prime}\)-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing a \(G^{\prime}\)-equivariant layer with a pre-trained prior of equivariance to \(G\), maintaining any previous training.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm.

A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers.

In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm.

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance.

Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation. The \(p4\) group consists of discrete translations and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes C_{4}\). The \(p4m\) group consists of discrete translations, reflections, and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes D_{4}\). As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance. This is a notably different paradigm than other works that equate parameter counts across architectures with different equivariance properties.

These are implemented by initializing all layers to be constrained to the prior symmetry group, then using the equivariance relaxation morphism on each layer. EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer.

Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

Networks with more equivariance constraints tend to learn faster, but equivariance relaxation may yield large gradients for newly unconstrained parameters and thus fast increases in performance.

Beyond NAS, the equivariance relaxation morphism could be used in other applications such as fine-tuning and distillation.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\).

The proposed mechanisms and algorithms are extendable beyond vision tasks to any architecture with parametrized equivariance to any discrete group.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In _International Conference on Learning Representations_, 2020.

A _symmetry group_\(G\) is a set of such mappings along with a binary operation \(\cdot\): \(G\times G\to G\), known as the _group product_, that satisfies axioms for closure, associativity, the identity, and the inverse (Herstein, 2006). A group \(G\) acts on a set \(\mathcal{X}\) via the _group action_ \(\cdot:G\times\mathcal{X}\rightarrow\mathcal{X},\ (g,x)\mapsto g.x\) that satisfies axioms for identity and compatibility: \(\mathcal{X}\) is called a _\(G\)-space_. Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\).

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism. With the preliminaries given in Section 4.1 and the case of \(G^{\prime}=T(2),\tilde{f}\) and \(\tilde{\psi}\) are computed such that \(\tilde{f}_{(c,s)}(g^{\prime})\coloneqq f_{c}(g^{\prime}s)\) and \(\tilde{\psi}_{(d,t),(c,s)}(g^{\prime})\coloneqq\psi_{d,c}(t^{-1}g^{\prime}s)\) for each \(g^{\prime}\in T(2)\), \(c\in[C_{l-1}]\), \(s,t\in R\), and \(d\in[C_{l}]\). Let \(S_{G}:=|R|\). The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

Equivariance relaxation morphism To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim is largely well-stated but slightly overstated in parts. The evidence thoroughly documents both proposed mechanisms (equivariance relaxation morphism and [G]-mixed equivariant layer) and demonstrates their implementation and effectiveness. However, while the claim implies these are completely novel approaches, the review comments and evidence note similarities to prior work by Basu et al., Wang et al., and others. The paper's key contribution appears to be in adapting and combining existing concepts in a novel way rather than proposing entirely new mechanisms. The figures provide strong empirical support showing how these methods work in practice.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose the equivariance relaxation morphism, which preserves functionality while reparametrizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the \([G]\)-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization.

The review comment to be evaluated is:

### Strengths:
1. The problem of finding the correct equivariance that can exploit symmetries in the data is important for the development of general-purpose NAS methods.
2. The proposed approaches are simple and natural procedures.
3. The writing is fairly clear and the experimental results are interesting.
4. Code will be released upon acceptance.

### Weaknesses:
1. The experimental evaluation is somewhat limited. Given the fairly direct extension of prior work (morphisms and weight-sharing) to actually developing the search algorithms, I would expect more emphasis to be placed on experimental depth. This could either come in the form of evaluation on interesting benchmarks where we might expect finding symmetries to outperform regular CNNs (e.g. Tu et al. (2022)) and/or more detailed analysis on more involved search spaces of whether the proposed methods are recovering the “right” symmetry for the dataset, even if synthetic.
2. The authors do not consider the random search baseline for their method. More generally, it would be useful to follow the NAS reproducibility checklist here (Lindauer & Hutter, 2020).

### References:
- Lindauer, Hutter. Best practices for scientific research on neural architecture search. JMLR 2020.
- Tu, Roberts, Khodak, Shen, Sala, Talwalkar. NAS-Bench-360: Benchmarking neural architecture search on diverse tasks. NeurIPS 2022.

The evidence to be evaluated is:The main contributions of Basu et al. (2021) and Agrawal & Ostrowski (2022) are similar to our proposed equivariant relaxation morphism. Basu et al. (2021) also utilizes subgroup decomposition but instead algorithmically builds up equivariances from smaller groups, while our work focuses on relaxing existing constraints.

The main contributions of Wang et al. (2022) and Finzi et al. (2021) are similar to our proposed \([G]\)-mixed equivariant layer. Wang et al. (2022) also uses a weighted sum of filters, but uses the same group for each filter and defines the weights over the domain of group elements. Finzi et al. (2021) uses an equivariant layer in parallel to a linear layer with weighted regularization, thus only using two layers in parallel and weighting them by regularization rather than parametrization.

In more diverse approaches, Zhou et al. (2020) and Yeh et al. (2022) represent symmetry-inducing weight sharing via learnable matrices. Romero & Lohit (2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances for each layer.

For discrete group \(G\), the \(l^{\text{th}}\)\(G\)-equivariant group convolutional layer (Cohen & Welling, 2016) of a group convolutional neural network (G-CNN) convolves1 the feature map \(f\colon G\to\mathbb{R}^{C_{l-1}}\) from the previous layer with a filter with kernel size \(k\) represented as learnable parameters \(\psi\colon G\to\mathbb{R}^{C_{l}\times C_{l-1}}\).

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

We define the _equivariance relaxation morphism_ from \(G\) to \(G^{\prime}\) as the reparametrization of \(\psi\) as \(\tilde{\psi}\) (Eq. 4) and reshaping of \(f\) as \(\tilde{f}\) (Eq. 3). We will show that the new layer, \([\tilde{f}\star_{G^{\prime}}\tilde{\psi}]_{(d,t)}(g^{\prime})\), is equivalent to \([f\star_{G}\psi]_{d}(g^{\prime}t)\) down to reshaping.

Thus, the convolution of \(\tilde{f}\) with \(\tilde{\psi}\) is equivariant to \(G\) but parametrized as a \(G^{\prime}\)-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing a \(G^{\prime}\)-equivariant layer with a pre-trained prior of equivariance to \(G\), maintaining any previous training.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm.

A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers.

In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm.

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance.

Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation. The \(p4\) group consists of discrete translations and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes C_{4}\). The \(p4m\) group consists of discrete translations, reflections, and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes D_{4}\). As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance. This is a notably different paradigm than other works that equate parameter counts across architectures with different equivariance properties.

These are implemented by initializing all layers to be constrained to the prior symmetry group, then using the equivariance relaxation morphism on each layer. EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer.

Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

Networks with more equivariance constraints tend to learn faster, but equivariance relaxation may yield large gradients for newly unconstrained parameters and thus fast increases in performance.

Beyond NAS, the equivariance relaxation morphism could be used in other applications such as fine-tuning and distillation.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\).

The proposed mechanisms and algorithms are extendable beyond vision tasks to any architecture with parametrized equivariance to any discrete group.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In _International Conference on Learning Representations_, 2020.

A _symmetry group_\(G\) is a set of such mappings along with a binary operation \(\cdot\): \(G\times G\to G\), known as the _group product_, that satisfies axioms for closure, associativity, the identity, and the inverse (Herstein, 2006). A group \(G\) acts on a set \(\mathcal{X}\) via the _group action_ \(\cdot:G\times\mathcal{X}\rightarrow\mathcal{X},\ (g,x)\mapsto g.x\) that satisfies axioms for identity and compatibility: \(\mathcal{X}\) is called a _\(G\)-space_. Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\).

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism. With the preliminaries given in Section 4.1 and the case of \(G^{\prime}=T(2),\tilde{f}\) and \(\tilde{\psi}\) are computed such that \(\tilde{f}_{(c,s)}(g^{\prime})\coloneqq f_{c}(g^{\prime}s)\) and \(\tilde{\psi}_{(d,t),(c,s)}(g^{\prime})\coloneqq\psi_{d,c}(t^{-1}g^{\prime}s)\) for each \(g^{\prime}\in T(2)\), \(c\in[C_{l-1}]\), \(s,t\in R\), and \(d\in[C_{l}]\). Let \(S_{G}:=|R|\). The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

Equivariance relaxation morphism To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find the claim to be generally well-supported but slightly overstated in some aspects. The evidence thoroughly demonstrates both mechanisms (equivariance relaxation morphism and [G]-mixed equivariant layer) with detailed mathematical formulations, visualizations, and experimental results. However, the claim implies these are fully novel contributions when the evidence shows they build upon existing work (e.g., Basu et al. 2021, Wang et al. 2022). The "preserves functionality" part is well-demonstrated through the mathematical proofs and implementations. The reviewer also notes the approaches are "simple and natural procedures" rather than breakthrough innovations.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose the equivariance relaxation morphism, which preserves functionality while reparametrizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the \([G]\)-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization.

The review comment to be evaluated is:

**Strengths:**
1. The formulation of partial/ approximate equivariances/ invariances $-$ as a mixture over subgroups is interesting and an important problem to tackle.
2. Strong experimental results for both their learning paradigms.
3. For a person with sufficient group theory background - the paper is easy to understand.

**Weaknesses and correspnding questions/ suggestions**:
1. Lack of comparison of the evolutionary paradigm with a similar approach [1] which searches over the subspace lattice  to identify te groups to be equivariant/ invariant to.
2. It is hard to see how this extends to non-finite groups which are not compact. Would suggest that the authors comment on this.
3. While the paper is understandable for someone who has the necessary group theory background - it is almost impossible for general audience - would recommend the authors add necessary background and preliminaries to the appendix.
4. Studies in out of distribution settings are missing (for e.g. train only contains images rotated by 0-15 degrees, but test has those between 60-180 degrees) - this is will give us more insight into the weights learned.

**Minor:**
1. Please make sure the title on openreview matches the title on the paper

**References**
1. Mouli, S. Chandra, and Bruno Ribeiro. "Neural Networks for Learning Counterfactual G-Invariances from Single Environments." ICLR 2021

The evidence to be evaluated is:The main contributions of Basu et al. (2021) and Agrawal & Ostrowski (2022) are similar to our proposed equivariant relaxation morphism. Basu et al. (2021) also utilizes subgroup decomposition but instead algorithmically builds up equivariances from smaller groups, while our work focuses on relaxing existing constraints.

The main contributions of Wang et al. (2022) and Finzi et al. (2021) are similar to our proposed \([G]\)-mixed equivariant layer. Wang et al. (2022) also uses a weighted sum of filters, but uses the same group for each filter and defines the weights over the domain of group elements. Finzi et al. (2021) uses an equivariant layer in parallel to a linear layer with weighted regularization, thus only using two layers in parallel and weighting them by regularization rather than parametrization.

In more diverse approaches, Zhou et al. (2020) and Yeh et al. (2022) represent symmetry-inducing weight sharing via learnable matrices. Romero & Lohit (2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances for each layer.

For discrete group \(G\), the \(l^{\text{th}}\)\(G\)-equivariant group convolutional layer (Cohen & Welling, 2016) of a group convolutional neural network (G-CNN) convolves1 the feature map \(f\colon G\to\mathbb{R}^{C_{l-1}}\) from the previous layer with a filter with kernel size \(k\) represented as learnable parameters \(\psi\colon G\to\mathbb{R}^{C_{l}\times C_{l-1}}\).

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

We define the _equivariance relaxation morphism_ from \(G\) to \(G^{\prime}\) as the reparametrization of \(\psi\) as \(\tilde{\psi}\) (Eq. 4) and reshaping of \(f\) as \(\tilde{f}\) (Eq. 3). We will show that the new layer, \([\tilde{f}\star_{G^{\prime}}\tilde{\psi}]_{(d,t)}(g^{\prime})\), is equivalent to \([f\star_{G}\psi]_{d}(g^{\prime}t)\) down to reshaping.

Thus, the convolution of \(\tilde{f}\) with \(\tilde{\psi}\) is equivariant to \(G\) but parametrized as a \(G^{\prime}\)-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing a \(G^{\prime}\)-equivariant layer with a pre-trained prior of equivariance to \(G\), maintaining any previous training.

The equivariance relaxation morphism generalizes this concept to any subgroup.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between a group and a subgroup. For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers. Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism presented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm.

A population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters.

Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers.

In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm.

In our formulation, this is not necessary as any mixed operation can be equivalently expressed as a single layer equivariant to any group \(G^{\prime}\) that is a common subgroup to all groups of the mixed operation (Eq. 11): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance.

Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation. The \(p4\) group consists of discrete translations and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes C_{4}\). The \(p4m\) group consists of discrete translations, reflections, and multiples of \(90^{\circ}\) rotations and may be represented as \(T(2)\rtimes D_{4}\). As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively.

We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance. This is a notably different paradigm than other works that equate parameter counts across architectures with different equivariance properties.

These are implemented by initializing all layers to be constrained to the prior symmetry group, then using the equivariance relaxation morphism on each layer. EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to \(D_{4}\) or \(C_{4}\) equivariance, without such regularization.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer.

Towards searching for discretely equivariant layers using the \([G]\)-mixed equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and straight-through estimation (Li et al., 2022) to avoid the discretization gap and thus may be effective for this application.

Networks with more equivariance constraints tend to learn faster, but equivariance relaxation may yield large gradients for newly unconstrained parameters and thus fast increases in performance.

Beyond NAS, the equivariance relaxation morphism could be used in other applications such as fine-tuning and distillation.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\).

The proposed mechanisms and algorithms are extendable beyond vision tasks to any architecture with parametrized equivariance to any discrete group.

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In _International Conference on Learning Representations_, 2020.

A _symmetry group_\(G\) is a set of such mappings along with a binary operation \(\cdot\): \(G\times G\to G\), known as the _group product_, that satisfies axioms for closure, associativity, the identity, and the inverse (Herstein, 2006). A group \(G\) acts on a set \(\mathcal{X}\) via the _group action_ \(\cdot:G\times\mathcal{X}\rightarrow\mathcal{X},\ (g,x)\mapsto g.x\) that satisfies axioms for identity and compatibility: \(\mathcal{X}\) is called a _\(G\)-space_. Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping \(h\colon\mathcal{X}\rightarrow\mathcal{Z}\) between two \(G\)-spaces is \(G\)-equivariant if for all \(g\in G\) and \(x\in\mathcal{X}\) we have: \(h(g.x)=g.h(x)\).

In this case, each individual is a neural network, with a genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as a parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children diverge from their siblings and parent, so that the next set of parents may be selected and this process repeats.

The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism. With the preliminaries given in Section 4.1 and the case of \(G^{\prime}=T(2),\tilde{f}\) and \(\tilde{\psi}\) are computed such that \(\tilde{f}_{(c,s)}(g^{\prime})\coloneqq f_{c}(g^{\prime}s)\) and \(\tilde{\psi}_{(d,t),(c,s)}(g^{\prime})\coloneqq\psi_{d,c}(t^{-1}g^{\prime}s)\) for each \(g^{\prime}\in T(2)\), \(c\in[C_{l-1}]\), \(s,t\in R\), and \(d\in[C_{l}]\). Let \(S_{G}:=|R|\). The learnable parameters of the \(G_{l}\)-equivariant \(l^{\text{th}}\) layer with \(C_{l}\) output channels, corresponding to \(\psi\), are stored as a tensor of size \(C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\). The filter transformation expands this filter tensor by performing the action of each \(r\in R\) on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size \(C_{l}\times S_{G_{l}}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}\), which is reshaped to \(C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}\). The input tensor to the \(l^{\text{th}}\) layer, corresponding to \(f\), is in the shape of \(B\times C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1}\), which is reshaped to \(B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}\) and convolved with the expanded filter.

Equivariance relaxation morphism To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

The resulting architecture dynamics of this experiment are shown in Figure 4, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the \([G]\)-mixed equivariant layer. This can explain why the EquiNAS\({}_{D}\) results are closer to random baselines than the EquiNAS\({}_{E}\) results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space.

Input: Figure 4: EquiNASD on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, I find the claim to be well-stated (score of 1). The evidence comprehensively supports both proposed mechanisms:

1. The equivariance relaxation morphism is thoroughly detailed mathematically and demonstrated through implementation details and visualizations (Figure 1A) showing how it preserves functionality while relaxing constraints.

2. The [G]-mixed equivariant layer is well-supported through mathematical formulation, implementation specifics, and extensive experimental results visualized in Figures 3,4,10-12 showing how it enables mixing of different equivariance constraints.

The reviewer comments note the strong experimental results and theoretical foundation, with only minor concerns about generalization and accessibility. The claim accurately reflects the capabilities and scope of the work without overstatement.

<score>1</score>