Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# A General Theory of Correct, Incorrect, and Extrinsic Equivariance

Dian Wang\({}^{1}\)   Xupeng Zhu\({}^{1}\)   Jung Yeon Park\({}^{1}\)   Mingxi Jia\({}^{2}\)   Guanang Su\({}^{3}\)

\({}^{1}\)Northeastern University  \({}^{2}\)Brown University  \({}^{3}\)University of Minnesota

{wang.dian,zhu.xup,park.jungy,r.platt,r.walters}@northeastern.edu

mingxi_jia@brown.edu su000265@umn.edu

Work done as students at Northeastern University

###### Abstract

Although equivariant machine learning has proven effective at many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in an equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially in the domain. In this work, we present a general theory for such a situation. We propose pointwise definitions of correct, incorrect, and extrinsic equivariance, which allow us to quantify continuously the degree of each type of equivariance a function displays. We then study the impact of various degrees of incorrect or extrinsic symmetry on model error. We prove error lower bounds for invariant or equivariant networks in classification or regression settings with partially incorrect symmetry. We also analyze the potentially harmful effects of extrinsic equivariance. Experiments validate these results in three different environments.

## 1 Introduction

Equivariant neural networks [9, 10] have proven to be an effective way to improve generalization and sample efficiency in many machine learning tasks. This is accomplished by encoding task-level symmetry into the structure of the network architecture so that the model does not need to explicitly learn the symmetry from the data. However, encoding a fixed type of symmetry like this can be limiting when the model symmetry does not exactly match the symmetry of the underlying function being modeled, i.e., when there is a symmetry mismatch. For example, consider the digit image classification task. Is it helpful to model this problem using a model that is invariant to 180-degree rotation of the image? For some digits, the label is invariant (e.g., \(\mathbf{0}\) and \(\mathbf{8}\)). However, for other digits, the label changes under rotation (e.g., \(\mathbf{6}\) and \(\mathbf{9}\)), suggesting that a rotationally symmetric model would be inappropriate here. However, recent work [58] suggests that this is not necessarily the case - symmetric models are sometimes helpful even when a symmetry mismatch exists between the problem and model. This raises the question - do the advantages obtained by using a symmetric model outweigh the errors introduced by the symmetry mismatch?

Figure 1: An example of correct, incorrect, and extrinsic equivariance. The ground truth function \(f(x)\) is shown in black and its probability density function \(p(x)\) is shown in orange. If we model \(f(x)\) using a \(G\)-invariant network where \(G\) is a reflection group that negates \(x\), different \(f(x)\) and \(p(x)\) will lead to correct, incorrect, and extrinsic equivariance. See Section 3 for details.

This paper makes four main contributions towards this problem. First, this paper extends the definitions for types of model symmetry with respect to true symmetry introduced in Wang et al. [58]. They classify models as having _correct_, _incorrect_, or _extrinsic equivariance_ (see Figure 1), where correct means the ground truth function has the same symmetry as the equivariant model, incorrect means the ground truth function disagrees with the symmetry in the model, and extrinsic means the model symmetry transforms in-distribution data to out-of-distribution data. We generalize this system into a continuum of equivariance types to reflect the fact that a single task may have different proportions of correct, incorrect, and extrinsic symmetry across its domain. For example, in the digit classification task, \(\mathbf{0}\) has correct equivariance, \(\mathbf{6}\) has incorrect equivariance, and \(\mathbf{4}\) has extrinsic equivariance.

Our second contribution is to introduce an analytical lower bound on model error in classification tasks resulting from incorrect model symmetry. This result can help guide model selection by quantifying error resulting from incorrect equivariance constraints. Our result generalizes that of Wang et al. [58] by removing the simplifying assumption that data density over the domain is group invariant. We prove the minimum error of an invariant classifier can be realized by assigning all data points in the same group orbit the label with the majority of the data density (Theorem 4.3).

Our third contribution is to develop new lower bounds on the \(L_{2}\) error for regression tasks in terms of the variance of the function to be modeled over the orbit of the symmetry group. Like our classification bound, this bound can assist in model selection in situations with symmetry mismatch.

Fourth, in contrast to Wang et al. [58] who show benefits of extrinsic equivariance, we theoretically demonstrate its potential harm. We perform experiments documenting the error rate across the correct-extrinsic continuum. Finally, we perform empirical studies illustrating the ideas of the paper and showing that the lower bounds obtained in our analysis appear tight in practice. This suggests our analysis can assist practitioners select symmetry groups appropriate for a given problem setting. Our code is available at [https://github.com/pointW/ext_theory](https://github.com/pointW/ext_theory).

## 2 Related Work

**Equivariant Learning.** Originally used for exploiting symmetry in image domains [9; 10], equivariant learning has been very successful in various tasks including molecular dynamics [2; 4], particle physics [6], fluid dynamics [59], trajectory prediction [53], pose estimation [31; 26; 32], shape completion [7], robotics [48; 65; 21; 55; 49; 41; 23; 22; 45] and reinforcement learning [52; 54; 56; 39; 63]. However, most prior work assumes that the symmetry of the ground truth function is perfectly known and matches the model symmetry. Wang et al. [58] go further and define correct, incorrect, and extrinsic equivariance to classify the relationship between model symmetry and domain symmetry. However, they do not discuss the possible combinations of the three categories, and limit their theory to a compact group and invariant classification. Our work extends [58] and allows for a continuum of equivariance types and analyzes error bounds in a more general setup.

**Symmetric Representation Learning.** Various works have proposed learning symmetric representations, using transforming autoencoders [20], restricted Boltzmann machines [50], and equivariant descriptors [47]. In particular, [30] shows that convolutional neural networks implicitly learn representations that are equivariant to rotations, flips, and translations, suggesting that symmetric representations are important inductive biases. Other works have considered learning symmetry-aware features using disentanglement [43], projection mapping [25], equivariance constraints [35], separation into invariant and equivariant parts [61] or subgroups [34]. Park et al. [42] propose learning a symmetric encoder that maps to equivariant features and Dangovski et al. [13] learn features that are sensitive and insensitive to different group representations. Other works assume no prior knowledge of symmetry and learn it from data [3; 64; 14; 40]. In particular, Moskalev et al. [40] estimate the difference between the true latent symmetry and the learned symmetry. Similarly, our work considers a gap between the true symmetry and model symmetry and theoretically analyze its effects on error.

**Theory of Equivariant Learning.** There are several lines of work on the theory of equivariant learning. Kondor and Trivedi [27] prove that convolutions are sufficient and necessary for equivariance of scalar fields on compact groups, later generalized to the steerable case by Cohen et al. [11]. Certain equivariant networks have been proved to be universal in that such networks can approximate any \(G\)-equivariant function [36; 62]. Another line of work has considered equivariant networks in terms of generalization error. Abu-Mostafa [1] show that an invariant model has a VC dimension less than or equal to that of a non-equivariant model. Other works studied the generalization error of invariant classifiers by decomposing the input space [51, 46]. Elesedy and Zaidi [17] quantify a generalization benefit for equivariant linear models using the notion of symmetric and anti-symmetric spaces. A PAC Bayes approach was used for generalization bounds of equivariant models [5, 33]. Our work is complimentary to these and quantifies approximation error for equivariant model classes.

**Data Augmentation.** Some methods use data augmentation [29, 28] to encourage the network to learn invariance with respect to transformations defined by the augmentation function [8]. Recent works have explored class-specific [19, 44] and instance-specific [38] data augmentation methods to further boost training by avoiding the potential error caused by a uniform augmentation function. Those methods can be viewed as applying data augmentation where pointwise correct or extrinsic invariance exist, while avoiding incorrect invariance.

## 3 Preliminaries

Problem Statement.Consider a function \(f\colon X\to Y\). Let \(p:X\to\mathbb{R}\) be the probability density function of the domain \(X\). We assume that there is no distribution shift during testing, i.e., \(p\) is always the underlying distribution during training and testing. The goal for a model class \(\{h:X\to Y\}\) is to fit the function \(f\) by minimizing an error function \(\operatorname{err}(h)\). We assume the model class \(\{h\}\) is arbitrarily expressive except that it is constrained to be equivariant with respect to a group \(G\). Let \(\mathbb{1}\) be an indicator function that equals to 1 if the condition is satisfied and 0 otherwise. In classification, \(\operatorname{err}(h)\) is the classification error rate; for regression tasks, the error function is a \(L_{2}\) norm function,

\[\operatorname{err}_{\mathrm{cls}}(h)=\mathbb{E}_{x\sim p}[\mathbb{1}\left(f(x) \neq h(x)\right)],\qquad\operatorname{err}_{\mathrm{reg}}(h)=\mathbb{E}_{x \sim p}[||h(x)-f(x)||_{2}^{2}]. \tag{1}\]

**Equivariant Function.** A function \(f:X\to Y\) is equivariant with respect to a symmetry group \(G\) if it commutes with the group transformation \(g\in G\), \(\hat{f}(gx)=gf(x)\), where \(g\) acts on \(x\in X\) through the representation \(\rho_{X}(g)\); \(g\) acts on \(y\in Y\) through the representation \(\rho_{Y}(g)\).

### Correct, Incorrect, and Extrinsic Equivariance.

Consider a model \(h\) which is equivariant with respect to a group \(G\). Since real-world data rarely exactly conforms to model assumptions, in practice there may often be a gap between the symmetry of the model and the ground truth function. Wang et al. [58] propose a three-way classification which describes the relationship between the symmetry of \(f\) and the symmetry of \(h\). In this system, \(h\) has correct equivariance, incorrect equivariance, or extrinsic equivariance with respect to \(f\).

**Definition 3.1** (Correct Equivariance).: For all \(x\in X,g\in G\) where \(p(x)>0\), if \(p(gx)>0\) and \(f(gx)=gf(x)\), \(h\) has _correct equivariance_ with respect to \(f\).

**Definition 3.2** (Incorrect Equivariance).: If there exist \(x\in X,g\in G\) such that \(p(x)>0,p(gx>0)\), but \(f(gx)\neq gf(x)\), \(h\) has _incorrect equivariance_ with respect to \(f\).

**Definition 3.3** (Extrinsic Equivariance).: For all \(x\in X,g\in G\) where \(p(x)>0\), if \(p(gx)=0\), \(h\) has _extrinsic equivariance_ with respect to \(f\).

**Example 3.4**.: Consider a binary classification task where \(X=\mathbb{R}\) and \(Y=\{0,1\}\). If the model \(h\) is invariant to a reflection group \(G\) where the group element \(g\in G\) acts on \(x\in X\) by \(gx=-x\), Figure 1 shows examples when correct, incorrect, or extrinsic equivariance is satisfied.

### Pointwise Equivariance Type.

Although Definitions 3.1- 3.3 are self-contained, they do not consider the mixture of different equivariance types in a single function. In other words, an equivariant model can have correct, incorrect, and extrinsic equivariance in different subsets of the domain. To overcome this issue, we define pointwise correct, incorrect, and extrinsic equivariance, which is a generalization of the prior work.

**Definition 3.5** (Pointwise Correct Equivariance).: For \(g\in G\) and \(x\in X\) where \(p(x)\neq 0\), if \(p(gx)\neq 0\) and \(f(gx)=gf(x)\), \(h\) has correct equivariance with respect to \(f\) at \(x\) under transformation \(g\).

Figure 2: Example of pointwise correct, incorrect, and extrinsic equivariance in a binary classification task. \(f(x)\) is in black and \(p(x)\) is in orange. \(G\) is a reflection group that negates \(x\).

**Definition 3.6** (Pointwise Incorrect Equivariance).: For \(g\in G\) and \(x\in X\) where \(p(x)\neq 0\), if \(p(gx)\neq 0\) and \(f(gx)\neq gf(x)\), \(h\) has incorrect equivariance with respect to \(f\) at \(x\) under transformation \(g\).

**Definition 3.7** (Pointwise Extrinsic Equivariance).: For \(g\in G\) and \(x\in X\) where \(p(x)\neq 0\), if \(p(gx)=0\), \(h\) has extrinsic equivariance with respect to \(f\) at \(x\) under transformation \(g\).

Notice that the definitions of pointwise correct, incorrect, and extrinsic equivariance are mutually exclusive, i.e., a pair \((x,g)\) can only have one of the three properties. The pointwise definitions are generalizations of the global Definitions 3.1- 3.3. For example, when pointwise correct equivariance holds for all \(x\in X\) and \(g\in G\), Definition 3.1 is satisfied.

**Example 3.8** (Example of Pointwise Correct, Incorrect, and Extrinsic Equivariance).: Consider the same binary classification task in Example 3.4. Figure 2 shows \(f(x)\), \(g(x)\), and four subsets of \(X\) where pointwise correct, incorrect, or extrinsic holds. For \(x\) in the correct section (green), \(p(x)>0,p(gx)>0,p(gx)>0,f(x)=f(gx)\). For \(x\) in the incorrect sections (red), \(p(x)>0,p(gx)>0,f(x)\neq f(gx)\). For \(x\) in the extrinsic section (blue), \(p(x)>0,p(gx)=0\).

**Definition 3.9** (Correct, Incorrect, and Extrinsic Sets).: The Correct Set \(C\subseteq X\times G\) is a subset of \(X\times G\) where pointwise correct equivariance holds for all \((x,g)\in C\). Similarly, the Incorrect Set \(I\) and the Extrinsic Set \(E\) are subsets where incorrect equivariance or extrinsic equivariance holds for all elements in the subset. Denote \(U\subseteq X\times G\) as the Undefined Set where \(\forall(x,g)\in U,p(x)=0\). By definition we have \(X\times G=C\amalg I\amalg E\amalg U\), where \(\amalg\) denotes a disjoint union.

## 4 Approximation Error Lower Bound from Incorrect Equivariance

Studying the theoretical error lower bound of an equivariant network is essential for model selection, especially when incorrect equivariance exists. Wang et al. [58] prove an error lower bound for an incorrect equivariant network, but their setting is limited to a classification task in the global situation of Definition 3.2 with a discrete group and an invariant density function. In this section, we find the lower bound of \(\operatorname{err}(h)\) for an equivariant model \(h\) in a general setting. To calculate such a lower bound, we first define the _fundamental domain_\(F\) of \(X\). Let \(d\) be the dimension of a generic orbit of \(G\) in \(X\) and \(n\) the dimension of \(X\). Let \(\nu\) be the \((n-d)\) dimensional Hausdorff measure in \(X\).

**Definition 4.1** (Fundamental Domain).: A closed subset \(F\) of \(X\) is called a fundamental domain of \(G\) in \(X\) if \(X\) is the union of conjugates2 of \(F\), i.e., \(X=\bigcup_{g\in G}gF\), and the intersection of any two conjugates has 0 measure under \(\nu\).

Footnote 2: A conjugate \(gF\) is defined as \(gF=\{gx|x\in F\}\).

We assume further that the set of all \(x\) which lie in any pairwise intersection \(\bigcup_{g_{1}F\neq g_{2}F}\left(g_{1}F\cap g_{2}F\right)\) has measure 0 under \(\nu\). Let \(Gx=\{gx:g\in G\}\) be the orbit of \(x\), then \(X\) can be written as the union of the orbits of all points in the fundamental domain \(F\) as such \(X=\bigcup_{x\in F}Gx\).

### Lower Bound for Classification

We first show the lower bound of the error \(\operatorname{err}_{\mathrm{cls}}(h)\) (Equation 1) given the invariant constraint in \(h\): \(h(gx)=h(x),g\in G\). In this section, the codomain \(Y\) of \(f\) is a finite set of possible labels. Since \(h\) is \(G\)-invariant, \(h\) has the same output for all inputs in an orbit \(Gx\). We call the label that causes the minimal error inside the orbit the _majority label3_, and define the error in the orbit as the _total dissent_.

Footnote 3: The majority label has more associated data than all other labels, but does not need to be more than 50%.

**Definition 4.2** (Total Dissent).: For the orbit \(Gx\) of \(x\in X\), the total dissent \(k(Gx)\) is the integrated probability density of the elements in the orbit \(Gx\) having a different label than the majority label

\[k(Gx)=\min_{y\in Y}\int_{Gx}p(z)\mathbb{1}\left(f(z)\neq y\right)dz. \tag{2}\]

We can also lift the integral to \(G\) itself by introducing a factor \(\alpha(x,g)\) to account for the Jacobian of the action map and size of the stabilizer of \(x\). (See Appendix A.)

\[k(Gx)=\min_{y\in Y}\int_{G}p(gx)\mathbb{1}\left(f(gx)\neq y\right)\alpha(x,g)dg. \tag{3}\]

**Theorem 4.3**.: \(\operatorname{err}(h)\) _is lower bounded by \(\int_{F}k(Gx)dx\)._

Proof.: Rewriting the error function of Equation 1, we have

\[\operatorname{err}(h)=\int_{X}p(x)\mathbb{1}(f(x)\neq h(x))dx=\int_{x\in F}\int_ {z\in Gx}p(z)\mathbb{1}(f(z)\neq h(z))dzdx, \tag{4}\]

using iterated integration (Appendix B) and Definition 4.1. We assume the measure of \(F\cap gF\) is 0. Since \(h(z)\) can only have a single label in orbit \(Gx\), we can lower bound the inside integral as

\[\int_{z\in Gx}p(z)\mathbb{1}(f(z)\neq h(z))dz\geq\min_{y\in Y}\int_{z\in Gx}p(z )\mathbb{1}(f(z)\neq y)dz=k(Gx).\]

We obtain the claim by integrating over \(F\). Notice that this is a tight lower bound assuming universal approximation. That is, there exists \(h\) which realizes this lower bound.

We can express the total dissent in terms of the Incorrect Set \(I\) (Definition 3.9).

**Proposition 4.4**.: \(k(Gx)=\min_{x^{\prime}\in(Gx)^{+}}\int_{G}p(gx^{\prime})\mathbb{1}((x^{\prime},g)\in I)\alpha(x^{\prime},g)dg\)_, where \((Gx)^{+}=\{x_{0}\in Gx|p(x_{0})>0\}\)._

Proof.: Consider Equation 3, since the minimum over \(y\) is obtained for \(y=f(x^{\prime})\) for some \(x^{\prime}\in Gx\) such that \(p(x^{\prime})>0\) (i.e., \(x^{\prime}\in(Gx)^{+}\)),

\[k(Gx)=\min_{x^{\prime}\in(Gx)^{+}}\int_{G}p(gx)\mathbb{1}(f(gx)\neq f(x^{\prime }))\alpha(x,g)dg.\]

Since \(x^{\prime}\in Gx\), then \(Gx^{\prime}=Gx\) and we have \(k(Gx)=k(Gx^{\prime})\). Thus,

\[k(Gx) =\min_{x^{\prime}\in(Gx)^{+}}\int_{G}p(gx^{\prime})\mathbb{1}(f(gx^ {\prime})\neq f(x^{\prime}))\alpha(x^{\prime},g)dg\] \[=\min_{x^{\prime}\in(Gx)^{+}}\int_{G}p(gx^{\prime})\mathbb{1}((x^ {\prime},g)\in I)\alpha(x^{\prime},g)dg.\]

**Example 4.5** (Lower bound example for a binary classification task using Proposition 4.4).:

Let \(f\colon X\to\{0,1\}\) be a binary classification function on \(X=\{x_{0},x_{1},x_{2},x_{3}\}\). Let \(G=C_{2}=\{e,r\}\) be the cyclic group of order two that permutes the elements in \(X\). Figure 3 shows \(X\), the label for each \(x\in X\), and how \(e,r\in G\) acts on \(x\in X\). \(\{x_{0},x_{3}\}\) forms a fundamental domain \(F\), and there are two orbits: \(Gx_{0}=\{x_{0},x_{1}\}\) and \(Gx_{2}=\{x_{2},x_{3}\}\). Since both \(X\) and \(G\) are discrete and \(g\in G\) acts on \(X\) through permutation, The lower bound can be written as \(\operatorname{err}(h)\geq\sum_{x\in F}\min_{x^{\prime}\in(Gx)^{+}}\sum_{g\in G }p(gx^{\prime})\mathbb{1}((x^{\prime},g)\in I)\). We can then calculate \(\sum_{g\in G}p(gx^{\prime})\mathbb{1}((x^{\prime},g)\in I)\) for \(x^{\prime}\in X\): \(x_{0}:0.4,x_{1}:0.3,x_{2}:0,x_{3}:0\). Taking the min over each orbit we have \(k(Gx_{0})=0.3,k(Gx_{2})=0\). Taking the sum over \(F=\{x_{0},x_{3}\}\) we obtain \(\operatorname{err}(h)\geq 0.3\).

**Example 4.6** (Lower bound example for a multi-class classification task using Proposition 4.4).: Consider a multi-class classification task \(f:\mathbb{R}^{2}\to Y\) with \(n=|Y|\) classes. For \(x=(u,v)\in[0,1]^{2}\) then \(p(u,v)=1\) and otherwise \(p(u,v)=0\); i.e., the support of \(p\) is a unit square. Let \(G\) denote the group of translations in the \(u\)-direction and \(h\) a \(G\)-invariant network. In a data distribution illustrated in Figure 4, we compute the lower bound for \(\operatorname{err}(h)\). Consider a fundamental domain \(F\) (brown line in Figure 4). In the blue area, there is one label across the orbit (i.e., the horizontalline), meaning \(\forall g\in G,(x^{\prime},g)\in C\), yielding Proposition 4.4 equals 0. For points in the yellow area, the majority label is yellow. This means that for \(g\in G\) such that \(gx\) is in yellow, \((x,g)\in C\); for other \(g\in G,(x,g)\in I\). Consequently, Proposition 4.4 is equivalent to the combined green and pink lengths. Taking the integral over \(F\) (Theorem 4.3), the lower bound equals the green and pink area (\(I\) in Figure 4). We define correct ratio (\(c\)) as the blue area's height and majority label ratio (\(m\)) as the yellow area's length. Adjusting \(c\) and \(m\) transitions incorrect to correct equivariance, leading to \(\operatorname{err}(h)\geq area(I)=(1-c)\times(1-m)\). Appendix H.2 shows an experiment where the empirical result matches our analysis.

Lower Bound When \(G\) is Finite and The Action of \(G\) is Density Preserving.In this section, we consider the lower bound in Theorem 4.3 when \(G\) is finite and the action of \(G\) is density preserving, i.e., \(p(gx)=p(x)\). Let \((Gx)_{y}=\{z\in Gx|f(z)=y\}\) be a subset of \(Gx\) with label \(y\). Define \(\mathcal{Q}(x)=(\max_{y\in Y}|(Gx)_{y}|)/|Gx|\), which is the fraction of data in the orbit \(Gx\) that has the majority label. Denote \(Q=\{\mathcal{Q}(x):x\in X\}\) the set of all possible values for \(\mathcal{Q}\). Consider a partition of \(X=\coprod_{q\in Q}X_{q}\) where \(X_{q}=\{x\in X:\mathcal{Q}(x)=q\}\). Define \(c_{q}=\mathbb{P}(x\in X_{q})=|X_{q}|/|X|\).

**Proposition 4.7**.: _The error lower bound \(\operatorname{err}(h)\geq 1-\sum_{q}qc_{q}\) from Wang et al. [58] (Proposition 4.1) is a special case of Theorem 4.3._

Proof in Appendix C. The proposition shows Theorem 4.3 is a strict generalization of [58, Prop 4.1].

### Lower Bound for Invariant Regression

In this section, we give a lower bound of the error function \(\operatorname{err}_{\operatorname{reg}}(h)\) (Equation 1) in a regression task given that \(h\) is invariant, i.e., \(h(gx)=h(x)\) for all \(g\in G\). Assume \(Y=\mathbb{R}^{n}\). Denote by \(p(Gx)=\int_{z\in Gx}p(z)dz\) the probability of the orbit \(Gx\). Denote by \(q(z)=\frac{p(z)}{p(Gx)}\) the normalized probability density of the orbit \(Gx\) such that \(\int_{Gx}q(z)dz=1\). Let \(\mathbb{E}_{Gx}[f]\) be the mean of function \(f\) on the orbit \(Gx\) defined, and let \(\mathbb{V}_{Gx}[f]\) be the variance of \(f\) on the orbit \(Gx\),

\[\mathbb{E}_{Gx}[f]=\int_{Gx}q(z)f(z)dz=\frac{\int_{Gx}p(z)f(z)dz}{\int_{Gx}p(z )dz},\qquad\mathbb{V}_{Gx}[f]=\int_{Gx}q(x)||\mathbb{E}_{Gx}[f]-f(z)||_{2}^{2}.\]

**Theorem 4.8**.: \(\operatorname{err}(h)\geq\int_{F}p(Gx)\mathbb{V}_{Gx}[f]dx\)_._

Proof.: The error function (Equation 1) can be written as:

\[\operatorname{err}(h)=\int_{X}p(x)||f(x)-h(x)||_{2}^{2}dx=\int_{x\in F}\int_{z \in Gx}p(z)||f(z)-h(z)||_{2}^{2}dzdx.\]

Denote \(e(x)=\int_{Gx}p(z)||f(z)-h(z)||_{2}^{2}dz\). Since \(h\) is \(G\)-invariant, there exists \(c\in\mathbb{R}^{n}\) such that \(h(z)=c\) for all \(z\in Gx\). Then \(e(x)\) can be written as \(e(x)=\int_{Gx}p(z)||f(z)-c||_{2}^{2}dz\). Taking the derivative of \(e(x)\) with respect to \(c\) and setting it to 0 gives \(c^{*}\), the minimum of \(e(x)\), \(c^{*}=\frac{\int_{Gx}p(z)f(z)dz}{\int_{Gx}p(z)dz}=\mathbb{E}_{Gx}[f]\). Substituting \(c^{*}\) into \(e(x)\) we have

\[e(x)\geq\int_{Gx}p(Gx)\frac{p(z)}{p(Gx)}||\mathbb{E}_{Gx}[f]-f(z)||_{2}^{2}dz= p(Gx)\mathbb{V}_{Gx}[f].\]

We can obtain the claim by taking the integral of \(e(x)\) over the fundamental domain \(F\).

### Lower Bound for Equivariant Regression

We now prove a lower bound for \(\operatorname{err}(h)\) in a regression task given the model \(h\) is equivariant, that is, \(h(\rho_{X}(g)x)=\rho_{Y}(g)h(x)\) where \(g\in G,\rho_{X}\) and \(\rho_{Y}\) are group representations associated with \(X\) and \(Y\). We will denote \(\rho_{X}(g)x\) and \(\rho_{Y}(g)y\) by \(gx\) and \(gy\), leaving the representation implicit. Assume \(Y=\mathbb{R}^{n}\) and \(\alpha(x,g)\) is the same as in equation 3. Let \(\operatorname{Id}\) be the identity. Define a matrix \(Q_{Gx}\in\mathbb{R}^{n\times n}\) and \(q(gx)\in\mathbb{R}^{n\times n}\) so that \(\int_{G}q(gx)dg=\operatorname{Id}\) by

\[Q_{Gx}=\int_{G}p(gx)\rho_{Y}(g)^{T}\rho_{Y}(g)\alpha(x,g)dg,\qquad q(gx)=Q_{Gx }^{-1}p(gx)\rho_{Y}(g)^{T}\rho_{Y}(g)\alpha(x,g). \tag{5}\]Here, for simplicity, we assume \(Q_{Gx}\) is an invertible matrix. (See Appendix D for general case).

If \(f\) is equivariant, \(g^{-1}f(gx)\) is a constant for all \(g\in G\). Define \(\mathbf{E}_{G}[f,x]\)

\[\mathbf{E}_{G}[f,x]=\int_{G}q(gx)g^{-1}f(gx)dg. \tag{6}\]

**Theorem 4.9**.: _The error of \(h\) has lower bound \(\operatorname{err}(h)\geq\int_{F}\int_{G}p(gx)||f(gx)\;-\;g\mathbf{E}_{G}[f,x] ||_{2}^{2}\alpha(x,g)dgdx\)._

See Appendix D for the proof. Intuitively, \(\mathbf{E}_{G}[f,x]\) is the minimizer obtained by taking the mean of all inversely transformed \(f(x)\) for all \(x\) in the orbit, see Figure 4(c) and Example 4.11 below.

**Corollary 4.10**.: _Denote \(p(Gx)=\int_{Gx}p(z)dz\). Denote \(q_{x}:g\mapsto q(gx)\). Define \(G\)-stabilized \(f\) as \(f_{x}:g\mapsto g^{-1}f(gx)\). When \(\rho_{Y}\) is an orthogonal representation \(\rho_{Y}:G\to\operatorname{O}(n)\subset GL(n)\), \(q_{x}\) is a probability density function on \(G\). Denote the variance of \(f_{x}\) as \(\mathbb{V}_{G}[f_{x}]\) where \(g\sim q_{x}\). The error has a lower bound \(\operatorname{err}(h)\geq\int_{F}p(Gx)\mathbb{V}_{G}[f_{x}]dx\)._

See Appendix E for the proof. Notice that Corollary 4.10 is a generalization of Theorem 4.8. That is, Theorem 4.8 can be recovered by taking \(\rho_{Y}(g)=\operatorname{Id}\) (See the proof in Appendix F).

**Example 4.11** (Lower bound example of a regression task).: Consider a regression problem where \(X=\{x_{0},x_{1},x_{2},x_{3}\}\) and \(Y=\mathbb{R}^{2}\). Assume \(p\) is uniform density. The cyclic group \(G=C_{4}=\{e,r,r^{2},r^{3}\}\) (where \(e=0\) rotation and \(r=\pi/2\) rotation) acts on \(X\) through \(x_{1}=rx_{0};x_{2}=rx_{1};x_{3}=rx_{2};x_{0}=rx_{3}\) (i.e., there is only one orbit \(Gx=X\)). \(g\in G\) acts on \(y\in Y\) through \(\rho_{Y}(g)=\left(\begin{smallmatrix}\cos g&-\sin g\\ sin&g&\cos g\end{smallmatrix}\right)\). Figure 4(a) shows the output of \(f(x),\forall x\in X\). **First**, consider a \(G\)-invariant network \(h\). Since there is only one orbit, Theorem 4.8 can be simplified as: \(\operatorname{err}(h)\geq\mathbb{V}_{X}[f]\), the variance of \(f\) over \(X\). This can be calculated by first taking the mean of \(f(x)\) then calculating the mean square error (MSE) from all \(x\) to the mean (Figure 4(b)). **Second**, consider a \(G\)-equivariant network \(h\). Since \(G\) is discrete, \(gx\) permutes the order of \(X\), \(\rho_{Y}\) is an orthogonal representation, and there is only one orbit, Corollary 4.10 can be written as \(\operatorname{err}(h)\geq\mathbb{V}_{G}[f_{x}]\), the variance of \(G\)-stabilized \(f\). First, to calculate \(\mathbb{E}_{G}[f_{x}]\), let \(x=x_{0}\), we stabilize \(g\) from \(f\) by \(g^{-1}f(gx)\) for all \(g\in G\), then take the mean (Figure 4(c)). We can then find \(\mathbb{V}_{G}[f_{x}]\) by calculating the MSE between \(f(x)\) and transformed mean \(g\mathbb{E}_{G}[f_{x}]\) (Figure 4(d)). Appendix H.3 shows an experiment in this example's environment.

## 5 Harmful Extrinsic Equivariance

Wang et al. [58] demonstrate that extrinsic equivariance, where the symmetry imposed on the model leads to out-of-distribution data with respect to the input distribution, can lead to a higher performance on the original training data. In this section, we argue that this is not necessarily true in all cases, and there can exist scenarios where extrinsic equivariance can even be harmful to the learning problem.

Consider a binary classification task where the domain is discrete and contains only a set of four points \(S\subset\mathbb{R}^{3}\), and their labels are either \(\{-1,+1\}\) as shown in Figure 5(a). We consider the probability density \(p\) to be uniform for this domain, i.e., \(p(x)=1/4\) for the four points \(S\), and \(p=0\) elsewhere.

Figure 5: An example regression task. (a) The value of \(f(x)\) and the transformation rule (purple) with respect to group \(G=C_{4}\) for all \(x\in X\). The four points belong to a single orbit. (b) When using an invariant network, the minimal error (red) is obtained when the invariant network outputs the mean value (green) of the orbit. (c) For an equivariant network, the minimizer (green) can be obtained by taking the mean of the \(G\)-stabilized \(f(x)\) (inversely transformed) (blue) for all \(x\) in the orbit with respect to the transformation rule in the orbit. (d) The minimal error of an equivariant network.

This domain is used for both model training and testing so there is no distribution shift. We consider two model classes, \(\mathcal{F}_{N}\), the set of all linear models, and \(\mathcal{F}_{E}\), the set of all linear models which are invariant with respect to the cyclic group \(C_{2}=\{1,g\}\), where \(g(x_{1},x_{2},x_{3})=(x_{1},x_{2},-x_{3})\). \(\mathcal{F}_{N}\) corresponds to an unconstrained or a non-equivariant model class and \(\mathcal{F}_{E}\) corresponds to an extrinsically equivariant class for this domain. For the labeling shown in Figure 6, the hyperplane \(x_{3}=0\) correctly classifies all samples and is contained in \(\mathcal{F}_{N}\). However, a function \(f_{e}\in\mathcal{F}_{E}\) is equivalent to a linear classifier on \(\mathbb{R}^{2}\) and effectively sees the data as Figure 5(b)4. This exclusive-or problem does not admit a linear solution (it can be correct for at most 3 points).

Footnote 4: Notice that the four additional points in Figure 5(b) compared with (a) are not in the domain, they are created through applying the transformation rule of \(\mathcal{F}_{E}\) onto the domain.

Concretely, we can compute the empirical Rademacher complexity, a standard measure of model class expressivity, for non-equivariant and extrinsically equivariant model classes and show that \(\mathcal{F}_{E}\) has lower complexity than \(\mathcal{F}_{N}\). Recall that empirical Rademacher complexity is defined as \(\mathfrak{R}_{S}\left(\mathcal{F}\right)=\mathbb{E}_{\sigma}\left[\sup_{f\in \mathcal{F}}\frac{1}{m}\sum_{i=1}^{m}\sigma_{i}f(x^{i})\right]\), where \(S\) is the set of \(m\) samples and \(\sigma=(\sigma_{1},\ldots,\sigma_{m})^{\top}\), \(\sigma_{i}\in\{-1,+1\}\) are independent uniform Rademacher random variables, and \(x^{i}\) is the \(i\)-th sample. As there exists some linear function \(f_{n}\in\mathcal{F}_{N}\) that fully classifies \(S\) for any combination of labels, \(\mathfrak{R}_{S}(\mathcal{F}_{N})=1\). For the extrinsic equivalence case, of the 16 possible label combinations, there are two cases where \(f_{e}\in\mathcal{F}_{E}\) can at most classify 3 out of 4 points correctly, and thus \(\mathfrak{R}_{S}(\mathcal{F}_{E})=\frac{31}{32}<\mathfrak{R}_{S}(\mathcal{F}_ {N})\) (see Appendix G for the calculations). This illustrates that in certain cases, extrinsic equivariance can lead to lower model expressivity than no equivariance and thus be harmful to learning.

## 6 Experiments

We perform experiments to validate our theoretical analysis on both the lower bounds (Section 4) and the harmful extrinsic equivariance (Section 5). We find that our bounds accurately predict empirical model error. In addition to the experiments in this section, Appendix H.2 shows an experiment verifying our classification bound (Theorem 4.3) and Appendix H.3 shows an experiment verifying our regression bound (Theorem 4.8 and 4.9). The experiment details are in Appendix I.

### Swiss Roll Experiment

We first perform an experiment in a vertically separated Swiss Roll data distribution, see Figure 6(a)5. This example, similar to that in Section 5, demonstrates that a \(C_{2}\)-invariant model effectively "flattens" the \(z\)-dimension of the data so it must learn the decision boundary between two spirals (Figure 6(b)), whereas the non-equivariant model only needs to learn a horizontal plane to separate the classes, a significantly easier task. Besides the extrinsic data distribution, we consider two other data distributions shown in Figure 6(c) and Figure 6(d), where a \(C_{2}\)-invariant model will observe incorrect and correct equivariance due to the mismatched and matched data labels in the two \(z\) planes.

Footnote 5: For visualization, we show a simpler version of the data distribution. See Appendix H.1 for the actual one.

We combine data from all three distributions in various proportions to test the performance of a \(z\)-invariant network (INV) with a baseline unconstrained network (MLP). Let \(c\) be the correct ratio, the proportion of data from the correct distribution. Define the incorrect ratio \(i\) and extrinsic ratio \(e\) similarly. We consider all \(c,i,e\) that are multiples of 0.125 such that \(c+i+e=1\). Figure 6(e) shows some example data distributions. Relative to INV, this mixed data distribution has partial correct, incorrect, and extrinsic equivariance, which is not fully captured in prior work [58]. Based on Proposition 4.4, we have \(k(Gx)=0.5\) for \(x\) drawn from the incorrect distribution, and \(k(Gx)=0\) otherwise. Since the data is evenly distributed, we can calculate the error lower bound \(\operatorname{err}(h)\geq 0.5i\)

Figure 6: An example dataset where extrinsic equivariance increases the problem difficulty. The samples are of the form \(x=(x_{1},x_{2},x_{3})\) and the labels are shown as different shapes. A \(C_{2}\)-equivariant linear model transforms the original data (a) into (b), which is equivalent to viewing the data as in (c). The original task has an easy solution (e.g. hyperplane at \(x_{3}=0\)), while the \(C_{2}\)-invariant view is the classic exclusive-or problem.

**Results.** Figure 7(a) shows the test success rate of INV compared with MLP when \(e\) and \(c\) vary with \(i=0\). When \(e\) increases, the performance of INV decreases while the performance of MLP shows an inverse trend, demonstrating that extrinsic equivariance is harmful in this experiment. Figure 7(b) shows the performance of INV and MLP when \(c\) and \(i\) vary while \(e=0\). The green line shows the upper bound of the test success rate (\(1-0.5i\)). The experimental result matches our theoretical analysis quite closely. Notice that when \(c\) increases, there is a bigger gap between the performance of the network and its theoretical upper bound, since classification in the correct distribution is a harder task. Appendix H.1 shows the complete results of this experiment.

### Digit Classification Experiment

In this experiment, we apply our theoretical analysis to a realistic digit classification task using both the printed digit dataset [16] and the MNIST handwritten digit dataset [15]. We compare a \(D_{4}\)-invariant network (\(D_{4}\)) with an unconstrained CNN. In the printed digit classification, \(D_{4}\) exhibits incorrect equivariance for 6 and 9 under a \(\pi\) rotation. Using Theorem 4.3, we can calculate a lower bound of error for \(D_{4}\) at 10%. However, as shown in Table 1 (top), the experimental results indicate that the actual performance is slightly better than predicted by the theory. We hypothesize that this discrepancy arises because a rotated 9 differs slightly from a 6 in some fonts. We conduct a similar experiment using the MNIST handwritten digit dataset (Table 1 bottom), where \(D_{4}\) achieves even better performance in classifying 6 and 9. This improvement is likely due to the more distinguishable handwriting of these digits, although the performance still underperforms the CNN as incorrect equivariance persists. It is important to note that there is a significant decrease in performance for \(D_{4}\) when classifying 2/5 and 4/7 compared to the CNN. This is because a vertical flip results in incorrect equivariance when classifying handwritten 2/5, and a similar issue arises for 4/7 under a \(\pi/2\) rotation followed by a vertical flip (notice that Weiler and Cesa [60] make a similar observation). These experiments demonstrate that our theory is useful not only for calculating the performance bounds of an equivariant network beforehand, but also for explaining the suboptimal performance of an equivariant network, thereby potentially assisting in model selection.

### Robotic Experiment

In this experiment, we evaluate our theory in behavior cloning in robotic manipulation. We first preform an experiment where the problem is a mixture of correct and incorrect equivariance for a \(D_{1}\)-equivariant policy network (\(D_{1}\)) where the robot's action will flip when the state is flipped

Figure 8: Result of the Swiss Roll experiment. (a) test success rate of an invariant network (red) and an unconstrained MLP (blue) with different extrinsic and correct ratio when incorrect ratio is 0. (b) same as (a) with different correct and incorrect ratio when extrinsic ratio is 0. Averaged over 10 runs.

Figure 7: (a) (b) The Swiss Roll data distribution that leads to harmful extrinsic equivariance. (c) (d) The correct and incorrect data distribution in the Swiss Roll experiment. Here the spirals overlap with mismatched and matched labels respectively. (e) (f) Data distribution example with different correct ratio (\(c\)), incorrect ratio (\(i\)), and extrinsic ratio (\(e\)) values.

horizontally. Specifically, the environment contains two possible tasks (Figure 9 left). Stacking requires the robot to stack a green triangle on top of a blue cube. Here flip equivariance is correct. Sorting requires the robot to push the red cube to the left and the yellow triangle to the right. Here flip equivariance is incorrect because the robot should not sort the objects in an opposite way when the state is flipped (in other words, \(D_{1}\) cannot distinguish left and right). We vary the probability \(c\) of the stacking task (correct ratio) in the task distribution, and compare the performance of \(D_{1}\) versus a standard CNN policy. If we view the sorting task as a binary classification task, we can calculate an upper bound of performance for \(D_{1}\) using Theorem 4.3: \(0.5+0.5c\). Figure 10 shows the result. Notice that the performance of \(D_{1}\) closely matches the theoretical upper bound, while the performance of CNN remains relatively stable for all Stacking-Sorting distributions.

We further evaluate \(D_{1}\) in a sorting task with harmful extrinsic equivariance. Here, the goal for the robot is the same as sorting above (push the red cube left and the yellow triangle right), however, the left and right sides of the workspace can now be differentiated by gray-scale colors. The shades of gray are discretized evenly into \(n\) bins, where the left side's color is randomly sampled from the odd-numbered bins, and the right side's color is randomly sampled from the even-numbered bins (Figure 9 right). The different color distributions of the left and right sides make \(D_{1}\) extrinsically equivariant, but it needs to learn the color distribution to distinguish left and right (while CNN can distinguish left and right directly). We set \(n=10\), and \(D_{1}\) achieves \(71.5\pm 1.6\%\) test success rate, while CNN achieves \(99.5\pm 0.5\%\), demonstrating that the \(D_{1}\) extrinsic equivariance is harmful in this task. See Appendix 1.5 for the details of the robot experiment.

## 7 Discussion

This paper presents a general theory for when the symmetry of the ground truth function and equivariant network are mismatched. We define pointwise correct, incorrect, and extrinsic equivariance, generalizing prior work [58] to include continuous mixtures of the three extremes. We prove error lower bounds for equivariant networks applied to asymmetric tasks including classification, invariant regression, and equivariant regression without the assumption of invariant data density. Our work discusses the potential disadvantage of extrinsic equivariance, and provides experiments that validate our theoretical analysis. The major limitation of this paper is that our theoretical lower bounds require domain knowledge like the density function over the domain. In future work, we will develop easy-to-apply model selection tools using our theory. Another future direction is theoretically understanding when extrinsic equivariance is helpful or harmful and analyzing the effect of extrinsic equivariance on the decision boundary of an equivariant network.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline Digit & Overall & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 \\ \hline Print CNN & 96.62 & 99.81 & 98.27 & 97.06 & 96.12 & 98.04 & 92.9 & 94.93 & 97.85 & 93.65 & 96.13 \\ Print D4 & 92.5 & 99.71 & 98.62 & 96.94 & 95.98 & 97.91 & 93.52 & **63.08** & 98.42 & 95.88 & **76.17** \\ Print D4 Upper Bound & 90 & 100 & 100 & 100 & 100 & 100 & 100 & 50 & 100 & 100 & 50 \\ \hline MNIST CNN & 98.21 & 99.51 & 99.61 & 98.62 & 98.83 & 98.08 & 98.47 & 97.99 & 97.04 & 96.98 & 96.81 \\ MNIST D4 & 96.15 & 98.93 & 99.21 & **91.84** & 98.28 & **95.49** & **95.04** & **93.71** & **95.67** & 97.73 & **95.34** \\ \hline \hline \end{tabular}
\end{table}
Table 1: \(D_{4}\)-invariant network compared with an unconstrained CNN in printed and MNIST handwritten digit classification tasks. Bold indicates that there is a \(>1\%\) difference in two models.

## Acknowledgments

This work is supported in part by NSF 1724257, NSF 1724191, NSF 1763878, NSF 1750649, NSF 2107256, NSF 2134178, NSF 2312171, and NASA 80NSSC19K1474.

## References

* A. Behboodi, G. Cesa, and T. Cohen (2022)A PAC-bayesian generalization bound for equivariant networks. arXiv preprint arXiv:2210.13150. Cited by: SS1.
* A. Bogatskiy, B. Anderson, J. Offermann, M. Roussi, D. Miller, and R. Kondor (2020)Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pp. 992-1002. Cited by: SS1.
* E. Chatzipantazis, S. Pertigkiozoglou, E. Dobriban, and K. Daniilidis (2023)\(\mathrm{SE}(3)\)-equivariant attention networks for shape reconstruction in function space. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: SS1.
* S. Chen, E. Dobriban, and J. H. Lee (2020)A group-theoretic framework for data augmentation. The Journal of Machine Learning Research21 (1), pp. 9885-9955. Cited by: SS1.
* T. Cohen and M. Welling (2016)Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999. Cited by: SS1.
* T. S. Cohen and M. Welling (2017)Steerable CNNs. In International Conference on Learning Representations, External Links: Link Cited by: SS1.
* T. S. Cohen, M. Geiger, and M. Weiler (2019)A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems32. Cited by: SS1.
* E. Coumans and Y. Bai (2016)Pybullet, a python module for physics simulation for games, robotics and machine learning. Note: [http://pybullet.org](http://pybullet.org) External Links: Link Cited by: SS1.
* R. Dangovski, L. Jing, C. Loh, S. Han, A. Srivastava, B. Cheung, P. Agrawal, and M. Soljacic (2021)Equivariant self-supervised learning: encouraging equivariance in representations. In International Conference on Learning Representations, External Links: Link Cited by: SS1.
* N. Dehmamy, R. Walters, Y. Liu, D. Wang, and R. Yu (2021)Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems34, pp. 2503-2515. Cited by: SS1.
* L. Deng (2012)The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine29 (6), pp. 141-142. Cited by: SS1.
* K. Dhama (2021)Printed numerical digits image dataset. Note: [https://github.com/kaydee0502/printed-digits-dataset](https://github.com/kaydee0502/printed-digits-dataset) Cited by: SS1.
* B. Elesedy and S. Zaidi (2021)Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning, pp. 2959-2969. Cited by: SS1.

* Finzi et al. [2020] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In _International Conference on Machine Learning_, pages 3165-3176. PMLR, 2020.
* Hauberg et al. [2016] Soren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen. Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. In _Artificial intelligence and statistics_, pages 342-350. PMLR, 2016.
* Hinton et al. [2011] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In _International conference on artificial neural networks_, pages 44-51. Springer, 2011.
* Huang et al. [2022] Haojie Huang, Dian Wang, Robin Walters, and Robert Platt. Equivariant transporter network. In _Robotics: Science and Systems_, 2022.
* Huang et al. [2023] Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt. Edge grasp network: A graph-based \(\mathrm{SE}(3)\)-invariant approach to grasp detection. In _International Conference on Robotics and Automation (ICRA)_, 2023.
* Jia et al. [2023] Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt. Seil: Simulation-augmented equivariant imitation learning. In _International Conference on Robotics and Automation (ICRA)_, 2023.
* Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_, 2014.
* Klee et al. [2022] David Klee, Ondrej Biza, Robert Platt, and Robin Walters. I2i: Image to icosahedral projection for \(\mathrm{SO}(3)\) object reasoning from single-view images. _arXiv preprint arXiv:2207.08925_, 2022.
* Klee et al. [2023] David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning equivariant features for efficient pose prediction. In _The Eleventh International Conference on Learning Representations_, 2023. URL [https://openreview.net/forum?id=_2bDpAtr7PI](https://openreview.net/forum?id=_2bDpAtr7PI).
* Kondor and Trivedi [2018] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In _International Conference on Machine Learning_, pages 2747-2755. PMLR, 2018.
* Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. _Advances in neural information processing systems_, 25, 2012.
* LeCun et al. [1998] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. _Proceedings of the IEEE_, 86(11):2278-2324, 1998.
* Lenc and Vedaldi [2015] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 991-999, 2015.
* Li et al. [2021] Xiaolong Li, Yijia Weng, Li Yi, Leonidas J Guibas, A Abbott, Shuran Song, and He Wang. Leveraging se (3) equivariance for self-supervised category-level object pose estimation from point clouds. _Advances in Neural Information Processing Systems_, 34:15370-15381, 2021.
* Liu et al. [2023] Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised category-level articulated object pose estimation with part-level SE(3) equivariance. In _The Eleventh International Conference on Learning Representations_, 2023. URL [https://openreview.net/forum?id=206tJ6hIaPA](https://openreview.net/forum?id=206tJ6hIaPA).
* Lyle et al. [2020] Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. _arXiv preprint arXiv:2005.00178_, 2020.
* Maile et al. [2023] Kaitlin Maile, Dennis George Wilson, and Patrick Forre. Equivariance-aware architectural optimization of neural networks. In _The Eleventh International Conference on Learning Representations_, 2023. URL [https://openreview.net/forum?id=a6rCdfABJXg](https://openreview.net/forum?id=a6rCdfABJXg).
* Marchetti et al. [2022] Giovanni Luca Marchetti, Gustaf Tegner, Anastasiia Varava, and Danica Kragic. Equivariant representation learning via class-pose decomposition. _arXiv preprint arXiv:2207.03116_, 2022.

* Maron et al. [2019] Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In _International conference on machine learning_, pages 4363-4371. PMLR, 2019.
* Maron et al. [2020] Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In _International conference on machine learning_, pages 6734-6744. PMLR, 2020.
* Miao et al. [2021] Ning Miao, Tom Rainforth, Emile Mathieu, Yann Dubois, Yee Whye Teh, Adam Foster, and Hyunjik Kim. Learning instance-specific augmentations by capturing local invariances. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 24720-24736. PMLR, 23-29 Jul 2023. URL [https://proceedings.mlr.press/v202/miao23a.html](https://proceedings.mlr.press/v202/miao23a.html).
* Mondal et al. [2022] Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh. EqR: Equivariant representations for data-efficient reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 15908-15926. PMLR, 17-23 Jul 2022. URL [https://proceedings.mlr.press/v162/mondal22a.html](https://proceedings.mlr.press/v162/mondal22a.html).
* Moskalev et al. [2022] Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold W.M. Smeulders. LieGG: Studying learned lie group generators. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_, 2022. URL [https://openreview.net/forum?id=9sKZ60VtRmi](https://openreview.net/forum?id=9sKZ60VtRmi).
* Pan et al. [2023] Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In _Conference on Robot Learning_, pages 1783-1792. PMLR, 2023.
* Park et al. [2022] Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric representations for equivariant world model. In _International Conference on Machine Learning_, 2022. URL [https://arxiv.org/abs/2204.11371](https://arxiv.org/abs/2204.11371).
* Quessard et al. [2020] Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. _Advances in Neural Information Processing Systems_, 33:19727-19737, 2020.
* Rommel et al. [2021] Cedric Rommel, Thomas Moreau, Joseph Paillard, and Alexandre Gramfort. Cadda: Class-wise automatic differentiable data augmentation for eeg signals. In _International Conference on Learning Representations_, 2021.
* Ryu et al. [2023] Hyunwoo Ryu, Hong in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: SE(3)-equivariant energy-based models for end-to-end visual robotic manipulation learning. In _The Eleventh International Conference on Learning Representations_, 2023. URL [https://openreview.net/forum?id=dnJZSPGmY50](https://openreview.net/forum?id=dnJZSPGmY50).
* Sannai et al. [2021] Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invariant/equivariant deep networks via quotient feature spaces. In _Uncertainty in Artificial Intelligence_, pages 771-780. PMLR, 2021.
* Schmidt and Roth [2012] Uwe Schmidt and Stefan Roth. Learning rotation-aware features: From invariant priors to equivariant descriptors. In _2012 IEEE Conference on Computer Vision and Pattern Recognition_, pages 2050-2057. IEEE, 2012.
* Simeonov et al. [2022] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In _2022 International Conference on Robotics and Automation (ICRA)_, pages 6394-6400. IEEE, 2022.
* Simeonov et al. [2023] Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomas Lozano-Perez, and Pulkit Agrawal. Se (3)-equivariant relational rearrangement with neural descriptor fields. In _Conference on Robot Learning_, pages 835-846. PMLR, 2023.

* Sohn and Lee [2012] Kihyuk Sohn and Honglak Lee. Learning invariant representations with local transformations. In John Langford and Joelle Pineau, editors, _Proceedings of the 29th International Conference on Machine Learning (ICML-12)_, ICML '12, pages 1311-1318, New York, NY, USA, July 2012. Omnipress. ISBN 978-1-4503-1285-1.
* Sokolic et al. [2017] Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel Rodrigues. Generalization error of invariant classifiers. In _Artificial Intelligence and Statistics_, pages 1094-1103. PMLR, 2017.
* van der Pol et al. [2020] Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. _Advances in Neural Information Processing Systems_, 33, 2020.
* Walters et al. [2020] Robin Walters, Jinxi Li, and Rose Yu. Trajectory prediction using equivariant continuous convolution. _arXiv preprint arXiv:2010.11344_, 2020.
* Wang et al. [2021] Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant \(Q\) learning in spatial action spaces. In _5th Annual Conference on Robot Learning_, 2021. URL [https://openreview.net/forum?id=IScz42A3iCI](https://openreview.net/forum?id=IScz42A3iCI).
* Wang et al. [2022] Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt. On-robot learning with equivariant models. In _6th Annual Conference on Robot Learning_, 2022. URL [https://openreview.net/forum?id=K8W6ObPZQyh](https://openreview.net/forum?id=K8W6ObPZQyh).
* Wang et al. [2022] Dian Wang, Robin Walters, and Robert Platt. \(\mathrm{SO}(2)\)-equivariant reinforcement learning. In _International Conference on Learning Representations_, 2022. URL [https://openreview.net/forum?id=7F9cOhdvfk_](https://openreview.net/forum?id=7F9cOhdvfk_).
* Wang et al. [2023] Dian Wang, Colin Kohler, Xupeng Zhu, Mingxi Jia, and Robert Platt. Bulletarm: An open-source robotic manipulation benchmark and learning framework. In _Robotics Research_, pages 335-350. Springer, 2023.
* Wang et al. [2023] Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L.S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. In _International Conference on Learning Representations_, 2023. URL [https://openreview.net/forum?id=P4MUGRM4Acu](https://openreview.net/forum?id=P4MUGRM4Acu).
* Wang et al. [2020] Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. _arXiv preprint arXiv:2002.03061_, 2020.
* Weiler and Cesa [2019] Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. _Advances in Neural Information Processing Systems_, 32, 2019.
* Winter et al. [2022] Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arne Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_, 2022. URL [https://openreview.net/forum?id=47lpv23LDPr](https://openreview.net/forum?id=47lpv23LDPr).
* Yarotsky [2022] Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. _Constructive Approximation_, 55(1):407-474, 2022.
* Zhao et al. [2023] Linfeng Zhao, Xupeng Zhu, Lingzhi Kong, Robin Walters, and Lawson L.S. Wong. Integrating symmetry into differentiable planning with steerable convolutions. In _The Eleventh International Conference on Learning Representations_, 2023. URL [https://openreview.net/forum?id=n7CPzHPKQ1](https://openreview.net/forum?id=n7CPzHPKQ1).
* Zhou et al. [2020] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. _arXiv preprint arXiv:2007.02933_, 2020.
* Zhu et al. [2022] Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. In _Robotics: Science and Systems_, 2022.

Integrals on the Group

Fundamental DomainsIn this paper, we are interested in cases in which the group \(G\) is not necessarily discrete but may have positive dimension. We do not assume the fundamental domain has non-empty interior, and thus domain is a misnomer. In this case the conjugates of the fundamental domain \(gF\) have measure 0 and the condition that their intersection have measure 0 is vacuous. Instead we assume a stronger condition, that the union of all pairwise intersections \(\bigcup_{g_{1}\neq g_{2}}\left(g_{1}F\cap g_{2}F\right)\) has measure 0. We also require that \(F\) and the orbits \(Gx\) are differentiable manifolds such that integrals over \(X\) may be evaluated \(\int_{X}f(x)dx=\int_{F}\int_{Gy}f(z)dzdy\) similar to Equation 8 from [18].

ReparameterizationConsider the integral

\[\int_{Gx}f(z)dz. \tag{7}\]

Denote the identification of the orbit \(Gx\) and coset space \(G/G_{x}\) with respect to the stabilizer \(G_{x}=\{g:gx=x\}\) by \(a_{x}\colon G/G_{x}\to Gx\). Then the integral can be written

\[\int_{G/G_{x}}f(\bar{g}x)\left|\frac{\partial a_{x}(\bar{g})}{\partial\bar{g}} \right|d\bar{g}.\]

We can also lift the integral to \(G\) itself

\[\int_{G/G_{x}}f(\bar{g}x)\left|\frac{\partial a_{x}(\bar{g})}{ \partial\bar{g}}\right|d\bar{g} =\left(\int_{G_{x}}dh\right)^{-1}\left(\int_{G_{x}}dh\right)\int_ {G/G_{x}}f(\bar{g}x)\left|\frac{\partial a_{x}(\bar{g})}{\partial\bar{g}} \right|d\bar{g}\] \[=\left(\int_{G_{x}}dh\right)^{-1}\int_{G/G_{x}}\int_{G_{x}}f(\bar {g}hx)\left|\frac{\partial a_{x}(\bar{g})}{\partial\bar{g}}\right|dhd\bar{g}\] \[=\left(\int_{G_{x}}dh\right)^{-1}\int_{G}f(gx)\left|\frac{ \partial a_{x}(\bar{g})}{\partial\bar{g}}\right|dg.\]

Define \(\alpha(g,x)=\left(\int_{G_{x}}dh\right)^{-1}\left|\frac{\partial a_{x}(\bar{g} )}{\partial\bar{g}}\right|\). Then

\[\int_{Gx}f(z)dz=\int_{G}f(gx)\alpha(g,x)dg.\]

## Appendix B Iterated Integral

Let \(X\) be an \(n\)-dimensional space, Definition 4.2 (Equation 2) defines \(k(Gx)\) as an integral over \(Gx\subseteq X\), which is a \(m\)-dimensional sub-manifold of \(X\). In Theorem 4.3, Equation 4 rewrites the error function (Equation 1) as an iterated integral over the orbit \(Gx\) and then the fundamental domain \(F\) using Definition 4.1. In the discrete group case, \(m\) would be 0, Equation 2 is an integral of a 0-form in a 0-manifold, which is a sum:

\[k(Gx)=\min_{y\in Y}\sum_{z\in Gx}p(z)\mathbb{1}(f(z)\neq y)=\min_{y\in Y}\sum_{ g\in G}p(gx)\mathbb{1}(f(gx)\neq y) \tag{8}\]

## Appendix C Proof of Proposition 4.7

Proof.: Consider the integral of probability density inside \(Gx\), for a given \(y\), it can be separated into two groups:

\[\int_{Gx}p(z)dz= \int_{Gx}p(z)\mathbb{1}(f(z)=y)dz\] \[+\int_{Gx}p(z)\mathbb{1}(f(z)\neq y)dz.\]We can then rewrite \(k(Gx)\) in Equation 2 as:

\[k(Gx)=\min_{y\in Y}\left[\int_{Gx}p(z)dz-\int_{Gx}p(z)\mathbb{1}\left(f(z)=y\right) dz\right]. \tag{9}\]

Letting \((Gx)_{y}=\{x^{\prime}\in Gx\mid f(x^{\prime})=y\}=f^{-1}(y)\cap Gx\), Equation 9 can be written as:

\[k(Gx)=\min_{y\in Y}\Big{[}\int_{Gx}p(z)dz-\int_{(Gx)_{y}}p(z)dz \Big{]}\] \[\int_{Gx}p(z)dz-\max_{y\in Y}\int_{(Gx)_{y}}p(z)dz.\]

Theorem 4.3 can be rewritten as:

\[\operatorname{err}(h) \geq\int_{F}\Big{(}\int_{Gx}p(z)dz-\max_{y\in Y}\int_{(Gx)_{y}}p( z)dz\Big{)}dx\] \[\geq\int_{F}\int_{Gx}p(z)dz-\int_{F}\max_{y\in Y}\int_{(Gx)_{y}}p (z)dz\] \[\geq 1-\int_{F}\max_{y\in Y}|(Gx)_{y}|p(x)dx. \tag{10}\]

The first term in Equation 10 uses the fact that \(X=\bigcup_{x\in F}Gx\) so the integral of the probability of the orbits of all points in the fundamental domain is the integral of the probability of the input domain \(X\) which is 1. The second term of Equation 10 uses \(p(gx)=p(x)\) so the integration of \(p(z)\) on \((Gx)_{y}\) becomes \(p(x)\) times the range of the limit which is the size of \((Gx)_{y}\), \(|(Gx)_{y}|\).

Now consider a partition of \(F=\coprod_{q}F_{q}\) where \(F_{q}=\{x\in F:(\max_{y\in Y}|(Gx)_{y}|)/|Gx|=q\}\). We can rewrite Equation 10 as:

\[\operatorname{err}(h) \geq 1-\int_{F}q|Gx|p(x)dx \tag{11}\] \[\geq 1-\sum_{q}\int_{F_{q}}q|Gx|p(x)dx\] (12) \[\geq 1-\sum_{q}q\int_{F_{q}}|Gx|p(x)dx. \tag{13}\]

Equation 11 uses the definition of \(q\). Equation 12 separates the integral over \(F\) into the partition of \(F\). Equation 13 moves \(q\) out from the integral because it is a constant inside the integral. Consider the definition of \(c_{q}\), we have:

\[c_{q} =\mathbb{P}(x\in X_{q})\] \[=\int_{X_{q}}p(x)dx\] \[=\int_{F_{q}}\int_{Gx}p(z)dzdx \tag{14}\] \[=\int_{F_{q}}|Gx|p(x)dx. \tag{15}\]

Equation 14 uses \(X_{q}=\bigcup_{x\in F_{q}}Gx\). Equation 15 uses \(p(x)=p(gx)\). Now we can write Equation 13 as:

\[\operatorname{err}(h)\geq 1-\sum_{q}qc_{q}.\]Proof of Theorem 4.9

Define \(q(gx)\in\mathbb{R}^{n\times n}\) such that

\[Q_{Gx}q(gx)=p(gx)\rho_{Y}(g)^{T}\rho_{Y}(g)\alpha(x,g). \tag{16}\]

In particular, \(q(gx)\) exists when \(Q_{Gx}\) is full rank. It follows that \(Q_{Gx}\int_{G}q(gx)dg=Q_{Gx}\). Moreover, \(Q_{Gx}\) and \(q(gx)\) are symmetric matrix.

Proof.: The error function (Equation 1) can be written

\[\mathrm{err}(h) =\mathbb{E}_{x\sim p}[||f(x)-h(x)||_{2}^{2}]\] \[=\int_{X}p(x)||f(x)-h(x)||_{2}^{2}dx\] \[=\int_{x\in F}\int_{g\in G}p(gx)||f(gx)-h(gx)||_{2}^{2}\alpha(x,g )dgdx.\]

Denote \(e(x)=\int_{G}p(gx)||f(gx)-h(gx)||_{2}^{2}\alpha(x,g)dg\). Since \(h\) is \(G\)-equivariant, for each \(x\in F\) the value \(c=h(x)\in\mathbb{R}^{n}\) of \(h\) at \(x\) determines the value of \(h\) across the whole orbit \(h(gx)=gh(x)=gc\) for \(g\in G\). Then \(e(x)\) can be written

\[e(x) =\int_{G}p(gx)||f(gx)-gc||_{2}^{2}\alpha(x,g)dg\] \[=\int_{G}p(gx)||g(g^{-1}f(gx)-c)||_{2}^{2}\alpha(x,g)dg\] \[=\int_{G}(g^{-1}f(gx)-c)^{T}p(gx)g^{T}g\alpha(x,g)(g^{-1}f(gx)-c)dg\] \[=\int_{G}(g^{-1}f(gx)-c)^{T}Q_{Gx}q(gx)(g^{-1}f(gx)-c)dg. \tag{17}\]

Taking the derivative of \(e(x)\) with respect to \(c\) we have

\[\frac{\partial e(x)}{\partial c} =\int_{G}\Big{(}(Q_{Gx}q(gx))^{T}+(Q_{Gx}q(gx))\Big{)}(c-g^{-1}f( gx))dg\] \[=\int_{G}2Q_{Gx}q(gx)(c-g^{-1}f(gx))dg.\]

Setting \(\partial e(x)/\partial c=0\) we can find an equation for \(c^{*}\) which minimizes \(e(x)\)

\[Q_{Gx}\int_{G}q(gx)dg\cdot c^{*} =Q_{Gx}\int_{G}q(gx)g^{-1}f(gx)dg\] \[Q_{Gx}c^{*} =Q_{Gx}\mathbf{E}_{G}[f,x]. \tag{18}\]

Substituting \(c^{*}\) into Equation 17 we have

\[e(x)\geq \int_{G}(g^{-1}f(gx)-c^{*})^{T}Q_{Gx}q(gx)(g^{-1}f(gx)-c^{*})dg\] \[= \int_{G}(g^{-1}f(gx))^{T}Q_{Gx}q(gx)(g^{-1}f(gx))\] \[-\Big{(}c^{*T}Q_{Gx}q(gx)g^{-1}f(gx)\Big{)}^{T} \tag{19}\] \[-c^{*T}Q_{Gx}q(gx)g^{-1}f(gx)\] \[+c^{*T}Q_{Gx}q(gx)c^{*}dg.\]

The term \(\int_{G}e^{*T}Q_{Gx}q(gx)g^{-1}f(gx)dg\) could be simplified as \[\int_{G}c^{*T}Q_{Gx}q(gx)g^{-1}f(gx)dg=\int_{G}\mathbf{E}_{G}[f,x]Q_{Gx} q(gx)g^{-1}f(gx)dg. \tag{20}\]

Notice that \(Q_{Gx}\), and \(q(gx)\) are symmetric matrix

\[\int_{G}c^{*T}Q_{Gx}q(gx)c^{*}dg =\int_{G}c^{*T}q(gx)Q_{Gx}c^{*}dg\] \[=\int_{G}\mathbf{E}_{G}^{T}[f,x]Q_{Gx}q(gx)\mathbf{E}_{G}[f,x]dg.\]

Thus Equation 19 becomes

\[e(x)\geq \int_{G}(g^{-1}f(gx))^{T}Q_{Gx}q(gx)(g^{-1}f(gx))\] \[-\left(\mathbf{E}_{G}^{T}[f,x]Q_{Gx}q(gx)g^{-1}f(gx)\right)^{T}\] \[-\mathbf{E}_{G}^{T}[f,x]Q_{Gx}q(gx)g^{-1}f(gx)\] \[+\mathbf{E}_{G}^{T}[f,x]Q_{Gx}q(gx)\mathbf{E}_{G}[f,x]dg\] \[=\int_{G}p(gx)||f(gx)-g\mathbf{E}_{G}[f,x]||_{2}^{2}\alpha(x,g)dg.\]

Taking the integral over the fundamental domain \(F\) we have

\[\operatorname{err}(h) =\int_{F}e(x)\] \[\geq\int_{F}\int_{G}p(gx)||f(gx)-g\mathbf{E}_{G}[f,x]||_{2}^{2} \alpha(x,g)dgdx. \tag{21}\]

## Appendix E Proof of Corollary 4.10

Proof.: When \(\rho_{Y}\) is an orthogonal representation, we have \(\rho_{Y}(g)^{T}\rho_{Y}(g)=I_{n}\), i.e., the identity matrix. Then \(q(gx)\) can be written as \(q(gx)=s(gx)\mathrm{Id}\) where \(s(gx)\) is a scalar. Since \(\int_{G}q(gx)dg=\mathrm{Id}\), we can re-define \(q(gx)\) to drop \(\mathrm{Id}\) and only keep the scalar, then \(q_{x}(g)\) can be viewed as a probability density function of \(g\) because now \(\int_{G}q_{x}(g)=1\).

With \(q_{x}(g)\) being the probability density function, \(\mathbf{E}_{G}[f,x]\) (Equation 6) naturally becomes the mean \(\mathbb{E}_{G}[f_{x}]\) where \(g\sim q_{x}\).

Now consider \(e(x)=\int_{G}p(gx)||f(gx)-g\mathbf{E}_{G}[f_{x}]||_{2}^{2}\alpha(x,g)dg\) in Theorem 4.9, it can be written as

\[e(x)= \int_{G}p(gx)||f(gx)-g\mathbb{E}_{G}[f_{x}]||_{2}^{2}\alpha(x,g)dg\] \[= \int_{G}p(gx)||g(g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}])||_{2}^{2} \alpha(x,g)dg\] \[= \int_{G}p(gx)(g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}])^{T}\rho_{Y}(g)^{ T}\rho_{Y}(g)(g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}])\alpha(x,g)dg.\]

Since \(\rho_{Y}(g)^{T}\rho_{Y}(g)=I_{n}\), we have

\[e(x)= \int_{G}p(gx)(g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}])^{T}(g^{-1}f(gx)- \mathbb{E}_{G}[f_{x}])\alpha(x,g)dg\] \[= \int_{G}p(gx)||g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}]||_{2}^{2}\alpha( x,g)dg. \tag{22}\]From Equation 5 we have \(p(gx)\alpha(x,g)=Q_{Gx}q(gx)\). Substituting in Equation 22 we have

\[e(x)= \int_{G}Q_{Gx}q(gx)||g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}]||_{2}^{2}dg.\]

Since \(Q_{Gx}=\int_{G}p(gx)\alpha(a,g)dg\) when \(\rho_{Y}(g)^{T}\rho_{Y}(g)=I_{n}\), we have

\[e(x)= Q_{Gx}\int_{G}q_{x}(g)||g^{-1}f(gx)-\mathbb{E}_{G}[f_{x}]||_{2}^{2}dg\] \[= Q_{Gx}\mathbb{V}_{G}[f_{x}]. \tag{23}\]

Now consider \(Q_{Gx}\) (Equation 5), when \(\rho_{Y}(g)^{T}\rho_{Y}(g)=I_{n}\), it can be written

\[Q_{Gx}= \int_{G}p(gx)\alpha(x,g)dg\] \[=\int_{Gx}p(z)dz\] \[=p(Gx).\]

Replacing \(Q_{Gx}\) with \(p(Gx)\) in Equation 23 then taking the integral of \(e(x)\) over the fundamental domain gives the result.

## Appendix F Lower Bound of Equivariant Regression when \(\rho_{Y}=\mathrm{Id}\)

**Proposition F.1**.: _When \(\rho_{Y}=\mathrm{Id}\), the error of \(h\) has lower bound \(\mathrm{err}(h)\geq\int_{F}p(Gx)\mathbb{V}_{Gx}[f]dx\), which is the same as Theorem 4.8._

Proof.: Consider Equation 5, when \(\rho_{Y}(g)=\mathrm{Id}\), we have

\[Q_{Gx}=\int_{G}p(gx)\alpha(x,g)dg.\]

Exchange the integration variable using \(z=gx\) we have

\[Q_{Gx}=\int_{Gx}p(z)dz. \tag{24}\]

Consider \(\mathbb{E}_{G}[f_{x}]=\int_{G}q_{x}(g)g^{-1}f(gx)dg\). When \(\rho_{Y}(g)=\mathrm{Id}\), it becomes

\[\mathbb{E}_{G}[f_{x}]=\int_{G}q(gx)f(gx)dg.\]

Substituting \(q(gx)\) with Equation 5, considering \(\rho_{Y}(g)=\mathrm{Id}\), we have

\[\mathbb{E}_{G}[f_{x}]=\int_{G}Q_{Gx}^{-1}p(gx)f(gx)\alpha(x,g)dg.\]

Exchange the integration variable using \(z=gx\) we have

\[\mathbb{E}_{G}[f_{x}]=\int_{Gx}Q_{Gx}^{-1}p(z)f(z)dz.\]

Substituting Equation 24 we have

\[\mathbb{E}_{G}[f_{x}] =\int_{Gx}\frac{p(z)}{\int_{Gx}p(z)dz}f(z)dz\] \[=\mathbb{E}_{Gx}[f].\]

Similarly, we can proof \(\mathbb{V}_{G}[f_{x}]=\mathbb{V}_{Gx}[f]\), thus when \(\rho_{Y}=\mathrm{Id}\), Corollary 4.10 is Theorem 4.8.

## Appendix G Rademacher Complexity of Harmful Extrinsic Equivariance Example

Let \(S=\{x^{1},x^{2},x^{3},x^{4}\}\), where the labels are \(y^{1},y^{2}=+1\) and \(y^{3},y^{4}=-1\). We consider two model classes \(\mathcal{F}_{N}\), the set of all linear models, and \(\mathcal{F}_{E}\), the set of all linear models equivariant to \(C_{2}\), and compute their empirical Rademacher complexity on \(S\).

For the data \(S\), an extrinsically equivariant linear model class has lower empirical Rademacher complexity than its unconstrained linear counterpart, demonstrating that extrinsic equivariance can be harmful to learning.

## Appendix H Additional Experiments

### Swiss Roll Experiment

Figure 11 and Figure 12 show the actual data distribution for the Swiss Roll experiment in Section 6.1. In the incorrect distribution, the data in the two \(z\) planes form two spirals with different labels but the same shape. The equivariance is incorrect because if we translate one spiral to the other spiral's plane, they will overlap but their labels are different. In the correct distribution, there are two different 'dashed' spirals copied into two \(z\)-planes. The equivariance is correct because after a \(z\)-translation, both the data and their labels exactly overlap. In all three cases, we assume the data has a uniform distribution. Figure 12(b) shows the ternary plot of MLP for all different \(c,ir,er\), where the performance of MLP decreases as the correct ratio increases. Figure 12(a) shows an inverse trend: the

Figure 11: The correct, incorrect, and extrinsic data distribution in the Swiss Roll experiment.

performance of INV increases as the correct ratio increases. Moreover, both extrinsic and incorrect equivariance harms the performance of INV, but incorrect equivariance is more devastating because the error is limited by a theoretical lower bound.

### Square Experiment

We consider the environment shown in Example 4.6. We vary \(m\in\{0.2,0.4,0.6,0.8,1\}\) and \(c\in\{0,0.2,0.4,0.6,0.8,1\}\). We train an \(u\)-invariant network and evaluate its test performance with the theoretical lower bound \(\operatorname{err}(h)\geq(1-c)\times(1-m)\). Figure 14 shows the test error of the trained network compared with the theoretical lower bound. The highest difference is below 3%, demonstrating the correctness of our theory.

### Regression Experiment

In this experiment, we validate our theoretical error lower bound for invariant and equivariant regression (Theorem 4.8, 4.9) in an environment similar to Example 4.11. Consider a regression task \(f:\mathbb{R}\times\mathcal{X}\rightarrow\mathbb{R}^{2}\) given by \((\theta,x)\mapsto y\), where \(\mathcal{X}=\{x_{0},x_{1},x_{2},x_{3}\}\). The group \(g\in G=C_{4}=\{e,r,r^{2},r^{3}\}\) acts on \((\theta,x)\) by \(g(\theta,x)=(\theta,gx)\) through permutation: \(x_{1}=rx_{0};x_{2}=rx_{1};x_{3}=\)

\begin{table}
\begin{tabular}{c c c} \hline \hline  & Invariant Network & Equivariant Network \\ \hline Empirical/Theoretical & 1.002 \(\pm\)0.000 & 1.001 \(\pm\)0.000 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Empirical \(\operatorname{err}(h)\) divided by theoretical \(\operatorname{err}(h)\) for invariant regression and equivariant regression. Results are averaged over 100 runs with different \(f\) for each regression. Empirical regression error matches theoretical error.

Figure 12: Data distribution example with different correct ratio (\(c\)), incorrect ratio (\(ir\)), and extrinsic ratio (\(er\)) values.

Figure 13: The ternary plot of the invariant network (a) and unconstrained network (b) with different correct, incorrect, and extrinsic ratio.

\(rx_{2};x_{0}=rx_{3}\). Let \(r^{k}\in G\) acts on \(y\) by \(\rho_{Y}(g)=\left(\begin{smallmatrix}\cos g&-\sin g\\ \sin g&\cos g\end{smallmatrix}\right)\) where \(g=k\pi/2\). Note that fixing a single value of \(\theta\) gives Example 4.11; in other words, this experiment has infinitely many orbits where each orbit is similar to Example 4.11.

We generate random polynomial function \(f\) that is not equivariant, i.e., \(\exists(\theta,x)\) s.t. \(g\cdot f(\theta,x)\neq\rho_{Y}(g)y\). Then we try to fit \(f\) using a \(G\)-invariant network and a \(G\)-equivariant network. We measure their error compared with the theoretical lower bound given by Theorem 4.8 and 4.9. As is shown in Table 2, both the invariant network and the equivariant network achieve an error rate nearly the same as our theoretical bound. The empirical error is slightly higher than the theoretical error due to the neural network fitting error. Please refer to I.4 for more experiment details.

## Appendix I Experiment Details

This section describes the details of our experiments. All of the experiment is performed using a single Nvidia RTX 2080 Ti graphic card.

### Swiss Roll Experiment

In the Swiss Roll Experiment in Section 6.1, we use a three-layer MLP for the unconstrained network. For the \(z\)-invariant network, we use a network with two DSS [37] layers to implement the \(z\)-invariance, each containing two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of \(10^{-3}\). The batch size is 128. In each run, there are 200 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and a maximum of 10000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.

### Square Experiment

In the Square Experiment in Section H.2, we use a network with two DSS [37] layers to implement the horizontal invariance, where each layer contains two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of \(10^{-3}\). The batch size is 128. In each run, there are 1000 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and a maximum of 10000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.

### Digit Classification Experiment

In the Digit Classification Experiment in Section 6.2, we use two similar five-layer convolutional networks for the \(D_{4}\)-invariant network and the CNN, where the \(D_{4}\)-invariant network is implemented using the e2cnn package [60]. Both networks have the similar amount of trainable parameters. We train the networks using the Adam [24] optimizer with a learning rate of \(5\times 10^{-5}\) and weight decay

Figure 14: Result of the square experiment in terms of the \(L_{1}\) distance between the network error and the theoretical lower bound in percentage. Each cell corresponds to an experiment with a particular correct ratio (\(c\)) and majority label ratio (\(m\)). Results are averaged over 10 runs.

of \(10^{-5}\). The batch size is 256. In each run, there are 5000 training data, 1000 validation data, and 1000 test data randomly sampled from the data distribution. The network is trained for a minimal of 50 epochs and a maximum of 1000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 50 epochs. We report the test success rate of the epoch model with the highest validation success rate.

### Regression Experiment

In the regression experiment, we validate our theoretical error lower bound for invariant and equivariant regression (Theorem 4.8, 4.9) by comparing empirical network fitting error and the theoretical fitting error of a function \(f\). Specifically, the function \(f\) maps a distance \(\theta\) and an index \(x\) pair to a vector \(y\):

\[f:\mathbb{R}\times\mathcal{X}\rightarrow\mathbb{R}^{2},\text{given by }(\theta,x )\mapsto y \tag{25}\]

where \(\mathcal{X}=\{x_{0},x_{1},x_{2},x_{3}\}\). The group \(g\in G=C_{4}=\{e,r,r^{2},r^{3}\}\) acts on \((\theta,x)\) by \(g(\theta,x)=(\theta,gx)\) through permuting the index \(x\): \(x_{1}=rx_{0};x_{2}=rx_{1};x_{3}=rx_{2};x_{0}=rx_{3}\). Let \(r^{k}\in G\) acts on vector \(y\) by rotation \(\rho_{Y}(g)=\left(\begin{smallmatrix}\cos g&-\sin g\\ \sin g&\cos g\end{smallmatrix}\right)\) where \(g=k\pi/2\).

We construct function \(f\) in the following way: for each \(x\in\mathcal{X}\), choose \(l_{x}:\mathbb{R}\rightarrow\mathbb{R}^{2}\) and define \(f(\theta,x)=l_{x}(\theta)\). Notice that when \(l_{gx}=\rho_{Y}(g)l_{x}(\theta)\), \(f\) is \(G\)-equivariant. We define \(l_{x}(\theta)=(p_{x}(\theta),q_{x}(\theta))\) where \(p_{x}\) and \(q_{x}\) are cubic polynomials of \(x\), i.g., \(p_{x}\) with coefficients \(a,b,c,d\) will be \(p_{x}=ax^{3}+bx^{2}+cx+d\). We choose \(p_{x}\) and \(q_{x}\) with different coefficients for each \(x\) such that \(f\) is not equivariant, i.e., \(l_{gx}\neq\rho_{Y}(g)l_{x}(\theta)\). For each run, we generate a function \(f\), sample data \(\theta,x\), and evaluate the data obtaining \(y\). Then we train neural networks using \(L2\) loss till converge. Eventually, we sample another set of data to evaluate the empirical \(L2\) error as well as the theoretical \(L2\) error.

### Robotic Experiment

In the robotic manipulation experiment, the state \(s\) is defined as a top-down RGBD image of the workspace centered at the gripper's position (Figure 15 middle). The action \(a=(x,y,z,\theta,\lambda)\) is defined as the change of position (\(x,y,z\)) and top-down orientation (\(\theta\)) of the gripper, with the

Figure 15: The robotic experiment setup and the \(D_{1}\)-equivariant policy network.

gripper open width (\(\lambda\)). For a \(D_{1}=\{1,g\}\) group where \(g\) represents a horizontal flip, the group action on the state space \(gs\) is defined as flipping the image; the group action on the action space \(ga\) is defined as flipping the \(y\) and \(\theta\) action and leaving the other action components unchanged, \(ga=(x,-y,z,-\theta,\lambda)\). We define a \(D_{1}\)-equivariant policy network \(\pi:s\mapsto a\) using e2cnn [60], where the output action of \(\pi\) will flip accordingly when the input image is flipped (Figure 15 bottom). We train the network using the Adam [24] optimizer with a learning rate of \(10^{-3}\) and weight decay of \(10^{-5}\). For each run, we train the network for a total of 20k training steps, where we perform evaluation for 100 episodes every 2k training steps. We report the highest success rate of the 10 evaluations as the result of the run.

We develop the experimental environments in the PyBullet [12] simulator, based on the BullatArm benchmark [57]. In the Stacking (correct equivariance) and Sorting (incorrect equivariance) experiment, we gather a total of \(400\) episodes of demonstrations, where \(400c\) of them are Stacking and the rest \(400(1-c)\) are Sorting. In evaluation, the task follows the same distribution, where \(100c\) of the evaluation episodes are Stacking and the rest are Sorting. Notice that the agent can distinguish the Stacking and Sorting tasks because the object colors are different for the two tasks (green and blue for stacking, yellow and red for sorting). In the Sorting (extrinsic equivariance) experiment, we also use \(400\) episodes of demonstrations.

Specifically, in Sorting, the cube and the triangle are initially placed randomly, within a distance of \(\pm 1.5cm\) from the horizontal mid-line of the workspace. The objective is to push the triangle at least \(9cm\) toward left and to push the cube at least \(9cm\) toward right, while ensuring that both objects remain within the boundaries of workspace. In Stacking, two blocks are randomly initialized on the floor of the workspace. The goal is to pick up the triangle and place it on top of the cube. The workspace has a size of \(30cm\times 30cm\times 25cm\).

Response:
## Review

### Summary

This paper investigates the behavior of equivariant models under conditions where the symmetries of the model do not align perfectly with those of the underlying data distribution. It extends prior work by Wang et al. (2023) by introducing the concepts of correct, incorrect, and extrinsic pointwise equivariance, allowing for a nuanced understanding of how these relationships impact model performance. The authors derive lower bounds on model error for classification and regression tasks, demonstrating how mismatches in symmetry can influence learning outcomes. Theoretical contributions are supported by a range of experiments that showcase the practical implications of these findings.

### Strengths

- The paper introduces novel theoretical concepts related to equivariance, extending existing frameworks to account for partial symmetries.
- It provides a comprehensive analysis of error bounds in equivariant models, supported by clear examples and experimental results.
- The writing is generally clear and well-structured, making complex ideas accessible.
- The practical relevance of the findings is emphasized, particularly concerning real-world applications of symmetry in machine learning.

### Weaknesses

- Certain definitions, such as those related to pointwise equivariance, could benefit from greater precision to clarify their dependence on specific models and contexts.
- The presentation of results may obscure the novelty of contributions, particularly the implications of pointwise definitions.
- Some experiments lack variety, especially concerning infinite symmetry groups, which could limit the demonstration of the new theoretical bounds.
- The discussion of extrinsic equivariance lacks a principled understanding of when it is beneficial or harmful in practice.

### Questions

- Can you clarify the definition of p and its relation to the underlying population distribution versus training data in the context of the paper?
- What specific scenarios can lead to extrinsic equivariance being harmful, and how can practitioners identify these cases?
- Could you expand on the third experiment and the role of state flipping in the analysis?

### Soundness

**Score:** 3

**Description:** Good: The theoretical results and proofs presented are sound, although some aspects may require further clarification or depth.

### Presentation

**Score:** 3

**Description:** Good: The paper is generally well-written and organized, although there are areas where clarity could be improved, especially regarding definitions.

### Contribution

**Score:** 3

**Description:** Good: The paper contributes significantly to the understanding of equivariant models, although the practical implications of some theoretical frameworks may need further exploration.

### Rating

**Score:** 7

**Description:** Accept: The paper is technically solid with high impact on the area of equivariant learning, but it requires minor improvements in clarity and depth of discussion.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a valuable extension of existing work on equivariant models, addressing important theoretical and practical issues in the field. Despite some weaknesses in clarity and depth of discussion, the overall contribution is significant, and the results are likely to provoke further research and application in the area. The decision to accept reflects the paper's strong technical foundation and its relevance to ongoing discussions in machine learning research.