Deep Neural Networks with Efficient Guaranteed Invariances
We address the problem of improving the performance and in particular the
sample complexity of deep neural networks by enforcing and guaranteeing
invariances to symmetry transformations rather than learning them from data.
Group-equivariant convolutions are a popular approach to obtain equivariant
representations. The desired corresponding invariance is then imposed using
pooling operations. For rotations, it has been shown that using invariant
integration instead of pooling further improves the sample complexity. In this
contribution, we first expand invariant integration beyond rotations to flips
and scale transformations. We then address the problem of incorporating
multiple desired invariances into a single network. For this purpose, we
propose a multi-stream architecture, where each stream is invariant to a
different transformation such that the network can simultaneously benefit from
multiple invariances. We demonstrate our approach with successful experiments
on Scaled-MNIST, SVHN, CIFAR-10 and STL-10.
Matthias Rath, Alexandru Paul Condurache
# Deep Neural Networks with Efficient Guaranteed Invariances

###### Abstract

We address the problem of improving the performance and in particular the sample complexity of deep neural networks by enforcing and guaranteeing invariances to symmetry transformations rather than learning them from data. Group-equivariant convolutions are a popular approach to obtain equivariant representations. The desired corresponding invariance is then imposed using pooling operations. For rotations, it has been shown that using invariant integration instead of pooling further improves the sample complexity. In this contribution, we first expand invariant integration beyond rotations to flips and scale transformations. We then address the problem of incorporating multiple desired invariances into a single network. For this purpose, we propose a multi-stream architecture, where each stream is invariant to a different transformation such that the network can simultaneously benefit from multiple invariances. We demonstrate our approach with successful experiments on Scaled-MNIST, SVHN, CIFAR-10 and STL-10.

## 1 Introduction

Deep Neural Networks (DNNs) are one of the core drivers of technological progress in various fields such as speech recognition, machine translation, autonomous driving or computer vision (LeCun et al., 2015). At the core of their success lies the ability to process large amounts of data to yield solutions with remarkable generalization properties (Lust and Condurache, 2020b). However, in many practical applications, the data is expensive to collect, store and label. Furthermore, it is rather difficult for humans to understand how such correlation-based methods work, which is a necessary first step in optimizing or adapting them to new application domains. Additionally, a certain degree of understanding and trust in the DNN's output is essential, e.g., when safety plays a major role (Lust and Condurache, 2020a).

Often, _prior knowledge_ about transformations that modify the desired output in a predictable way is available before training. Leveraging prior knowledge increases the interpretability of DNNs while also improving the sample complexity - hence reducing the amount of data needed to obtain a desired performance. When incorporating this valuable prior knowledge into deep learning architectures, it is advantageous to _guarantee_ the corresponding in- or equivariances. We differentiate between _invariance_, which is the property of a map to yield the same output for a transformed input and _equivariance_ which is the property of a map to preserve the transformation of the input, such that the output is transformed predictably. One common example of leveraging equivariance in DNNs are convolutional layers (Fukushima, 1980; LeCun et al., 1990). These are equivariant to translations and can be easily used to generate a translation-invariant representation (e.g. by pooling).

For many tasks, we can define a set of _symmetry transformations_ that affect the desired output in a predictable way. For example, in the case of image classification, symmetry transformations of an input object map it within the same image class and thus do not change the desired classification output. Typical examples are rotations, translations and scales. Enforcing invariance to such transformations as an inductive bias decreases the sample complexity by reducing the search space while training a DNN. Moreover, guaranteed invariances contribute towards gaining an intuition on how inference is conducted in the DNN. Cohen and Welling (2016) first used _group equivariant convolutions_ (G-Convs) in DNNs to enforce equivariance to transformations such as rotations and flips (Cohen and Welling, 2016; Worrall et al., 2017; Weiler et al., 2018; Weiler and Cesa, 2019) or scales (Xu et al., 2014; Kanazawa et al., 2014; Sosnovik et al., 2020, 2021). For classification DNNs, those equivariant layers are usually followed by a max pooling operation among the group and spatial dimensions
to obtain invariant features that are processed by the final classification layers. While max pooling guarantees invariance, it destroys important information that could be leveraged by a classifier and therefore lacks efficiency. Since the transfer from equi- to invariant features has not been extensively investigated, it promises further improvement capabilities.

Invariant Integration (II) is an algorithm to construct a complete feature space with respect to (w.r.t.) a transformation group (Schulz-Mirbach, 1992). So far, II has been used to replace the global spatial pooling operation for rotation-invariant classification DNNs. This resulted in an improved sample complexity by efficiently leveraging the available prior knowledge while adding targeted model capacity (Rath and Conducnache, 2020, 2022). However, II has not been extended to other relevant symmetry transformations such as _scales_.

Group-equivariant DNNs usually incorporate prior knowledge about a single transformation. It is an open challenge how to proceed when _multiple symmetries_ are involved because it may be impossible to solve the constraints needed to design transformation-steerable filters depending on the involved groups. Even when avoiding the constraints via interpolation methods, simply expanding the regular equivariant G-Convs is computationally inefficient since the representation grows multiplicatively. For example, for a kernel with 8 rotations and 4 scales, we would have to store \(8\cdot 4=32\) responses per kernel.

In this contribution, we extend the II framework beyond rotations and efficiently apply it to multiple transformations at once via a multi-stream architecture. Our **core contributions** are:

* We adapt rotation-II to also include flips to achieve **invariance** to the 2D Euclidean group **E(2)**.
* We **expand II towards scales**, thus covering a larger set of symmetry transformations.
* We address the issue of **multiple invariances** within a single architecture that effectively **combines** several streams, each one with **specific invariances** (see Figure 1). This significantly **extends the practical applicability** of **II**.
* We **evaluate** our approach on **Scaled-MNIST** and on the **real-world datasets** SVHN, CIFAR-10 and STL-10. On STL-10 using only labeled data, we report new **state-of-the-art** results.

## 2 Related Work

### Group-Equivariant Neural Networks

_Group-equivariant convolutional_ layers were proposed by Cohen and Welling (2016) and applied to discrete 90\({}^{\circ}\) rotations and flips by transforming the filters and storing all responses among a _group channel_. This approach uses the _regular_ group-representation. Extensions apply this principle to finer-grained rotations via interpolation (Bekkers et al., 2018; Hoogeboom et al., 2018), rotation-steerable filters (Weiler et al., 2018; Weiler and Cesa, 2019) or by learning all rotated versions of a filter with invariant coefficients (Diaconu and Worrall, 2019). The maximum response can be stored as the orientation in a vector field (Marcos et al., 2017). This is closely related to the _irreducible representation_ which achieves continuous rotation-equivariance via complex-valued responses (Worrall et al., 2017). In general, it has been proven that G-Convs are the most general equivariant linear map and a necessary condition for equivariant DNNs (Kondor and Trivedi, 2018; Cohen et al., 2019; Esteves, 2020).

Besides rotations, in- or equivariance to scale transformations plays a major role in many practical applications. In- or equivariance can again be achieved by sharing filters among different scales using bi-linear interpolation (Xu et al., 2014), scaling the input (Kanazawa et al., 2014) or processing the maximum response and the corresponding scale as a vector field (Marcos et al., 2018). Filters

Figure 1: Triple-stream invariant Wide-ResNet16-4 architecture. Includes standard convolutions (grey), rotation-flip-steerable convolutions (E(2), orange), scale-steerable convolutions (blue), invariant integration layers (red), a weighted sum (green) and fully connected layers (purple). Residual shortcut connections are omitted for clarity.
for scale-equivariant convolutions can be constructed using scale-steerable filters with log-radial harmonics (Ghosh and Gupta, 2019), Hermite polynomials (Sosnovik et al., 2020), optimized discrete bases (Sosnovik et al., 2021), or with separable Fourier-Bessel bases (Zhu et al., 2019). A scale-equivariant G-Conv operating on scale-spaces was introduced in (Worrall and Welling, 2019).

Other work investigates general input domains such as the 3D Euclidean space (Worrall and Brostow, 2018; Weiler et al., 2018; Cesa et al., 2022), spheres (Cohen et al., 2018; Coors et al., 2018; Esteves et al., 2020; Defferrard et al., 2020; Kondor et al., 2018; Jiang et al., 2019; Shakerinava and Ravanbakhsh, 2021), general manifolds (Cohen et al., 2019a; Finzi et al., 2020), or general groups (Bekkers, 2020; Finzi et al., 2021). Moreover, equivariant versions of non-linear maps such as attention and transformers have been introduced (Fuchs et al., 2020, 2021; Romero and Hoogendoorn, 2020; Romero et al., 2020; Romero andordonnier, 2021; Hutchinson et al., 2021; He et al., 2021). The non-linearities and sub-sampling layers used in equivariant CNNs have been investigated in (Franzen and Wand, 2021; Xu et al., 2021).

### Invariant Neural Networks

When solving tasks that require invariance, group-equivariant DNNs are usually followed by a global (max) pooling layer among the group and spatial dimensions. While max pooling guarantees invariance, it is affected by a loss of information, since all but the maximum value are discarded. Obtaining invariance in a more sophisticated way thus promises to further improve invariant DNNs.

One approach to achieve invariance while also ensuring separability is _Invariant Integration_ (II), proposed by Schulz-Mirbach (1992, 1994). II is an algorithm to construct a complete feature space w.r.t. a group, i.e., similar patterns are mapped to the same point while distinct patterns are mapped to distinct points. II has been used in combination with conventional machine learning classifiers for image classification (Schulz-Mirbach, 1995), event detection within a cascaded feature extractor to obtain invariance to anthropometric changes (Condurache and Mertins, 2012) or robust speech recognition (Muller and Mertins, 2009, 2010, 2011). Rotation-II has been used to replace the global spatial pooling layer within rotation-invariant deep learning architectures (Rath and Conducache, 2020, 2022) and shown to further increase the sample complexity of such networks. Puny et al. (2021) solve the group average, which II is based on, for larger, intractable groups by integrating over a subset. They applied their method to classification for motion-invariant point clouds and graph DNNs integrating over the whole DNN. The invariant integral has also been used to prove that in- and equivariance improve generalization when the target distribution is in- or equivariant (Elesedy and Zaidi, 2021).

Whereas Rath and Conduche (2020, 2022) focused on II for the group of rotations, we expand this framework to scales as well as flips (using E(2)). Thereby, we show that the framework can be expanded to general group transformations and generally improves the sample complexity of group-equivariant CNNs in classification tasks. Most related work focuses on single transformation groups. Through our multi-stream architecture, we propose a novel approach that allows our network to learn the best possible combination of invariant features among multiple transformations at once. Another method that combines equivariance to both rotations and scales is the Polar Transformer Network (PTN) which process inputs in the polar coordinate system (Esteves et al., 2018). However, working in polar coordinates, although advantageous for rotations, may prove to be detrimental to translation equivariance. Indeed PTNs are by design invariant to translation, but it is not clear how much other relevant information is destroyed. At the same time, Spatial Transformer Networks (Jaderberg et al., 2015) in general cannot offer invariance guarantees, as they rely on the localization network to learn the correct transformation. The invariance is not fully guaranteed but approximated w.r.t the estimated transformation (STNs) or object center (PTNs). The same is valid for deformable (Dai et al., 2017) and tiled convolutions (Le et al., 2010).

## 3 Theoretical Background

### In- and Equivariance

In- and equivariance are mathematical concepts describing the behavior of a map \(f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}\) under transformations of the input that can be modeled using the mathematical abstraction of a group. A group is a set \(G\) equipped with a group operation \(\cdot:G\times G\to G\) fulfilling the four group axioms: closure, associativity, identity and invertibility. The map \(f\) is called equivariant w.r.t. \(G\), if left group actions \(L_{g}x\) acting on the input \(x\in\mathbb{R}^{n}\) result in predictable changes \(L_{g^{\prime}}\) of the output

\[\forall x\;\forall g\;\exists g^{\prime}\;\text{s.t.}\;f(L_{g}x)=L_{g^{\prime }}f(x), \tag{1}\]

where the left group action \(L_{g}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}\) is defined for each group element \(g\in G\). The group elements \(g\) and \(g^{\prime}\) are not necessarily equal, i.e., the transformation of the output can be different from the one applied on the input, but is predictable. If the output does not change, i.e. \(\forall x\;\forall g,\)\(f(L_{g}x)=f(x)\), the function is called invariant.

In the context of CNNs, the network and the layers are maps between feature spaces that can be described by \(f:\mathbb{Z}^{2}\rightarrow\mathbb{R}^{n}\). The effect of input transformations on the feature maps and outputs can thus be studied using Group Theory. In the course of the paper, we use left actions on the input \(x\) by lifting the left group action of \(G\) on \(\mathbb{Z}^{2}\) via
\(L_{g}x(y)=x(g^{-1}y)\) where \(y\in\mathbb{Z}^{2}\) are the pixel coordinates. The transformed input is equivalent to the original input at the point \(g^{-1}y\) that gets mapped to \(y\) by \(g\).

### Group-Equivariant Convolutions

Group-equivariant convolutions are the most general linear map achieving equivariance to a transformation group \(G\)(Kondor and Trivedi, 2018; Cohen et al., 2019b; Esteves, 2020). Cohen and Welling (2016) first used G-Convs in the context of DNNs to learn representations with guaranteed equivariant to group transformations. The continuous G-Conv of two functions \(f\) and \(\psi\) is defined as

\[(f\star_{G}\psi)(u)=\int_{g\in G}f(g)\psi(u^{-1}g)d\mu(g), \tag{2}\]

where \(d\mu(g)\) is the Haar measure with \(\int_{g\in G}d\mu(g)=1\). The in- and output are defined on the group itself with \(g,u\in G\). The _standard convolution_ is a special case of G-Convs where the elements are given by \(y,t\in\mathbb{Z}^{2}\) and the inverse action \(t^{-1}y\) results in shifts \(y-t\). A DNN is group equivariant, if and only if each of its mappings is equivariant to or commutes with the group (Kondor and Trivedi, 2018). In many cases, the transformation group \(G\) can be split into the translation group T defined on \(\mathbb{Z}^{2}\) and the corresponding quotient group \(H=G/T\). In order to achieve equivariance to \(G\), a \(H\)-equivariant convolution can be applied at all spatial locations of the input using a standard convolution. Different representations can be used to compute and store the equivariant features. _Irreducible_ representations require the minimal possible number of parameters, but often involve complex calculations. For 2D rotations, using irreducible representations results in complex-valued feature spaces where the orientation information is stored via the complex phase. The _regular_ representation of a discrete group stores all responses of transformed filters among an additional channel called _group channel_.

To compute regular G-Convs, either the input or the filters need to be transformed for all \(g\in G\). A simple method is to transform the filters using interpolation methods such as bi-linear interpolation. However, this introduces sampling artifacts and consequently weakens the equivariance guarantees. An alternative approach is to use transformation- steerable filters, which have been first introduced for rotations (Freeman and Adelson, 1991). While steerable filters introduce a computational overhead compared to simpler interpolation methods, arbitrarily transformed versions can be calculated in closed-form and are thus not afflicted by sampling effects. This concept can be effectively used for G-Convs by restricting the learned filters to linear combinations of steerable basis filters. Weiler et al. (2018b) built steerable filters for rotation G-Convs using a Gaussian kernel which Weiler and Cesa (2019) expanded to the general E(2)-group. Sosnovik et al. (2020, 2021) constructed scale-steerable filter CNNs using 2D Hermite polynomials or a learned discrete basis. We use the state-of-the-art methods for rotation- (E(2)-STCNNs, Weiler and Cesa 2019) and scale-invariant (DISCO, Sosnovik et al. 2021) tasks as our baseline.

### Invariant Representations in DNNs

Invariance plays a major role in many DNN applications. For the example of classification, input transformations that do not change the desired class output should not change the learned feature space. Group-equivariant DNNs typically use pooling to transfer from equi- to invariant representations. When operating on regular G-Convs, the pooling procedure is two-fold: Pooling among the transformation channel creates an equivariant representation where an input-transformation induces the same transformation in the feature space; and pooling among the spatial dimension obtains the final invariance. For closed groups, such as rotations and flips, spatial average or max pooling can be used to obtain invariant representations. This is different for the scale group, where average pooling over the spatial dimension does not lead to invariant, but rather homogeneous features, i.e., scaling by \(s\) modifies the output by multiplying with \(s^{2}\).

### Invariant Integration

Invariant Integration is an algorithm to construct a complete feature space \(\mathcal{F}\) w.r.t. a transformation group \(G\) introduced in Schulz-Mirbach (1992). A feature space is complete, if all equivalent patterns w.r.t. the transformation are mapped to the same point while all distinct patterns are mapped to different points. For this mapping, II uses the group average \(A[f](x)\) which integrates over all possible transformations \(g\in G\) of an input \(x\) processed by a polynomial \(f\)

\[A[f](x)=\int_{g\in G}f(L_{g}x)d\mu(g). \tag{3}\]

In our case, \(x\) is the output of the final feature map after pooling among the group dimension. For \(f\), Schulz-Mirbach (1994); Schulz-Mirbach (1995) used the set of monomials \(m(x)=\prod_{i=1}^{M}x_{i}^{b_{i}}\) with \(\sum_{i}b_{i}\leq|G|\), defined as a product of individual signal values \(x_{i}\) with exponents \(b_{i}\), which have been shown to be a good choice to maintain a high expressiveness of the invariant features (Noether, 1916). For 2D rotations and translations, II with monomials within a local neighborhood defined by distances \(d_{i}\) results in

\[A[\text{m}](x)=\frac{1}{N_{\phi}UV}\sum_{\phi,t}\prod_{i=1}^{M}x[t-L_{\phi}d_{i }]^{b_{i}}. \tag{4}\]

Rath and Condurache (2020) applied II on top of equivariant G-Convs within a DNN. The II layer is differentiable, if \(x_{i}>0\)\(\forall i\), which allows for an end-to-end optimization of
the DNN via backpropagation. To select a meaningful subset of monomials, an iterative algorithm based on the least square solution of a linear classifier can be used. However, a pruning-based approach leads to a more streamlined training procedure (Rath and Condurache, 2022). Additionally, selecting the monomials can be avoided by replacing them with alternatives well-known in deep learning literature such as self-attention, a weighted sum (WS) or a multi-layer-perceptron. For rotation-II, the special case using a WS achieves the best performance while being easier to train (Rath and Condurache, 2022). II using a WS with a learnable kernel \(\psi\in\mathbb{R}^{k\times k}\) applied to \(N_{\phi}\) finite rotations \(\phi\in[0^{\circ},\frac{360^{\circ}}{N_{\phi}},\ldots]\) obtained using bi-linear interpolation with input dimensions \(U\times V\) is defined as

\[A[\text{WS}](x)=\frac{1}{N_{\phi}UV}\sum_{\phi,t}\sum_{y\in\mathbb{Z}^{2}}x(y) L_{\phi}\psi(y-t). \tag{5}\]

## 4 Method

In this section, we extend rotation-II using a WS to the E(2)-group involving rotations and flips (Section 4.1). We then show why II cannot be straightforwardly applied to scale transformations and introduce an alternative to obtain scale-invariants based on II (Section 4.2). We then propose a scale-invariant CNN including Scale-II (Section 4.3). Finally, we introduce a multi-stream DNN architecture that efficiently combines invariances to multiple transformations within a single DNN (Section 4.4).

### E(2)-Invariant Integration

Equation 5 can straightforwardly be expanded to each discrete subgroup of E(2) (as used by Weiler and Cesa 2019), which contains flips \(L_{f}\psi(y)\) in addition to rotations

\[A[\text{WS}](x)=\frac{1}{2N_{\phi}UV}\sum_{f,0,t}\sum_{y\in\mathbb{Z}^{2}}x(y )L_{\phi}L_{f}\psi(y-t). \tag{6}\]

### Scale-Invariant Integration

In images, objects naturally appear at different scales, e.g., due to variable camera-to-object distances. Hence, DNNs for object classification or detection benefit from invariance to scales. We propose to use II in combination with a scale-equivariant CNN to obtain scale-invariant features. In comparison to the rotation group, discrete scale transformations are not circular and non-invertible due to the loss of information (e.g. during down-scaling). Thus, scales do not satisfy all group axioms and can only be modeled as a semi-group. Schulz-Mirbach (1992) demonstrated that it is impossible to construct invariants by integrating over the scale semi-group while at the same time achieving separability. This prohibits constructing a complete feature space w.r.t scales using the standard II approach. Nevertheless, the group average w.r.t. translations is a homogeneous function w.r.t. scales when using polynomials (Schulz-Mirbach, 1994). This means that the effect of scaling by \(s\) on the features obtained using translation-II is defined as \(A[f](L_{s}x)=s^{K}A[f](x)\) where the order \(K\) is defined by the polynomial order of \(f\) with the scale operator \(L_{s}[f](y)=f(s^{-1}y)\ \forall s>0\).

As shown by Schulz-Mirbach (1994), a complete feature space w.r.t. the scale-translation semi-group \(G_{S}=S\rtimes T\) can be calculated by dividing homogeneous functions of the same order. When using monomials \(m\), one resulting scale-invariant integral is given by dividing monomials of the same order \(\sum b_{1,i}=\sum b_{2,i}\) with \(t\in\mathbb{Z}^{2}\)

\[A_{G_{S}}[m](x)=\frac{A_{T}[m_{1}](x)}{A_{T}[m_{2}](x)}=\frac{\sum_{t}\prod_{i }x(t-d_{1,i})^{b_{1,i}}}{\sum_{t}\prod_{i}x(t-d_{2,i})^{b_{2,i}}}. \tag{7}\]

The special case of translation-II that combines values within a fixed neighborhood using a learnable WS as function \(f\) results in a standard convolution followed by Average Pooling and is equivalent to a polynomial of order 1. Consequently, we can choose a divisor of the same homogeneous order to obtain a scale-invariant representation and use the mean of the feature map. We thus introduce the WS-based scale-group average \(A_{G_{S}}\) based on Average Pooling over a standard convolution without bias with translations \(L_{t}\), \(y,t\in\mathbb{Z}^{2}\) divided by the mean

\[A_{G_{S}}[\text{WS}](x)=\frac{A_{T}[\text{WS}](x)}{\sum_{y}x(y)}=\frac{\sum_{ y}\sum_{t}x(y)\psi(y-t)}{\sum_{y}x(y)}. \tag{8}\]

**Proof of Invariance.** Since translation-II and the mean are both homogeneous w.r.t. scales with factor \(s^{2}\), it is easy to see that dividing them leads to a scale-invariant solution

\[\frac{A_{T}[f](L_{s}x(y))}{\sum_{y}L_{s}x(y)}=\frac{s^{2}A_{T}[f](x)}{s^{2} \sum_{y}x(y)}=\frac{A_{T}[f](x)}{\sum_{y}x(y)}. \tag{9}\]

### Application to DNNs

Inspired by the work on rotations in Rath and Condurache (2020, 2022), we apply E(2)- and Scale-II (Formulas 6 - 8) on top of the corresponding equivariant features learned using regular G-Convs with steerable filters: E(2)-STCNNs (Weiler and Cesa, 2019) and DISCO (Sosnovik et al., 2021). Those features are processed via max pooling among the group dimension. II then replaces the spatial pooling operation to obtain invariant features that are in turn processed by the final classification layers. For E(2)-II, we use the WS approach. For Scale-II, we investigate both proposed variants. For Scale-II with monomials, we follow Rath and Condurache (2022) and randomly select monomials that are iteratively pruned during training to find the most relevant ones. We ensure the same monomial order between dividated and divisor by normalizing the divisor's exponents with \(\frac{\sum_{i}b_{1,i}}{\sum_{i}b_{2,i}}\). Additionally, \(x_{i}>0\) ensures a
differentiable solution. For Scale-II with WS, each convolution depends on all input channels. Consequently, we divide by the mean over all input channels for the WS-based II. Moreover, it is important that \(\sum_{y}x(y)>0\). Hence, we apply both II variants to the output of a ReLU layer minimized to a small value \(\epsilon>0\).

### Multi-Stream Invariance

To obtain features with guaranteed invariance to multiple transformations, we implement a DNN architecture with multiple streams. Each stream is invariant to a dedicated transformation enforced using G-Convs and II (see Figure 1). While multiple transformations could in theory be embedded into a single network, the regular representations the network needs to process would grow with \(\mathcal{O}(N\cdot M)\) for group sizes \(N\) and \(M\). In contrast, using a dedicated stream per transformation only increases the number of representations by \(\mathcal{O}(N+M)\ll\mathcal{O}(N\cdot M)\).

The input is processed separately by each transformation-invariant stream using G-Convs with steerable filters. II is applied on top of the learned scale- and E(2)-equivariant features to obtain invariant representations. For the standard convolution stream (std.) we use average pooling. Thus, we have \(\mathbf{x}_{j}\in\mathbb{R}^{C_{j}}\) with \(j\in\{\text{e2},\text{scale},\text{std}\}\). We combine two or three streams, each one invariant to either rotations and flips, scales or translations (std.), using two steps. First, we map all features to the same dimension using a linear map \(\mathbf{\tilde{x}}_{j}=\mathbf{W}_{j}\mathbf{x}_{j}\) with \(\mathbf{W}_{j}\in\mathbb{R}^{C_{\text{map}}\times C_{j}}\). We then combine these streams via a normalized learnable WS, e.g. for the case of three streams \(\mathbf{x}_{\text{combined}}=\mathbf{w}_{\text{e2}}\circ\mathbf{\tilde{x}}_{ \text{e2}}+\mathbf{w}_{\text{scale}}\circ\mathbf{\tilde{x}}_{\text{scale}}+ \mathbf{w}_{\text{std}}\circ\mathbf{\tilde{x}}_{\text{std}}\) with \(\mathbf{w}_{j}\in\mathbb{R}^{C_{\text{map}}}\) initialized to 1 and normalized s.t. \((\mathbf{w}_{\text{e2}}+\mathbf{w}_{\text{scale}}+\mathbf{w}_{\text{std}})_{i}=1\) with \(i=1,\dots,C_{\text{map}}\) and the Hadamard product 0 inspired by the learnable channel-wise scaling used in BatchNorm layers (Ioffe and Szegedy, 2015). This approach allows to combine the invariant features with an explicitly learned factor s.t. the network can learn, which invariances are the most relevant for the corresponding task.

Each stream is pre-trained individually. We then combine the architecture, freeze all convolutional weights and only train the linear maps \(\mathbf{W}_{j}\), the WS \(\mathbf{w}_{j}\) and the classification weights \(\mathbf{w}_{\text{out}}\in\mathbb{R}^{C_{\text{map}}\times n_{c}}\) with \(n_{c}\) classes. The combination head worked best, when mapping the other streams to the E(2)-output, i.e., \(C_{\text{map}}=C_{\text{e2}}\) and keeping \(\mathbf{W}_{\text{e2}}\) fixed as the identity. Appendix B.1 provides results with different combinations, e.g., mapping all streams or concatenation. Our dedicated training procedure allows to fine-tune each stream individually, which provides a good initialization point for the combination head and further improves the sample complexity compared to a full end-to-end training (cf. Appendix B.2). While this causes an overhead at train time, all operations can easily be fused into a single network at inference time.

## 5 Experiments & Discussion

The proposed scale-II algorithm is evaluated on Scaled-MNIST (Sohn and Lee, 2012) and the invariant multi-stream networks on SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky, 2009) and STL-10 (Coates et al., 2011). We use the full dataset and limited subsets with \(N_{t}\) samples to assess the sample complexity. The subsets are sampled with constant class balance and are the same for all variants. For Scaled-MNIST, we use \(N_{t}\in\{10,50,100,500,1k,5k,10k,12k\}\). For SVHN and CIFAR-10 we use \(N_{t}\in\{100,500,1k,5k,10k,50k\}\).

For SVHN, CIFAR-10 and STL-10 we use Wide-ResNets (WRNs, Zagoruyko and Komodakis 2016) as backbone. We use the respective _standard data augmentations_: scales for Scaled-MNIST, no augmentations for SVHN, shifts and crops for CIFAR-10, and shifts, crops and Cutout (Devries and Taylor, 2017) for STL-10. For all single-streams, the number of trainable parameters is constant. We report results for the multi-stream architectures with full streams and constant number of parameters. We optimize all hyper-parameters (HPs) using a 80:20 validation split and Bayesian Optimization with Hyperband (BOHB, Falkner et al. 2018). For all II-WS-layers, we use \(k=3\) and a constant number of in- and output channels. For II with monomials, we use an iterative pruning-based selection with \(n_{m}=\{25,12,5\}\) monomial pairs after 0, 5 and 10 epochs following Rath and Condurache (2022). The exact HPs, optimization settings and network architectures can be found in Appendix C. If not mentioned otherwise, we report the mean and standard deviation over three runs.

### Evaluating Scale-Invariant Integration

We evaluate our scale-II layer on the Scaled-MNIST dataset, which consists of hand-written digits artificially scaled with factor \(s\in[0.3,1]\). We use the architecture from Sosnovik et al. (2020) (SES-CNN) and Sos

Figure 2: Log. Test Error (TE) on Scaled-MNIST subsets.
novik et al. (2021) (DISCO) built of three convolutional and two dense layers using \(n_{S}=4\) scales and replace the final global pooling layer with the scale-II layer. Following Sosnovik et al. (2020, 2021), we compare the invariance error of the scale-II layers to the methods they use to obtain scale-invariant representations: a mixed pooling approach including average and max pooling for Scaled-MNIST and average pooling for STL-10. The invariance error is a simplified version of the equivariance error given as \(\Delta=\frac{1}{S}\sum_{s}\frac{|\psi(x)-\psi(L_{s}x)|^{2}_{2}}{|\psi(x)|^{2}_ {2}}\). We compute \(\Delta\) when directly processing the input, i.e., \(\psi\) is the II- or pooling-layer and when \(\psi\) is the randomly initialized CNN including the respective layer using 100 samples from Scaled-MNIST scaled with \(s\in[0.5,0.55,...,1.0]\). The results in Table 1 show that scale-II guarantees invariance as opposed to the pooling approaches. While Scale-II with WS achieves a better error directly on the input, the monomial variant is slightly better when applied within the DNN.

We then evaluate the performance of our scale-II layer on Scaled-MNIST for classification. Here, scale-invariance is paramount to obtain correct results because the test set contains more variability than the training set, thus benefiting scale-invariant algorithms. Table 2 shows the results on the full dataset, Figure 2 with limited training data. We report mean and standard deviation over the six pre-defined data splits. II outperforms the pooling approach on full data and in the limited sample regime highlighting the improved sample complexity of our approach. In summary, the mixed pooling approach used by Sosnovik et al. (2021) does not guarantee scale-invariance which leads to a decreased performance. II allows for invariance guarantees and the WS-variant is easier to optimize than the monomials-variant. Hence, II with a WS outperforms the latter in the limited data domain and is used for all further experiments.

### Multi-Stream Digit Classification

The Street View House Number (SVHN) dataset contains single digits taken from house numbers. SVHN includes digits with different colors, font types, orientations and backgrounds and is thus harder to solve than MNIST. For all experiments on SVHN, we use the core training data, a WRN16-4 architecture, \(n_{r}=8\) rotations for all E(2)-G-Convs and \(n_{S}=3\) scales for the scale stream. The II step is calculated using the same number of rotations and flips. The results on the full SVHN dataset are shown in Table 3 while Figure 3 depicts the results using limited training data.

Using II instead of pooling improves the accuracy for all invariant architectures showing that II better preserves the information when transferring to invariance. Both invariant single-stream networks outperform the std. CNN in the limited and full data domain. This indicates that invariance to rotations and flips as well as scales is valuable information the classifier needs to learn during training - and the training data does not cover enough variability w.r.t rotations and scales for the baseline to learn this information. The rotation- and flip-invariant DNN achieves a better sample complexity than the scale-invariant one, indicating that rotation-invariance is more valuable for this dataset.

The multi-stream networks significantly outperform all single-stream variants including the baseline in all data regimes. The multi-stream architecture learns meaningful combinations of the generated invariants and is able to automatically choose the invariance best-fit for the training data at hand. This allows for best performances on all dataset sizes. In addition, for the rather simple task of classifying numbers, combining only a rotation- and a scale-invariant stream outperforms a standard CNN and leads to almost the same performance as with additional std. convolutions. In this problem setup, rotation- and scale-invariances seem sufficient for optimal performance and are able to recover all necessary object invariances. We conjecture that the learned features in the network's layers within the invariant streams focus on global invariances like illumination and noise while the dedicated II layer handles specific object invariances, e.g. to scales, similar to the scattering transformation (Oyallon et al., 2019).

### Multi-Stream Object Classification

Finally, we evaluate our proposed architecture for the more complex task of object classification on CIFAR-10 and STL-10. Both datasets contain RGB images of ten different object classes. STL-10 is a subset of ImageNet containing 5k training images that is commonly used as a bench

\begin{table}
\begin{tabular}{c c c} \hline \hline Layer & Input & CNN \\ \hline Average Pooling & 0.222 & 0.090 \\ Mixed Pooling & 0.017 & 0.039 \\
**Scale-II** Monomials & 1.24e-4 & **2.64e-7** \\
**Scale-II** WS & **2.97e-9** & 5.94e-3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Invariance Error \(\Delta\) when obtaining a scale-invariant representation using the respective layer directly on the input and a randomly initialized scale-equivariant network.

\begin{table}
\begin{tabular}{c c c} \hline \hline Method & Scale-II & TE [\%] \\ \hline CNN & - & 1.60 \(\pm\) 0.09 \\ SES-CNN & - & 1.42 \(\pm\) 0.07 \\ DISCO & - & 1.35 \(\pm\) 0.05 \\
**II-DISCO** & Monomials & **1.30 \(\pm\)** 0.06 \\
**II-DISCO** & WS & **1.30 \(\pm\)** 0.02 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Test Error (TE) on Scaled-MNIST using a CNN with 5 layers, data augmentation with random scales and an upsampling layer to double the input size.
mark for the limited data performance of object classification networks. STL-10 is more challenging than CIFAR-10 since it contains bigger and more diverse images. For CIFAR-10, we use WRN28-10 and the E(2)-stream with 8, 8 and 4 rotations per residual block. For STL-10, we use WRN16-8 and the E(2)-architecture with 8, 4 and 1 rotation per residual block (Weiler and Cesa, 2019). The E(2)-II-layer is used for 4-rotations and flips or flips-only, respectively. For both datasets, the scale-stream uses 3 scales (Sosnovik et al., 2021). The STL-10 and CIFAR-10 results on full data are shown in Table 3, CIFAR-10 results on limited data in Figure 4. For a fair comparison on STL-10, we adapted the official implementation of Sosnovik et al. (2021) which uses a stride of 1 in the initial convolutional layer by using stride 2 as in Devries and Taylor (2017); Weiler and Cesa (2019).

On CIFAR-10, we again demonstrate an increased sample efficiency of the invariant streams leading to superior performance for small dataset sizes (Figure 4). While the E(2)-invariant network is able to outperform the std. baseline in the full data regime, the scale-invariant network achieves subpar performance. We interpret the latter as a sign of less variance along the scale mode in this problem setup due to the rather small images contained in CIFAR-10. Thus, other invariances are more important to decide for the correct class. The scale-invariant stream seems to be too restrictive in the sense that it is unable to learn the full set of object invariants that the baseline architecture leverages to classify the objects. On the full data, II improves the performance for the scale-invariant architecture. However, the performance on the E(2)-invariant architecture is only on par. Nevertheless, Rath and Condurache (2022) show that a rotation-invariant architecture using II outperforms the one with pooling in limited data regimes even when achieving slightly worse results on full data. We further investigate the advantages of II in Section 5.4. Our combined network is able to achieve the best results for all data regimes by learning to combine the best information at each dataset size. The triple stream outperforms the dual variant which indicates that the std. stream is able to capture important additional object invariances that are neglected by the restricted, invariant streams.

\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{2}{c}{II} & \\ \hline E(2) & Scale & TE [\%] \\ \hline x & x & 7.51 \(\pm\) 0.12 \\ x & ✓ & 7.35 \(\pm\) 0.09 \\ ✓ & x & 6.34 \(\pm\) 0.07 \\ ✓ & ✓ & **5.90**\(\pm\) 0.05 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Role of the II layer for our triple-stream network on STL-10. An ’x’ marks an invariant stream without II.

Figure 4: TE on subsets of CIFAR-10 with full streams.

Figure 3: Log. TE on subsets of SVHN with full streams.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline II & Streams & Invariance & SVHN [\%] & C-10 [\%] & STL-10 [\%] \\ \hline  & PTN & Rot. \& Scale & 2.97 & 6.72 & - \\ x & Single & Std. & 2.93 \(\pm\) 0.02 & 3.89 \(\pm\) 0.08 & 12.02 \(\pm\) 0.05 \\ x & Single & E(2) & 2.64 \(\pm\) 0.05 & 2.91 \(\pm\) 0.13 & 9.80 \(\pm\) 0.40 \\ x & Single & Scale & 2.71 \(\pm\) 0.01 & 4.04 \(\pm\) 0.03 & 8.07 \(\pm\) 0.08 \\ \hline ✓ & Single & E(2) & 2.36 \(\pm\) 0.05 & 2.95 \(\pm\) 0.04 & 7.67 \(\pm\) 0.07 \\ ✓ & Single & Scale & 2.54 \(\pm\) 0.04 & 3.91 \(\pm\) 0.12 & 7.92 \(\pm\) 0.09 \\ ✓ & Dual\({}^{\star}\) & Scale & E(2) & 2.20 \(\pm\) 0.05 & 2.96 \(\pm\) 0.10 & 6.38 \(\pm\) 0.15 \\ ✓ & Dual & Scale & E(2) & 2.12 \(\pm\) 0.11 & 2.75 \(\pm\) 0.04 & 5.95 \(\pm\) 0.11 \\ ✓ & Triple\({}^{\star}\) & Scale, E(2) \& Std. & 2.29 \(\pm\) 0.03 & 2.74 \(\pm\) 0.08 & 6.46 \(\pm\) 0.08 \\ ✓ & Triple & Scale, E(2) \& Std. & **2.10**\(\pm\) 0.07 & **2.68**\(\pm\) 0.03 & **5.90**\(\pm\) 0.05 \\ \hline \hline \end{tabular}
\end{table}
Table 3: TE on SVHN, CIFAR-10 and STL-10. WRN16-4, WRN28-10 and WRN16-8 are used as baseline architecture. \({}^{\star}\) indicates constant number of parameters.
On STL-10, the E2-II network outperforms its counterpart without II significantly. Scale-II also slightly improves upon the baseline and II is clearly beneficial when combining multiple streams (see Table 4). Our multi-stream network achieves a new state-of-the-art result (Table 3), even with constant number of parameters. This shows that incorporating prior knowledge about multiple transformations improves the performance of classification DNNs in the limited data domain, even for complex real-world datasets. Additionally, the results show that the features learned by each stream are complementary, preserved by our II layers and are effectively combined by our proposed multi-stream head. The multi-stream architecture successfully improves the sample complexity while raising the number of parameters - hence increasing generalization.

### Ablation Studies

Multi-Stream: Number of Parameters.In addition to the full multi-stream architecture, we report the performance when keeping the number of parameters constant. Therefore, we shrink each stream by factor 2 or 3 for the dual or triple-stream network, respectively. The results are shown in Table 3 marked with \({}^{\star}\). Limited data results for SVHN and CIFAR-10 are shown in Appendix A.

Our approach still achieves state-of-the-art performance on all datasets and in all data domains. Combining multiple streams is beneficial, even with constant number of parameters. The dual-stream performs slightly better than the triple-stream. We believe this occurs since the individual streams of the triple-stream architecture are too thin, particularly in early layers.

Furthermore, we train a triple-stream architecture consisting of three standard streams on STL-10. We achieve a test error (TE) of 10.29% which is worse than the single-stream networks with invariance. This shows that the enforced invariance plays a key role for our multi-stream networks.

Multi-Stream: Importance of Invariant Integration.To quantify and demonstrate the importance of the II layer, we compare our multi-stream architecture including II to variants without II on STL-10. This includes a multi-stream architecture, where only pooling is used. The results in Table 4 demonstrate that on a more complex classification task, in low-data regime (i.e. when the training data does not properly cover all variability present in the test data) the multi-stream approach works best when both streams use II rather than pooling.

Training a standard WRN augmented with random 90\({}^{\circ}\) rotations and scales \(s\in[0.25,1]\) on STL-10 achieves a TE of 21.80%, which is clearly detrimental compared to the performance without those augmentations (12.08%). Hence, layer-wise, guaranteed in- and equivariance play a key role in the improved sample complexity of our approach.

## 6 Conclusion

In this contribution, we expanded II to scale transformations and showed its effectiveness on Scaled-MNIST. Since Scale-II using a WS is easier to optimize than the monomial variant, we applied it in a multi-stream DNN which besides scales includes a standard convolutional and a rotation-and-flip-invariant stream. This multi-stream DNN covers a variety of practically interesting use cases as shown by an improved sample complexity on SVHN and CIFAR-10 and new state-of-the-art results on STL-10 using only labeled data. We impose invariance to scales and rotation-and-flips in dedicated streams that also learn other global invariances and cover the remaining object invariances with a standard convolutional stream. This guarantees multiple invariances without suffering from the multiplicative increase when directly combining the groups.

Our framework is thought to leverage and honor prior knowledge. Therefore, it is focused on invariance guarantees, which may be rather restrictive in some cases. Specifically, invariance guarantees improve the sample complexity of DNNs leading to a performance boost, when training data is limited. In the large data domain, Vision Transformers (Dosovitskiy et al., 2021) with less geometrical constraints outperform conventional CNNs. Hence, our experiments focus on small-scale datasets. II is a general method that can be expanded beyond rotations and scales, but is restricted to transformations that can be modeled as (semi-) groups. Furthermore, we require an equivariant backbone before transferring to invariance. Nevertheless, the multi-stream network can in theory be enhanced with streams that achieve invariance without using II or G-Convs.

In the future, it is interesting to apply II to tasks where equivariance is helpful, e.g. to infer the pose in object detection. II could still be used for the parts of the network that benefit from invariance. On CIFAR-10 and SVHN, more sophisticated architectures and training methods than WRNs achieve a better performance (Foret et al., 2021; Lim et al., 2019). For a fair comparison, we stuck to WRNs in our experiments. Nevertheless, guaranteed invariances can generally be applied to many architectures in order to improve their sample complexity.

## Acknowledgments

We would like to thank the reviewers and our colleagues Lukas Enderich, Julia Lust and Paul Wimmer for their valuable remarks and contributions.
