A Novel Convolutional Neural Network Architecture with a Continuous
  Symmetry
This paper introduces a new Convolutional Neural Network (ConvNet)
architecture inspired by a class of partial differential equations (PDEs)
called quasi-linear hyperbolic systems. With comparable performance on the
image classification task, it allows for the modification of the weights via a
continuous group of symmetry. This is a significant shift from traditional
models where the architecture and weights are essentially fixed. We wish to
promote the (internal) symmetry as a new desirable property for a neural
network, and to draw attention to the PDE perspective in analyzing and
interpreting ConvNets in the broader Deep Learning community.
Yao Liu, Hang Shao, Bing Bai
# A Novel Convolutional Neural Network Architecture with a Continuous Symmetry

###### Abstract

This paper introduces a new Convolutional Neural Network (ConvNet) architecture inspired by a class of partial differential equations (PDEs) called quasi-linear hyperbolic systems. With comparable performance on the image classification task, it allows for the modification of the weights via a continuous group of symmetry. This is a significant shift from traditional models where the architecture and weights are essentially fixed. We wish to promote the (internal) symmetry as a new desirable property for a neural network, and to draw attention to the PDE perspective in analyzing and interpreting ConvNets in the broader Deep Learning community.

Keywords:Convolutional Neural Networks Partial Differential Equations Continuous Symmetry

## 1 Introduction

With the tremendous success of Deep Learning in diverse fields from computer vision [11] to natural language processing [23], the model invariably acts as a black box of numerical computation [2], with the architecture and the weights largely fixed, i.e., they can not be modified without changing the output (for a fixed input), except by permuting neurons or units from the same layer. One would say that the **symmetry** of the neural network, the set of transformations that do not affect the model's prediction on any input, is

\[\text{Sym}(\text{model})=\prod_{i}S_{n_{i}}\,,\]

the product of symmetric groups on \(n_{i}\) "letters," where \(n_{i}\) is the number of _interchangeable_ neurons in the \(i\)-th layer.

Although quite large as a group, it does not permit us to modify the model in any meaningful way. In the case of Convolutional Neural Networks (ConvNets), the channels are essentially fixed and frozen in place, due to the presence of coordinate-wise activation functions [26] (such as ReLU), which arguably build certain semantic contents in the channels [17], thus mixing them would destroy the model. The _nonlinear_ activation unit is generally thought to be an essential component for the neural network to fit arbitrary _nonlinear_ functions.
With inspiration and guidance from partial differential equations (PDEs), specifically _first-order quasi-linear hyperbolic systems_[1], we introduce a new architecture of ConvNet with a different type of nonlinearity, which allows us to remove most of the activation functions without degrading performance. As a result, the new architecture admits a _continuous_ group of symmetry (i.e., a Lie group, as opposed to a discrete group) that allows mixing of the channels; in one version, it's the full _general linear_ (GL) _group_, the set of all invertible \(n_{i}\times n_{i}\) matrices:

\[\mathrm{Sym}(\mathrm{model})=\prod_{i}GL(n_{i},\mathbb{R})\,.\]

With a judicious choice of such transformations, we may alter the weights so that the connections become more sparse, resulting in a smaller model (a kind of lossless pruning). Since the group is continuous, one might use the method of gradient descent to search for it. In addition, it may also lead to a better understanding of the inner workings of the neural network, much like how matrix diagonalization leads to the decoupling of a system of (linear) differential equations into different "modes", which are easier to interpret.

We primarily present the simplest version of our model based on ResNet50, illustrated in Fig. 1 alongside the corresponding PDE. The nonlinearity is at the element-wise multiplication of the two branches (3x3 and 1x1 convs), and we apply activation functions _only_ at the end of 4 of the 16 blocks. See SS5.1 for details.

Figure 1: Schematic of a single block of our ConvNet architecture based on Eq. (3), to replace the bottleneck block of ResNet50. The trapezoids represent the increase/decrease in the number of channels. The corresponding components of the PDE are illustrated. (Best viewed in color.)
This is a preliminary report on our new architecture, and the relevant parts of partial differential equations behind it. It is our hope that the research community builds upon and analyzes the properties of this new architecture, and to take the PDE perspective seriously in designing, analyzing, or simply describing _all_ aspects of ConvNets. Moreover, given that the Transformer architecture [23] also involves a similar kind of nonlinearity apart from softmax and activation functions, it could potentially be made to admit a continuous symmetry as well.

## 2 Related Work

The link with **differential equations** has been recognized, and well exploited [8, 16, 20, 4, 10], soon after the introduction of ResNet [11] by He _et al._ in 2015, if not known implicitly before; see also [21] for a more recent analysis. With a few exceptions [16, 20], most discussions do not make an emphasis on _partial_ differential equations, and to the best of our knowledge, little activity has been devoted to _designing_ new architecture from this perspective. Even though a PDE can be regarded as an ODE in which the state space is "infinite-dimensional", or upon discretization, a finite but large system with interactions only between neighboring "pixels," we find the PDE perspective, specifically of hyperbolic systems, more illuminating and _fruitful_ (albeit limited to ConvNets), and deserves more attention and further study in the broader Deep Learning community.

A related but distinct field of research is using Deep Learning methods to solve various PDEs of interests to physicists, applied mathematicians, and engineers. We shall only mention two pioneering works that have attracted the most attention: Fourier Neural Operators (FNO) [15] and Physics-informed Neural Networks (PINN) [19].

**Symmetry** and **equivariance** often appear in the theoretical discussions of ConvNets and Graph Neural Networks (GNN) [6, 3], though it's worth pointing out the distinction from our usage: More often, we say a neural network has a (translational or permutation) _symmetry_, or is _equivariant_ or _invariant_ (under translations or permutations), if when we transform the input in a certain way, the output is also transformed accordingly, or does not change at all; the model itself remains fixed. In our scenario, we are directly transforming the weights of the model, which incidentally does not need to be trained. Nevertheless, much of our work involves finding a good enough model that achieves comparable performance on standard training sets. (It shall be apparent that, as with conventional ConvNets, our model is also equivariant under translations.)

One type of operation, known under the term **structural reparametrization**[7], can claim to modify the model after training. However, it can only merge consecutive layers or operations that are both linear; the basic example is conv followed by batchnorm. As such, it is better regarded as a trick in training: for whatever reason, it is better to train with a deeper and more complicated architecture than is necessary for the model, and is fundamentally different from the kind of symmetry that our model has.
## 3 Designing ConvNets from the PDE perspective

Given that ResNet is numerically solving a particular system of PDEs,

\[\frac{\partial u_{i}}{\partial t} = \sigma\biggl{(}\sum_{j}L_{ij}u_{j}\biggr{)}\,,\quad\mbox{for}\quad i =1,\ldots,n\] \[L_{ij} := \alpha_{ij}\frac{\partial}{\partial x}+\beta_{ij}\frac{\partial} {\partial y}+\gamma_{ij}\frac{\partial^{2}}{\partial x\partial y}+\cdots\]

of \(n\) unknowns \(u_{i}\equiv u_{i}(x,y,t)\), with initial condition at \(t=0\), wherein the coefficients are learned (for background, see Appendix), it is natural to take inspiration from other PDEs as found in mathematics and physics, and see what new ConvNet architecture would come out. Here are some natural changes that one could make:

* Make the constant coefficients _variables_ (of \(x\) and \(y\)), e.g., simply as linear or polynomial functions. The equation would still be linear, but now the space of PDEs would include this special equation: (\(n=1\)) \[\frac{\partial u}{\partial t}=-y\,\frac{\partial u}{\partial x}+x\,\frac{ \partial u}{\partial y}\,,\] which is solved by simply _rotating_ the initial data \(f(x,y)\) by angle \(t\) (around the origin): \[u(x,y,t)=f(x\cos t-y\sin t,x\sin t+y\cos t)\,,\] as can be readily verified. It is reasonable to expect that such a variation on ResNet would allow the model to make rotations and dilations -- in addition to translations -- on the input image.
* One might think that the standard "zero padding" of conv layer corresponds to the Dirichlet boundary condition: the value of \(u\) on the boundary being fixed at a prescribed value for \(t>0\). On closer inspection, it is slightly different. Furthermore, one could also experiment with other boundary conditions; the other natural one in physics is the Neumann condition, that the _normal derivative_ of \(u\) on the boundary is prescribed. The different conditions have the effect that the signals would "bounce back" off the boundary differently.
* In a typical PDE, the matrix of coefficients is constant or slowly varying with time \(t\), while in neural networks the weights from different layers are initialized independently, drawn from a (normal) distribution. One could try to force the weights from neighboring layers to correlate, either by weight-sharing or by introducing a term in the loss function that penalizes large variations between layers.

Having experimented with some of these ideas on small datasets, we did not find a specific variation that yields convincing results on the full ImageNet. We then looked into ways that the coefficients may depend on \(u\) itself, which makes
the equation nonlinear (apart from the activation functions). It may be viewed as a kind of "dynamic kernel," but we draw inspiration from a class of PDEs called _quasi-linear hyperbolic systems_, which may be the simplest, well-studied nonlinear systems for which the number of equations can be arbitrary.

In two spatial and one time dimensions, a first-order **quasi-linear system** is typically of the form

\[\frac{\partial u_{i}}{\partial t}=\sum_{j}\mathcal{A}_{ij}(u)\frac{\partial u_ {j}}{\partial x}+\sum_{j}\mathcal{B}_{ij}(u)\frac{\partial u_{j}}{\partial y}\,, \tag{1}\]

where the coefficient matrices may depend on \(u\) (but not derivatives of \(u\)), and it is **hyperbolic3** if any linear combination of \(\mathcal{A}\) and \(\mathcal{B}\) is diagonalizable with only _real_ eigenvalues (e.g., \(\mathcal{A}\) and \(\mathcal{B}\) are symmetric) for _any_ value of \(u\). Leaving aside the latter condition, the simplest example is to make each entry a linear function of \(u\):

Footnote 3: The designation may sound cryptic. It originates from the classic wave equation, which has some semblance in form with the equation of a hyperbola or hyperboloid; see Appendix §A.3. To avoid any confusion, it is _not_ related to hyperbolic plane/space/geometry that has made into machine learning

\[\mathcal{A}_{ij}(u)=\sum_{k}\mathcal{A}_{ijk}u_{k}\,, \tag{2}\]

and similarly for \(\mathcal{B}\). By dimension count, such a tensor would be very large (for large \(n\)), and it would deviate too much from typical ConvNets. Instead, we shall restrict to

\[\frac{\partial u_{i}}{\partial t}=\sum_{j}A_{ij}\sum_{k}C_{jk}u_{k}\frac{ \partial u_{j}}{\partial x}+\sum_{j}B_{ij}\sum_{k}D_{jk}u_{k}\frac{\partial u_ {j}}{\partial y}\,, \tag{3}\]

which is straightforward to turn into a ConvNet (see SS5.1 and Fig. 1 for details), and the number of parameters is kept at a reasonable level. Since nonlinearity is already built-in, we thought it would not be necessary to add activation functions, at least not at every turn; and much to our surprise, the model trains just as well, if not better. With this simple change, the model now has a _continuous_ symmetry, from mixing of the channels, that is not present in conventional ConvNets with coordinate-wise activation functions after every conv layer, and we believe it is a more significant contribution than matching or breaking the state of the art (SOTA). It is likely that, with enough compute, ingenuity, and perhaps techniques from Neural Architecture Search, variations of this architecture could compete with the best image models of comparable size. (We have not tried to incorporate the modifications listed earlier, as they are all linear, and we wish to explore the new nonlinearity on its own.)

It is observed that, once we remove all the activation functions, or use the standard ReLU, the training is prone to breaking down: it would fail at a particular epoch, with one or more samples (either from the training or validation sets) causing the network to output NaN, and the model could not recover from it. To mitigate this, we add activation functions such as hardtanh that clip off large
values, only once every few blocks. It is also observed that resuming training with a smaller learning rate may get around the "bad regions" of the parameter space. More analyses are needed to determine the precise nature and cause of this phenomenon (it might be related to the formation of "shock waves" in nonlinear hyperbolic equations [1]), and perhaps other ways to avoid it.

We provide here the details of the activation functions. In standard PyTorch [18], nn.Hardtanh is implemented as

\[\text{hardtanh}(x):=\begin{cases}\text{max\_val}&x>\text{max\_val}\\ \text{min\_val}&x<\text{min\_val}\\ x&\text{otherwise}\,.\end{cases}\]

We typically use \(\pm 1\) as the clip-off values. We also introduce two multi-dimensional variants that we call "hardball" and "softball," which appear to give better performance. Hardball is so defined that it takes a vector \(\boldsymbol{x}\in\mathbb{R}^{n}\) and maps it into the ball of radius \(R\),

\[\text{hardball}(\boldsymbol{x}):=\begin{cases}\boldsymbol{x}&|\boldsymbol{x}| <R\\ R\boldsymbol{x}/|\boldsymbol{x}|&|\boldsymbol{x}|\geq R\,,\end{cases}\]

where \(|\boldsymbol{x}|\) is the Euclidean norm. We set \(R\) to be the square root of \(n\) (the number of channels), though other choices may be better. Softball is a soft version,

\[\text{softball}(\boldsymbol{x}):=\frac{\boldsymbol{x}}{\sqrt{1+|\boldsymbol{x }|^{2}/R^{2}}}\,,\]

and they are both spherically symmetric. (One may perhaps regard them as normalization layers rather than activation functions.)

## 4 Symmetry of the model

The symmetry of our model would depend on the specific implementation, and may be more complicated than one would naively expect. We shall first consider it on the level of the PDE.

With a change of coordinates \(\tilde{u}_{i}=\sum_{j}T_{ij}u_{j}\) for an invertible matrix \(T\), the general equation (1) with (2)

\[\frac{\partial u_{i}}{\partial t}=\sum_{j,k}\mathcal{A}_{ijk}u_{k}\frac{ \partial u_{j}}{\partial x}+\sum_{j,k}\mathcal{B}_{ijk}u_{k}\frac{\partial u_{ j}}{\partial y}\]

would transform _only_ in the coefficient tensors \(\mathcal{A}_{ijk}\) and \(\mathcal{B}_{ijk}\). Indeed,
(For clarity, we omit the second half involving \(\mathcal{B}\) and \(\frac{\partial}{\partial y}\).)

\[\frac{\partial\tilde{u}_{i}}{\partial t} =\sum_{j}T_{ij}\frac{\partial u_{j}}{\partial t}\] \[=\sum_{j}T_{ij}\sum_{k,l}\mathcal{A}_{jkl}u_{l}\frac{\partial u_{ k}}{\partial x}\] \[=\sum_{j}T_{ij}\sum_{k,l}\mathcal{A}_{jkl}\sum_{m}T_{lm}^{-1} \tilde{u}_{m}\sum_{r}T_{kr}^{-1}\frac{\partial\tilde{u}_{r}}{\partial x}\] \[=\sum_{m,r}\Biggl{(}\underbrace{\sum_{j,k,l}T_{ij}\mathcal{A}_{jkl }T_{lm}^{-1}T_{kr}^{-1}}_{\tilde{\mathcal{A}}_{irm}}\Biggr{)}\tilde{u}_{m} \frac{\partial\tilde{u}_{r}}{\partial x}\,.\]

We note in passing that similar calculations are commonplace in _classical_ differential geometry when making a change of coordinates _on the base space_. Here, we are making a change of coordinates on the "dependent" variables. From a more abstract point of view, this is the induced representation of \(GL(V)\) on the tensor product \(V\otimes V^{*}\otimes V^{*}\cong\mathrm{Hom}(V\otimes V,\,V)\) for a vector space \(V\cong\mathbb{R}^{n}\).

On the level of the neural network, we only need to make sure that the \(T^{-1}\) comes from the previous layer, i.e., it is the inverse of the \(T\) that appears in transforming the previous block (if different). With such a transformation at each block, we find that the overall symmetry of the model is

\[\mathrm{Sym}(\mathrm{model})=\prod_{i}G_{i}\,,\]

with

\[G_{i}=\begin{cases}S_{n}&\sigma=\text{relu, or any element-wise activation function}\\ O(n)&\sigma=\text{hardball, softball, etc.}\lx@note{footnote}{The group $O(n):=\{M\in GL(n,\mathbb{R})\mid MM^{T}=I_{n}\}$ is known as the _orthogonal group_, and it preserves the Euclidean norm on $\mathbb{R}^{n}$.}\end{cases}\]

As noted before, this "fully connected" block would be too costly to train, if we are to match ResNet in which the last stage uses as many as \(n=512\) channels. One simple way to reduce the number of parameters is to make the tensor "block diagonal", and the transformation would only mix channels from the same block. The bottleneck block of ResNet addresses this by shrinking the number of channels before applying the 3x3 conv, but a similar approach would introduce additional layers that shield the main operations that we would like to transform.

If we are to take \(\mathcal{A}_{ijk}\) to be of the special form as in Eq. (3), i.e., as the product of two matrices \(A_{ij}C_{jk}\) (no summation), then it is not guaranteed that
the transformed tensor would still factorize in the same way, for a generic \(T\). By simple dimension count, the set of tensors that are factorizable in the prescribed way is a subvariety (submanifold) of dimension at most \(2n^{2}\), and one wishes to find a \(T\in GL(n,\mathbb{R})\) that keeps the resulting \(\tilde{\mathcal{A}}_{ijk}\) on the subvariety. Given that the dimension of \(GL(n,\mathbb{R})\) is \(n^{2}\) and that \(2n^{2}+n^{2}\ll n^{3}\), it is not _a priori_ obvious that such transformations exist, apart from simple scalings and \(S_{n}\), that are universal for all \(\mathcal{A}_{ijk}\).

What one may hope for is that, for a specific \(\mathcal{A}_{ijk}\) that factors, we can find such a \(T\). For example, if for some pair of indices \(j,j^{\prime}\), we have \(A_{ij}C_{jk}=A_{ij^{\prime}}C_{j^{\prime}k}\) for all \(i,k\), then we can perform a rotation in the plane of the \(j\) and \(j^{\prime}\) directions:

\[\begin{cases}\tilde{u}_{j}=\alpha u_{j}+\beta u_{j^{\prime}}\\ \tilde{u}_{j^{\prime}}=\gamma u_{j}+\delta u_{j^{\prime}}\end{cases}\qquad \alpha\delta-\beta\gamma\neq 0\,.\]

Further investigation, either theoretical or numerical, may be needed to answer this question satisfactorily. It may be the case that there exists a symmetry that preserves the output _not_ for all inputs, but only those inputs that are "similar" to the dataset. It would be a weaker form of symmetry, but no less useful in practice.

Lastly, it should be remarked that there is a trivial "symmetry" in the tensor \(\mathcal{A}_{ijk}\) in the last two indices, i.e., \(\mathcal{A}_{ijk}\) and \(\mathcal{A}_{ikj}\) can be interchanged (so long as their sum is fixed). One may regard this as a redundancy in the parameter space, for we can force \(\mathcal{A}_{ijk}=\mathcal{A}_{ikj}\), or \(\mathcal{A}_{ijk}=0\) for \(j<k\) (and reduce the dimension roughly by half), and not due to mixing of the channels. We have not exploited this in the present work.

## 5 Experimental Results

### Details of the Architecture

How do we turn Eq. (3) into a ConvNet? We first make the differential operators \(\partial/\partial x\) and \(\partial/\partial y\) into 3x3 convolutional kernels, but we allow the weights to be trainable instead of fixed. Each incoming channel (each \(u_{j}\)) would split into 2 -- or better, 4 -- channels, and this is conveniently implemented in nn.Conv2d by setting groups to equal the number of input channels, as in "depthwise convolution" [5]. The matrices are simply 1x1 conv layers, with \(A\) and \(B\) stacked into one, and \(C\) and \(D\) stacked into one. Batchnorm is applied after the 3x3 and at the end. As with standard ResNet, the time derivative turns into the skip connection, and we arrive at the architecture of a single block as illustrated in Fig. 1. We do not enforce symmetry of these matrices to make the equation hyperbolic in the technical sense. As a general rule, we need not strictly follow the equation, but take the liberty in relaxing the weights whenever convenient.

One novelty is to make the weights of the 3x3 conv shared across the groups, which would make the transformations easier to implement. One may achieve this by making an nn.Conv2d with 1 input channel and 4 output channels, and at forward pass, we "repeat" the \(4\times 1\times 3\times 3\) weights in the
acting on the input tensor. Most of our experiments are with this "minimalist" 3x3 conv, except the ones marked "no ws" (no weight-sharing) in Table 1.

It may be possible to implement this kind of "variable-coefficient convolution" _natively_, instead of using the existing torch.nn layer which is tailored to conventional convolutions.

For the full design of the neural network, we simply take the classic ResNet50 with [3,4,6,3] as the numbers of blocks in the four stages. No activation function is applied except once only in each stage (e.g., at each downsampling), and we use nn.Hardtanh or our variants, hardball and softball (see SS3 for definitions), instead of ReLU, test the training would fail completely or the resulting model would not work as well.

### Experiments

As is standard in computer vision since the 2012 Deep Learning revolution, we have trained our model as an image classification task on the ImageNet dataset. The ImageNet-1k contains 1000 classes of labeled images, and for the sake of faster iterations, we primarily trained on a 100-class subset on a single GPU, while maintaining the standard image size of 224x224.

We use the timm library of PyTorch image models [24] for best practices in implementing ResNet and its training [25]. We took the official training script for ResNeXt-50 (SGD, cosine learning rate, with a warmup of 5 epochs, batch size of 192, etc.) except that the peak learning rate is set to 0.3 instead of 0.6, and the total number of epochs is set to 50. The results are in Table 1, where we mainly record two ways of modifying the model: changing only the activation function, and altering the placements of the conv layers within the block, corresponding to modifying Eq. (3) into Eqs. (4)-(7):

\[\frac{\partial u_{i}}{\partial t} =\sum_{k}C_{ik}u_{k}\sum_{j}A_{ij}\frac{\partial u_{j}}{\partial x }+\sum_{k}D_{ik}u_{k}\sum_{j}B_{ij}\frac{\partial u_{j}}{\partial y} \tag{4}\] \[\frac{\partial u_{i}}{\partial t} =\sum_{k}C_{ik}u_{k}\frac{\partial u_{i}}{\partial x}+\sum_{k}D_{ ik}u_{k}\frac{\partial u_{i}}{\partial y}\] (5) \[\frac{\partial u_{i}}{\partial t} =\sum_{j}A_{ij}\frac{\partial}{\partial x}\sum_{k}C_{jk}u_{j}u_{k} +\sum_{j}B_{ij}\frac{\partial}{\partial y}\sum_{k}C_{jk}u_{j}u_{k}\] (6) \[\frac{\partial u_{i}}{\partial t} =\sum_{j}A_{ij}\frac{\partial}{\partial x}\sum_{k}C_{jk}u_{j}u_{k} +\sum_{j}B_{ij}\frac{\partial}{\partial y}\sum_{k}D_{jk}u_{j}u_{k} \tag{7}\]

Note that Eq. (5) only has each \(u_{i}\) depending on its own derivatives, and the model is thus smaller and limited in expressivity or capacity. Eqs. (6) and (7) have the derivative acting on the product \(u_{j}u_{k}\), and it is often called a system of _conservation laws_. They can easily be rewritten in the form of Eq. (3), and the difference in performance may be attributable simply to the model size.
It is expected that, when going to the full ImageNet-1k, and allowing for longer training and hyperparameter tuning, the best-performing model may be different from the ones we found in Table 1.

We refrain from making assertions on _why_ -- or _if_ -- this class of PDEs is superior as a base model for ConvNets, or _how_ the theory of PDE can provide the ultimate answer to the _effectiveness_ of neural networks as universal function approximators by means of gradient descent. Whatever mechanisms that make ResNet work, also make our model work.

## 6 Conclusion

We present a new ConvNet architecture inspired by a class of PDEs called quasi-linear hyperbolic systems, and with preliminary experiments, we found a simple implementation that showed promising results. Even though it is known, within small circles, the close connection between PDE and ConvNet, we made the first architecture design directly based on a nonlinear PDE, and as a result, we are able to remove most of the activation functions which are generally regarded as indispensable. The new architecture admits a continuous symmetry that could be exploited, hopefully in future works. We expect that this work opens up a new direction in neural architectural design, demonstrates the power of the PDE perspective for ConvNets, and opens the door for other concepts and techniques in nonlinear PDE, both theoretical and numerical, for improved understanding of neural networks.

\begin{table}
\begin{tabular}{c c c c} model & \#parameters & top1-acc & activation \\ \hline ResNet50 & 23.7M & 84.52 & \\ MobileNet\_v3\_large & 4.33M & 82.91 & \\ \hline Eq. (3) & 8.61M & 82.06 & relu@all \\ Eq. (3) & 8.61M & 82.34 & hardtanh@ds \\ Eq. (3) & 8.61M & 83.50 & hardball@ds \\ Eq. (3) & 8.61M & 83.66 & softball@ds \\ Eq. (3) (no ws) & 8.73M & 84.24 & softball@ds \\ Eq. (3) (no ws, x6) & 13.0M & 84.58 & softball@ds \\ \hline Eq. (4) & 5.70M & 81.88 & \\ Eq. (5) & 4.26M & 78.64 & \\ Eq. (6) & 5.61M & 82.52 & softball@ds \\ Eq. (7) (no ws, x6) & 13.0M & **84.96** & \\ \end{tabular}
\end{table}
Table 1: Performance on a 100-class subset of ImageNet-1k, trained for 50 epochs with identical training strategy. For our model, activation is applied either at the end of each block (@all), or only at downsampling (@ds). Inside the block, the number of channels increases by a factor of 4, except when indicated with “x6”.
## Acknowledgements

This work is supported in part by Provincial Key R&D Program of Zhejiang under contract No. 2021C01016, in part by Young Elite Scientists Sponsorship Program by CAST under contract No. 2022QNRC001.

## Addendum

It has come to our attention that the recent work [27] and related ones are highly relevant and largely complementary to ours, in terms of exploiting the kind of continuous symmetry presented here. Specifically, our activation functions, hardball and softball, are examples of _radial_ activation functions, and the symmetries are more aptly described as symmetries _in the parameter space_. We apologize for our omission in the published version.
