Encoding the input scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale input signals. We study, in this paper, a scale-equivariant CNN architecture with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve scale-equivariant representations. To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low-frequency components. A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation. Numerical experiments demonstrate that the proposed scale-equivariant neural network with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at a reduced model size. Convolutional neural networks (CNNs) have achieved great success in machine learning problems such as image classification (Krizhevsky et al., 2012) , object detection (Ren et al., 2015) , and semantic segmentation (Long et al., 2015; Ronneberger et al., 2015) . Compared to fully-connected networks, CNNs through spatial weight sharing have the benefit of being translation-equivariant, i.e., translating the input leads to a translated version of the output. This property is crucial for many vision tasks such as image recognition and segmentation. However, regular CNNs are not equivariant to other important group transformations such as rescaling and rotation, and it is beneficial in some applications to also encode such group information explicitly into the network representation. Several network architectures have been designed to achieve (2D) roto-translation-equivariance (SE(2)-equivariance) (Cheng et al., 2019; Marcos et al., 2017; Weiler et al., 2018b; Worrall et al., 2017; Zhou et al., 2017) , i.e., roughly speaking, if the input is spatially rotated and translated, the output is transformed accordingly. The feature maps of such networks typically include an extra index for the rotation group SO(2). Building on the idea of group convolutions proposed by Cohen & Welling (2016) for discrete symmetry groups, Cheng et al. (2019) and Weiler et al. (2018b) constructed SE(2)-equivariant CNNs by conducting group convolutions jointly across the space and SO(2) using steerable filters (Freeman & Adelson, 1991) . Scaling-translation-equivariant (ST -equivariant) CNNs, on the other hand, have typically been studied in a less general setting in the existing literature (Kanazawa et al., 2014; Marcos et al., 2018; Xu et al., 2014; Ghosh & Gupta, 2019) . In particular, to the best of our knowledge, a joint convolution across the space and the scaling group S has yet been proposed to achieve equivariance in the most general form. This is possibly because of two difficulties one encounters when dealing with the scaling group: First, unlike SO(2), it is an acyclic and unbounded group; second, an extra index in S incurs a significant increase in model parameters and computational burden. Moreover, since the scaling transformation is rarely perfect in practice (due to changing view angle or numerical discretization), one needs to quantify and promote the deformation robustness of the equivariant representation (i.e., is the model still "approximately" equivariant if the scaling transformation is "contaminated" by a nuisance input deformation), which, to the best of our knowledge, has yet been studied in prior works. The purpose of this paper is to address the aforementioned theoretical and practical issues in the construction of ST -equivariant CNN models. Specifically, our contribution is three-fold: 1. We propose a general ST -equivariant CNN architecture with a joint convolution over R 2 and S, which is proved in Section 4 to be both sufficient and necessary to achieve ST -equivariance. 2. A truncated decomposition of the convolutional filters under a pre-fixed separable basis on the two geometric domains (R 2 and S) is used to reduce the model size and computational cost. 3. We prove the representation stability of the proposed architecture up to equivariant scaling action of the input signal. Our contribution to the family of group-equivariant CNNs is non-trivial; in particular, the scaling group unlike SO(2) is acyclic and non-compact. This poses challenges both in theory and in practice, so that many previous works on group-equivariant CNNs cannot be directly extended. We introduce new algorithm design and mathematical techniques to obtain the first general ST -equivariant CNN in literature with both computational efficiency and proved representation stability. We propose, in this paper, a ST -equivaraint CNN with joint convolutions across the space R 2 and the scaling group S, which we show to be both sufficient and necessary to impose ST -equivariant network representation. To reduce the computational cost and model complexity incurred by the joint convolutions, the convolutional filters supported on R 2 × S are decomposed under a separable basis across the two domains and truncated to only low-frequency components. Moreover, the truncated filter expansion leads also to improved deformation robustness of the equivaraint representation, i.e., the model is still approximately equivariant even if the scaling transformation is imperfect. Experimental results suggest that ScDCFNet achieves improved performance in multiscale image classification with greater interpretability and reduced model size compared to regular CNN models. For future work, we will study the application of ScDCFNet in other more complicated vision tasks, such as object detection/localization and pose estimation, where it is beneficial to directly encode the input scale information into the deep representation. Moreover, the memory usage of our current implementation of ScDCFNet scales linearly to the number of the truncated basis functions in order to realize the reduced computational burden explained in Theorem 2. We will explore other efficient implementation of the model, e.g., using filter-bank type of techniques to compute convolutions with multiscale spatial filters, to significantly reduce both the computational cost and memory usage. Proof of Theorem 1. We note first that (4) holds true if and only if the following being valid for all l ≥ 1,
We construct scale-equivariant convolutional neural networks in the most general form with both computational efficiency and proved deformation robustness. The authors propose a CNN architecture that is theoretically equivariant to isotropic scalings and translations by adding an extra scale-dimension to activation tensors.