WavPool: A New Block for Deep Neural Networks
Modern deep neural networks comprise many operational layers, such as dense
or convolutional layers, which are often collected into blocks. In this work,
we introduce a new, wavelet-transform-based network architecture that we call
the multi-resolution perceptron: by adding a pooling layer, we create a new
network block, the WavPool. The first step of the multi-resolution perceptron
is transforming the data into its multi-resolution decomposition form by
convolving the input data with filters of fixed coefficients but increasing
size. Following image processing techniques, we are able to make scale and
spatial information simultaneously accessible to the network without increasing
the size of the data vector. WavPool outperforms a similar multilayer
perceptron while using fewer parameters, and outperforms a comparable
convolutional neural network by ~ 10% on relative accuracy on CIFAR-10.
Samuel D. McDermott, M. Voetberg, Brian Nord
# WavPool: A New Block for Deep Neural Networks

###### Abstract

Modern deep neural networks comprise many operational layers, such as dense or convolutional layers, which are often collected into blocks. In this work, we introduce a new, wavelet-transform-based network architecture that we call the _multi-resolution perceptron_: by adding a pooling layer, we create a new network block, the _WavPool_. The first step of the multi-resolution perceptron is transforming the data into its multi-resolution decomposition form by convolving the input data with filters of fixed coefficients but increasing size. Following image processing techniques, we are able to make scale and spatial information simultaneously accessible to the network without increasing the size of the data vector. WavPool outperforms a similar multilayer perceptron while using fewer parameters, and outperforms a comparable convolutional neural network by \(\sim 10\%\) on relative accuracy on CIFAR-10. 2

Footnote 2: FERMILAB-CONF-23-278-CSAID

## 1 Introduction

Discrete signals have the potential to represent an amount of information that scales exponentially with the number of bits. Despite this fact, modern computational methods have demonstrated an ability to abstract patterns from data with far less than an exponential degree of complexity. Sometimes, the only independent variable underlying these patterns is their location - e.g., position in a sentence provides nearly all of the information necessary for natural language processing. In other signals, patterns are present that are not apparent as a function of the given variables but are presented in the variables of a conjugate space. Frequency and time are conjugate to one another, which is why Fourier analysis is useful in analyzing sounds. Spatial scale and position are conjugate to one another, which is why spherical harmonic analysis is useful in analyzing the cosmic microwave background (Peebles and Yu, 1970; Bond and Efstathiou, 1984). Most data, however, are encoded in signals that are not organized in any single variable, instead exhibiting regularities of both position and scale that are not simply accessible from spatial or scale data alone. This suggests a need for a type of signal processing that remains simultaneously sensitive to conjugate pairs of variables.

Many methods have been used to extract information and discover patterns in challengingly complex data sets. For example, matched filters have a long and successful history in the field of signal
processing Turin (1960). Recently, deep learning has upended long-held notions of what is possible in pattern recognition, with significant implications for science and society. For example, the multi-layer perceptron (MLP) Rosenblatt (1958) gave the first hint of how stacking many layers of matrix arithmetic between layers of nonlinearity could produce adaptable and flexible learning systems Cybenko (1989); Hornik et al. (1989). Architectures that further generalized these techniques, like convolutional neural networks (CNNs) LeCun et al. (1998) and the transformer Vaswani et al. (2017), have proven transformative in extending deep learning techniques to two-dimensional images and text, respectively. Nevertheless, traditional architectures require significant complexity (e.g., multiple stacked convolutional layers) to identify data features.

Wavelets are mathematical tools that can be used to provide simultaneous sensitivity to conjugate data - e.g., position and size or time and frequency. Therefore, they have the potential to provide unique insights on data that are not purely characterized by spatial or scale information. Wavelet families comprise functions of finite spatial extent that are related to one another through scaling and translational transformations. If a family provides a complete, orthonormal basis of functions, then convolving the family with a set of data will provide a decomposition of the data. Wavelets were first discovered in the early 20th century, but many of their more remarkable properties were not formalized, generalized, or fully explored until the 1980s and 1990s (Daubechies, 1992; Mallat, 1989; Meyer, 1993; Daubechies, 1996). Methods based on the wavelet transform have since played a foundational role in many fields. For example, they undergird modern computational methods like the JPEG image compression algorithm (Skodras et al., 2001).

The wavelet transform and related functions like the Laplacian pyramid technique have been used in combination with deep learning techniques to achieve state-of-the-art performance on various image-related tasks - image reconstruction with autoencoders (Chen et al., 2017) or more generally convolutional layers (Lai et al., 2017; Liu et al., 2018), image classification (Fujieda et al., 2018), representation learning (Yao et al., 2022), and learnable wavelets for denoising with reinforcement learning Michau et al. (2022).

In this work, we combine wavelets that have compact support (Mallat, 1989) with deep learning operations in a new method. We use Daubechies wavelets (Daubechies, 1992) to decompose data into components on different scales without increasing the size of the data vector. The wavelet decomposition facilitates training on features of different size and scale in parallel across all length scales spanned by the input. This is not possible for networks comprised of filters of fixed size, as in a CNN. By construction, this will contain the same information content as the original, non-decomposed data without increasing the size of the data vector. However, spatial and scale information about the data will be made simultaneously accessible to the neural network. We train a series of MLPs in parallel on the decomposed images. We refer to these multiple, parallel MLPs as a multi-resolution perceptron (MRP). We use a pooling layer to extract a classifier from the MRP. This trainable block is called "WavPool."

## 2 Theory of Multi-Resolution Decomposition

In this section, we provide a review of some basic results in wavelet methods. The reader who is familiar with this formalism can skip to Sec. 3.

Consider an input signal \(S\) defined on a D-dimensional grid. In this work, we will use a wavelet transform to partition this data into its multi-resolution decomposition (MRD). The MRD of \(S\) is a set of transformed signals,

\[\mathcal{M}_{S}=\{C(S),W^{a}_{L}(S),\ldots,W^{a}_{1}(S)\}, \tag{1}\]

where \(a\) is an index that depends on the dimensionality of \(S\); \(L\) is the number of levels of the decomposition, which depends on the wavelet that is used for the decomposition; \(W(S)\) are the _details_ of \(S\), which are the features of \(S\) at a particular spatial scale; and \(C(S)\) is the _smoothest view_ of \(S\), which is its global average for the Haar wavelet, but is a non-trivial matrix for other wavelets. The size of a signal \([S]\) is equal to the sum of the sizes of all of the constituent signals \(\zeta\) that comprise its MRD: \([S]=\sum_{\zeta\in\mathcal{M}_{S}}[\zeta]\). For a D-dimensional signal, the index \(a\) in Eq. (1) takes on \(2^{\mathrm{D}}-1\) different values. For example, for a 2-dimensional signal, \(a\) takes on three values, corresponding to the horizontal, vertical, and diagonal details at each level, as described in more detail presently. The number of levels \(L\) is set by the size of the signal and the _number of vanishing moments_\(n_{v}\) of the
wavelet used in the decomposition:

\[L=\lfloor\log_{2}[S]\rfloor-n_{v}+1. \tag{2}\]

The wavelets used in this work, first described by Daubechies [1], are indexed by the number of vanishing moments they have, meaning the degree of the polynomial function that they can set to zero: the Daubechies-\(n_{v}\) wavelet will set a polynomial of degree \(n_{v}-1\) to zero.

We now describe the components of the MRD. This requires two functions: a _smoothing wavelet_\(\phi\) and a _differencing wavelet_\(\psi\). The differencing wavelet is given by reversing the smoothing wavelet while simultaneously alternating the parity. If \(\phi\) has \(N\) entries indexed by \(i\), then the \(i^{\rm th}\) entry of \(\psi\) is \(\psi_{i}=(-1)^{i}\phi_{N-i}\). In Eq. (1), \(C(S)\) is the smoothest view of \(S\): this is also the mean of \(S\), and it is obtained by

\[C(S)=\phi\underbrace{\circ(\phi\circ(\cdots\atop L\ {\rm times}))}_{L\ {\rm times}}, \tag{3}\]

where \(\circ\) is the convolution operation and \(L\) is the maximum number of times \(\phi\), which is the same as the number of levels (see Eq. (2)). Each of the products in the convolution reduces the size of the input, so that each level of the MRD is a different size. The convolution procedure is applied iteratively to the smoothed images until a single number \(C(S)\) remains. One can generalize the smoothest view \(C(S)\) from Eq. (3) to signals that have been partially smoothed to level \(\ell\):

\[C_{\ell}(S)=\phi\underbrace{\circ(\phi\circ(\cdots\atop\ell\ {\rm times}))}_{ \ell\ {\rm times}}, \tag{4}\]

where \(\ell\) is the index of the spatial scale - the \(C_{\ell}(S)\) for larger values of \(\ell\) have been increasingly smoothed. The \(W^{a}_{\ell}(S)\) from Eq. (1) are the _details_ of \(S\). They are given by applying a differencing wavelet \(\psi^{a}\) on the partially smoothed images:

\[W^{a}_{\ell+1}(S)=\psi^{a}\circ C_{\ell}(S)=\psi^{a}\circ(\phi\underbrace{ \circ(\phi\circ(\cdots\atop\ell\ {\rm times})}_{\ell\ {\rm times}})). \tag{5}\]

\(W^{a}_{\ell}(S)\) represents features of \(S\) that exist at different spatial scales and exist for different spatial symmetries (as indexed by \(a\)). For larger values of \(\ell\), \(W^{a}_{\ell}(S)\) represents features of \(S\) that have larger spatial extent. The essential feature of the MRD is that different levels are of different sizes and contain distinct information about different spatial scales. This feature enables partitioning of the data to provide access to different information than is provided by the original data.

In this work, we will exclusively use the Daubechies-1 wavelet (\(n_{v}=1\)), also known as the Haar wavelet. For the specific case of the one-dimensional Haar wavelet, the smoothing and differencing wavelets are

\[\phi_{1,1}=\frac{1}{\sqrt{2}}\left(\begin{array}{cc}1&1\end{array}\right), \qquad\psi_{1,1}=\frac{1}{\sqrt{2}}\left(\begin{array}{cc}1&-1\end{array} \right), \tag{6}\]

which are convolved over the input signal \(S\) with stride two. Therefore, the \(W_{\ell}(S)\) details are half as long as the \(C_{\ell-1}(S)\) smooth images from which they are obtained. For two-dimensional inputs, the Haar wavelets are

\[\begin{array}{ll}\phi_{1,2}=\phi_{1,1}\otimes\phi_{1,1},&\psi^{v}_{1,2}=\psi _{1,1}\otimes\phi_{1,1},\\ \psi^{h}_{1,2}=\phi_{1,1}\otimes\psi_{1,1},&\psi^{d}_{1,2}=\psi_{1,1}\otimes \psi_{1,1},\end{array} \tag{7}\]

where \(\otimes\) is the outer product and \(a=v,h,d\) designate the vertical, horizontal, and diagonal differences that are possible in two dimensions. For an MRD of a two-dimensional image, the two-dimensional Haar smoothing wavelet \(\phi_{1,2}\) is a \(2\times 2\) matrix, all of whose entries are \(1/2\). The two-dimensional Haar differencing wavelets \(\psi^{a}_{1,2}\) are also \(2\times 2\) matrices. Thus, \(C_{\ell}(S)\) and the \(W^{a}_{\ell}(S)\) each contain one-quarter as many entries as the \(C_{\ell-1}(S)\) from which they are obtained. The Haar wavelet also has the property that the sums of the details at level \(\ell\) equal the difference of the smoothed images at levels \(\ell-1\) and \(\ell\), when the signal at level \(\ell\) have expanded by repeating each entry. This is reminiscent of the Laplacian pyramid algorithm [1], except that the MRD generates these details constructively and provides orientation information in dimensions larger than one. We provide worked examples of the 2-dimensional Haar decomposition in Apps. B.1 and B.2.
Importantly, the contents of the MRD, the set \(\mathcal{M}_{S}\), form a _partition_ of \(S\), such that \(S\) can be reconstructed via

\[S=\phi^{-1}C(S)+\sum_{\ell=1}^{L}\sum_{a}(\psi^{-1})^{a}W_{\ell}^{a}(S), \tag{8}\]

where \(\phi^{-1}\) and \(\psi^{-1}\) are the "reconstruction" or "inverse" wavelets, respectively. The numerical values of these depend on the conventions adopted for the MRD. We show a concrete example in the appendix where the inverse wavelets are identical to the wavelets used in the MRD.

One implication of Eq. (8) is that \(\mathcal{M}_{S}\) and \(S\) are two fully equivalent, lossless representations of exactly the same information. This can be confirmed by counting degrees of freedom. For a one-dimensional input \(S\) of length \(2^{L}\) that we have decomposed with the Daubechies-1 wavelet, \(W_{\ell}(S)\) has length \(2^{L-\ell}\), and \(\mathcal{M}_{S}\) has length \(1+\sum_{\ell=0}^{L-1}2^{i}=2^{L}\), where the leading \(1\) comes from \(C(S)\) and the remaining terms come from the \(W_{\ell}(S)\). Despite retaining all the information of the original image, the MRD \(\mathcal{M}_{S}\) has rearranged this data in a novel way according to scale and, in dimensions larger than one, orientation.

The MRD reorganizes the information contained in the input so as to enable direct access to information that is encoded in any way other than purely through local spatial correlations. Heuristically, the arrangement of an MRD is similar to telescopic series. This makes the MRD a convenient set of inputs to a DL network if we want that network to learn - in a size-agnostic way - aspects of data aside from those encoded purely through spatial locality. This stands in contrast to a CNN, which encodes spatial information through filters of fixed, predetermined size.

In this work, we will henceforth only use the Haar wavelet applied to input signals \(S\) with two dimensions - e.g., images. This fixes the index \(a\) to take on three values - corresponding to vertical, horizontal, and diagonal differences, as in Eq. (7). For notational convenience, we will drop the explicit dependence on \(S\) in the average and the details.

We summarize many of the technical terms we use in this work in App. A. We also connect our terminology to alternate terminology found throughout the literature.

## 3 New Method: The WavPool Block

We introduce the _WavPool_ block to incorporate the wavelet decomposition into a neural network context. The WavPool takes in the levels of the MRD on an input image and applies a dense layer to each component of the transform. We refer to the output of this collection of many dense layers as the multi-resolution perceptron (MRP). The _Pool_ of WavPool is derived from the fact that we apply a maximum pooling operation to the MRP. The pooling is critical to capture the different layers of

Figure 1: The MicroWav operation. The three-tailed operation applies the wavelet decomposition at a defined level \(\ell\) and assigns each of the three details to their own dense layer of size \(N_{\ell}\).
granularity of the transform and allow the training process to determine the importance of different decomposition components.

The fundamental _MicroWav_ layer is shown in Fig. 1. Each MicroWav component, which makes up the first part of the WavPool, operates on a specific level \(\ell\) of the MRD. Each image detail \(W^{a}_{\ell}\) is fed into one of three independent dense layers. The output of each level \(\ell\) has its own hidden size, \(N_{\ell}\). After performing experiments for an optimized network, as discussed in Sec. 4, we find that hidden layers of different sizes lead to better performance.

To turn the output of the MicroWav layers into a classifier, we first collect the dense outputs corresponding to all \(L\) levels of the MRD. We call this combined output a multi-resolution perceptron (MRP): it is inspired by the perceptron Rosenblatt (1958), but the data has been processed by the MRD. We wish to train on all of the content of the MRP, but we also want to allow the layers to have different sizes to accommodate the different information content of each layer of the MRD. We fix the hidden layer sizes to scale inversely with \(\ell\), such that \(\ell\times N_{\ell}\) is constant. This has multiple implications. First, this scaling reduces the number of trainable parameters in the network because higher MicroWav levels have fewer outputs. Second, because this scaling is sub-exponential, the higher MicroWav levels are smaller. This helps the network avoid over-fitting to higher levels of the decomposition, which would otherwise have an unbalanced node-to-input feature ratio. Finally, this ensures that the MRD is not inverted at the end of the network, as it is in previous embeddings of wavelets in neural networks. Instead, this choice enforces _sparsity_ inside the WavPool, effectively compressing the lower levels of the decomposition.

To enable a pooling operation on the differently sized outputs of the MicroWav layers, we apply a padding operation to the output of each of these sub-blocks. This ensures a uniform shape across the block, such that the shape of the stacked output of the MRP is \(L\times 3\times N_{1}\). The resulting block is compressed by a pooling operation. We name the entire network the _WavPool_. The complete WavPool diagram is shown in Fig. 2.

In Tab. 1, we estimate the time complexity of components of the WavPool. The WavPool block itself derives most of its time complexity from the dense layers that compose the majority of its calculations. The Haar wavelet itself has linear time complexity with the size of its input \(n\), and the operations within MicroWav are sequentially applied. Thus, the MicroWav has time complexity of \(O(n)+O(\sum_{t=0}^{T}N_{t}\times n^{2})\). Within the MicroWav, \(T=3\), so this complexity can be simplified to \(O(n+3n^{3})\), or \(O(n^{3})\). This further applies to the full block. The stacking operation takes place in constant time, and the 3D Pooling operation is an operation \(O(n^{3})\). As all these operations are

Figure 2: Architectural diagram of the WavPool block. The block comprises individual hidden-layer triplets (“MicroWav”; Fig. 1) that account for each level of the MRD, which are then fed into a pooling layer. This produces an output with shape \(L\times 3\times\max(N_{\ell})\). A 3D Pooling operation is applied to this output, followed by a dense layer. The breakdown of parameter and time complexity of each point in the network can be seen in Table 1
linearly applied, this produces a block with time complexity of \(O(H)+O(\sum_{t=0}^{T}N_{t}\times H^{2})+O(L)+O(\prod_{i=0}^{3}(k_{i})+O(m\times N_{ t}^{2})\), or more compactly, \(O(n^{3})\) The matrix multiplication required to compute dense layers is also \(O(n^{3})\). Therefore, the MLP we compare to has a time complexity of \(O(2n^{3})\), putting the complexity on par with the WavPool. Convolutional networks are more dependent on the size of their filters and number of channels, which is described as \(O(m_{in}\times m_{out}\times\prod_{i=0}^{D}k_{i}^{2})\), He and Sun (2014), where \(m\) refers to the channels in the convolution and \(k_{D}\) is the kernel size in each dimension. Because \(k_{i}\) has a maximum size of \(n\) (i.e., the kernel cannot exceed the size of the input data), in the case of our 2D experiment we constrain the complexity of the CNN to be \(O(m_{in}\times m_{out}\times n^{4})\) at a maximum. In some circumstances (e.g., with very large kernels or with many in and out channels), the CNN is more time-intensive than the maximum-complexity form of the WavPool.

## 4 Experiments: Comparing WavPool, MLP, and CNN

To investigate the efficacy of WavPool networks, we conduct experiments that compare the WavPool network, a two-hidden layer dense network, and a CNN. The CNN comprises two convolutional layers, each of which has a batch norm layer between them.

We test the networks on multiple datasets: MNIST LeCun et al. (2010); Fashion MNIST Xiao et al. (2017), and CIFAR-10 Krizhevsky (2009). In our experiments, CIFAR-10 was transform into greyscale to constrain the problem to two-dimensional space. We perform three trials on each data set with each network. For each trial, we changed the initialization weights and the subset of each dataset used by setting a different random seed. For all experiments, we use 4000 training images and 2000 validation images. Each trial is run for 120 epochs maximum - though subject to early stopping if the validation loss stalled after waiting for five epochs of patience. Networks are built with PyTorch Paszke et al. (2019). The wavelet-based networks are built with PyWavelet Lee et al. (2019) as a calculation method for the wavelet decomposition. All calculations were performed on a 10-core CPU.

First, we perform these experiments with non-optimized parameters, for the purpose of showing how each block performs without an in-depth parameter exploration period. The parameters are chosen by leaving most parameters as the default parameters of PyTorch. The results of the non-optimized experiments can be found in Tab. 2. These values are the average of the scores obtained on the validation sets of the three different runs. Values without error bars did not have significant variation between runs.

Next, we use Bayesian hyperparameter optimization methods Nogueira (2014, 2015) to find the best possible networks for each architecture and task combination. This was done three times with the goal of demonstrating the best capabilities of each network. This includes optimizing the learning rate, hidden layer sizes, and optimizer for all networks, and additionally the convolution kernel size and number of hidden channels for the CNN. The exact range of the optimization space can be found in App. D. The quality of a network is assessed after an abbreviated training period of 20 epochs, based on the F1 score on the validation set. The final network is then re-initialized and allowed to run for up to 120 epochs subject to early stopping as in the non-optimized case. The results of the experiments on optimized networks are shown in Tab. 3. We also provide a relative comparison of performance on these metrics in Tab. 4. We show the results of these experiments in Fig. 3. The solid lines show the mean training performance, the shaded regions show the envelope of the performance across the three runs, and the dotted lines shown the results from the best-performing validation run.

\begin{table}
\begin{tabular}{l l l} Step & Output Size & Time Complexity \\ \hline MicroWav (single \(\ell\)) & \(1\times 3\times N_{\ell}\) & \(O(n+3n^{3})\) \\ Padding/Stacking & \(L\times 3\times N_{1}\) & \(O(L)\) \\
3D Pooling & \((L-k_{1}+1)\times(4-k_{1})\times(N_{1}-k_{2}+1)\) & \(O(n^{3})\) \\ Dense Output & \(m\) & \(O(n^{2}m)\) \\ \end{tabular}
\end{table}
Table 1: The time complexity and resulting matrix size for each step in the WavPool operation. \(N_{\ell}\) is the size of the output of the dense network that has been applied to each _detail_\(W_{\ell}\); \(k_{n}\) is the selected kernel size, a hyperparameter for the shape of the pooling operation; and \(m\) is the selected output size.
WavPool requires significantly more trainable parameters than a single-channel CNN block with comparable architecture and complexity. This is attributed to the MicroWav's use of dense layers as means of holding trained component information. Reduction in the size of these dense layers is still feasible, as no operations (e.g., weight-pruning or quantization) have been applied to these dense layers in the MicroWav blocks, nor have they been individually tuned for ideal size given the size of the wavelet transform output. Because many signals are sparse in the wavelet representation Mallat (2008), further optimization of the WavPool network is an interesting and promising topic for future work. Nevertheless, the results show that the WavPool is less information-hungry than the CNN. For the MNIST and Fashion MNIST tasks, the number of parameters in the WavPool only increases by 2% from the non-optimized to the optimized version, while the optimized versions of the CNN require many more parameters than the non-optimized version. For WavPool, the number

\begin{table}
\begin{tabular}{l l l l l} Network & Task & Parameters & ROC AUC & Accuracy \\ \hline \hline \multirow{3}{*}{WavPool} & MNIST & 182192 & **0.953\(\pm\)0.001** & **0.916\(\pm\)0.002** \\  & Fashion-MNIST & 182192 & 0.907\(\pm\)0.001 & 0.832\(\pm\)0.001 \\  & CIFAR & 235964 & **0.629\(\pm\)0.004** & **0.331\(\pm\)0.007** \\ \hline \multirow{3}{*}{MLP Block} & MNIST & 209890 & 0.942 & 0.895\(\pm\)0.001 \\  & Fashion-MNIST & 209890 & **0.91\(\pm\)0.001** & **0.839\(\pm\)0.002** \\  & CIFAR & 273250 & 0.604\(\pm\)0.002 & 0.288\(\pm\)0.003 \\ \hline \multirow{3}{*}{CNN Block} & MNIST & **6806** & 0.95 & 0.91 \\  & Fashion-MNIST & **6806** & 0.898 & 0.815 \\ \cline{1-1}  & CIFAR & **9046** & 0.603\(\pm\)0.003 & 0.285\(\pm\)0.006 \\ \end{tabular}
\end{table}
Table 2: Parameter count and validation score metrics for non-optimized networks. The network hyperparameters were arbitrarily chosen, along with the width of the networks/kernel size. The receiver operating characteristic (ROC) curve and accuracy values are for each network are obtained from a validation set. Without tuning, the WavPool performs the best on two out of the three tasks. The MLP narrowly outperforms WavPool on the Fashion-MNIST task.

Figure 3: Performance of optimized networks on Fashion MNIST and CIFAR-10. The error bars show the variation of three trials with different network initialization and different dataset subsets. Validation for a single trial is represented with a dashed line.
of parameters does not significantly increase for the Fashion-MNIST and MNIST tasks. The number of parameters decreases for CIFAR-10, while increasing predictive power significantly.

For both non-optimized and optimized experiments, the WavPool outperforms the MLP and the CNN with respect to accuracy and AUC ROC for almost all classification problems. The exception is that the MLP outperforms WavPool at the sub-percent level on Fashion-MNIST in the non-optimized case. Despite its relatively high complexity (e.g., more independent layers) compared to a basic MLP, the WavPool contains at least \(10^{4}\) fewer trainable parameters in all cases. Trainable weights in the network are the ones in each MicroWav layer (see Fig. 1) and on the final dense layer of the WavPool (see Fig. 2). We show this from the increase in accuracy across the different blocks tested. While the blocks were exposed to the same amount of data in identical training schemes, the wavelet showed a \(\sim 10\%\) increase in accuracy over a standard MLP and a CNN block for tests on CIFAR-10. CIFAR-10 is itself a real image dataset, so the correlation of the pixels within a given image is both local and non-local Krizhevsky (2009). The decomposition of the input applied by WavPool is lossless, yet contains non-local and local information broken into different views of the data. The initial decomposition allows different parts of the network to learn information independently.

## 5 Conclusion

In this work, we demonstrate the functionality and efficacy of a multi-resolution decomposition (MRD) within a neural network. Specifically, we decompose data using a Haar wavelet and train classifiers on this stacked, decomposed data. On each level of the transformed data, we apply a dense layer, which we refer to as a MicroWav unit. By training all of the MicroWav units in parallel, we create a multi-resolution perceptron (MRP), on which we use pooling to create the WavPool block. We observe better performance with this classifier relative to a comparable CNN on MNIST, Fashion-MNIST, and CIFAR-10 datasets.

It is notable that WavPool has the capacity to outperform a CNN with comparable architectural size and complexity. CNNs abstract scale information from an image using fixed filter sizes and dynamic, trainable weights. The abstraction comes at the expense of increasing the size of the data vector in some implementation scenarios. In contrast, the MRP uses filters with fixed values, but varying in kernel size/spatial extent, to partition the data into components of varying size without increasing the size of the input data vector. We conclude that the multi-resolution aspect of the MRP makes information from all scales of the image readily available to the classifier. In addition, on multichannel data, we expect that WavPool will have better scaling properties while retaining its high performance.

The MRP also holds promise for a wider range of applications. Similar techniques in combination with CNNs have shown promising, near-state-of-the-art performance on a variety of tasks Chen et al. (2017); Lai et al. (2017); Liu et al. (2018); Fujeda et al. (2018); Yao et al. (2022). Supplementing CNNs with MRPs or MicroWav-like blocks in these architectures is a promising route for future investigation. There is also potential for this method in Quantum Neural Network contexts. For example, the Haar wavelet is an operation requiring only 1 or -1, (as seen in equation 6 and 7),

\begin{table}
\begin{tabular}{l l l l l} Network & Task & Parameters & ROC AUC & Accuracy \\ \hline \hline \multirow{3}{*}{WavPool} & MNIST & 186722 & **0.978\(\pm\)0.001** & **0.96\(\pm\)0.001** \\  & Fashion-MNIST & 186722 & **0.925** & **0.865** \\  & CIFAR & 201672 & **0.658** & **0.386\(\pm\)0.001** \\ \hline \multirow{3}{*}{MLP Block} & MNIST & 216250 & 0.947 & 0.905 \\  & Fashion-MNIST & 217045 & 0.913\(\pm\)0.001 & 0.843\(\pm\)0.001 \\  & CIFAR & 215290 & 0.599 & 0.279 \\ \hline \multirow{3}{*}{CNN Block} & MNIST & **96375** & 0.977\(\pm\)0.002 & 0.959\(\pm\)0.003 \\  & Fashion-MNIST & **154154** & 0.887\(\pm\)0.012 & 0.797\(\pm\)0.02 \\ \cline{1-1}  & CIFAR-10 & **125175** & 0.642\(\pm\)0.01 & 0.352\(\pm\)0.019 \\ \end{tabular}
\end{table}
Table 3: The same networks and datasets from table 2, but optimized using a Bayesian Optimization scheme.
so it can be easily implemented in the schema required for quantum neural networks, because the operations involved in the Haar wavelet mimic those used on qubits Kwak et al. (2021).

## Acknowledgments

We acknowledge the Deep Skies Lab as a community of multi-domain experts and collaborators who've facilitated an environment of open discussion, idea-generation, and collaboration. This community was important for the development of this project. We thank Aleksandra Ciprijanovic, Andrew Hearin, and Shubhendu Trivedi for comments on the manuscript. This manuscript has been authored by Fermi Research Alliance, LLC under Contract No. DE-AC02-07CH11359 with the U.S. Department of Energy, Office of Science, Office of High Energy Physics.
