# WIRE: Wavelet Implicit Neural Representations

Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan,
Ashok Veeraraghavan, Richard G. Baraniuk
Rice University
https://vishwa91.github.io/wire

###### Abstract

Implicit neural representations (INRs) have recently advanced numerous vision-
related areas. INR performance depends strongly on the choice of the nonlinear
activation function employed in its multilayer perceptron (MLP) network. A
wide range of nonlinearities have been explored, but, unfortunately, current
INRs designed to have high accuracy also suffer from poor robustness (to
signal noise, parameter variation, etc.). Inspired by harmonic analysis, we
develop a new, highly accurate and robust INR that does not exhibit this
tradeoff. Wavelet Implicit neural REpresentation (WIRE) uses a continuous
complex Gabor wavelet activation function that is well-known to be optimally
concentrated in space-frequency and to have excellent biases for representing
images. A wide range of experiments (image denoising, image inpainting, super-
resolution, computed tomography reconstruction, image overfitting, and novel
view synthesis with neural radiance fields) demonstrate that WIRE defines the
new state of the art in INR accuracy, training time, and robustness.

##  1 Introduction

Implicit neural representations (INRs), which learn a continuous function over
a set of data points, have emerged as a promising general-purpose signal
processing framework. An INR consists of a multilayer perceptron (MLP) with
alternating linear layers and element-wise nonlinear activation functions.
Thanks to the MLP, INRs do not share the locality biases that can limit the
performance of convolutional neural networks (CNNs). Consequently, INRs have
advanced the state of the art in numerous vision-related areas, including
computer graphics [29, 30, 24], image processing [11], inverse problems [43],
and signal representations [42].

Figure 1: Wavelet implicit neural representation (WIRE). We propose a new
nonlinearity for implicit neural representations (INRs) based on the
continuous complex Gabor wavelet that has high representation capacity for
visual signals. The top row visualizes two commonly used nonlinearities: SIREN
with sinusoidal nonlinearity and Gaussian nonlinearity, and WIRE that uses a
continuous complex Gabor wavelet. WIRE benefits from the frequency compactness
of sine, and spatial compactness of a Gaussian nonlinearity. The bottom row
shows error maps for approximating an image with strong edges. SIREN results
in global ringing artifacts while Gaussian nonlinearity leads to compact but
large error at edges. WIRE produces results with the smallest and most
spatially compact error. This enables WIRE to learn representations rapidly
and accurately, while being robust to noise and undersampling of data.

Currently, INRs still face a number of obstacles that limit their use. First,
for some applications, especially those with high-dimensional data such as 3D
volumes, fitting an INR to high accuracy can still take too long (tens of
seconds) for real time applications. Second, INRs are not robust to signal
noise or insufficient measurements. Indeed, most works on INRs in the
literature assume virtually no signal noise and large amounts of data. We find
in our own experiments that current INR methods are ineffective for tasks such
as denoising or super-resolution. Finally, INRs still have room for
improvement in representational accuracy, especially for fine details.

In this paper, we develop a new, faster, more accurate, and robust INR that
addresses these issues and takes INR performance to the next level. To achieve
this, we take inspiration from harmonic analysis and reconsider the nonlinear
activation function used in the MLP. Recent work has shown that an INR can be
interpreted as a structured signal representation dictionary [53], where the
activation nonlinearity dictates the atoms of the dictionary. For example, the
sine activation creates a pseudo-Fourier transform representation of the
signal that is maximally concentrated in the frequency domain [53].

An important conclusion one can draw from the past four decades of harmonic
analysis research is that Fourier methods are suboptimal for representing the
kinds of signals that feature in typical vision tasks [26]. These kinds of
signals, e.g., natural images from photographs, are much more concisely and
robustly represented using wavelet atoms that are optimally concentrated in
space–frequency. Sparse compositions of wavelet atoms are known to have
excellent biases for representing images; cf. the seminal work in computer
vision (e.g., Laplacian pyramid), computational neuroscience [32], and the
JPEG2000 compression standard.

In this paper, we introduce Wavelet Implicit neural REpresentation (WIRE), a
new INR based on a complex Gabor wavelet activation function (see Figure 1).
Through a wide range of experiments, we demonstrate that WIRE defines the new
state of the art in INR accuracy, training time, and robustness. We showcase
that WIRE’s increased robustness is particularly useful for solving difficult
vision inverse problems, including image denoising (robustness), image
inpainting and super-resolution (superior interpolation), and 2D computed
tomography (CT) reconstruction (solving higher-dimensional inverse problems).
WIRE also outperforms other INRs for signal representation tasks such as
overfitting images and learning point cloud occupancy volumes. Finally, we
show that WIRE enables faster, more robust novel view synthesis with neural
radiance fields (NeRF) [29] from critically few training views.

##  2 Prior Work

#### Regularization for inverse problems.

Inverse problems involve estimating a signal from a linear or nonlinear set of
measurements. Inevitably, the measurements are degraded by noise (such as
camera readout or photon noise), or the problem is ill-conditioned,
necessitating regularization. There are many forms of regularization,
including ridge regression, Lasso [46], total variation (TV) [10], and
sparsity-based [7] techniques that seek to penalize the $\ell_{1}$ norm the
signal or some transform thereof. In the past decade, data-driven
regularization, including overcomplete dictionary-based [4] and generative
network-based [31, 37, 36] ones, have been developed. The classical model-
based approaches are inadequate for severely ill-conditioned problems, while
the data-driven ones critically depend on data.

#### Convolutional neural networks (CNNs).

CNNs, the most popular neural network architectures in computer vision for the
past decade, have been shown to exhibit strong implicit biases that favor
image-like signals. This has been demonstrated with works like deep image
prior (DIP) [48] and its variations [21, 15] that produce remarkable results
on image-related linear inverse problems without any prior training data.
However, such CNN-based priors are tied to a discrete grid-like signal
representation which is not applicable to problems such as novel view
synthesis, or for solving ordinary and partial differential equations, and not
scalable for very high dimensional signals such as 3D tomographic volumes,
gigapixel images, or large point clouds.

#### Deep image prior.

Neural networks, and particularly CNNs, exhibit implicit biases due to their
specific architectures (such as a UNet [38]), implying that even untrained
neural networks can be used for regularization. This was leveraged to build a
deep image prior (DIP) [48] that produces outputs that tend to look like
images. The key idea is to recast regularization as optimizing for the weights
of the network for each instance of the problem. The performance of DIPs is
considerably superior to classical regularization approaches. However, DIPs
exhibit good performance only when over-parameterized and are tied to a grid-
like discretized representation of the signal, implying DIPs do not scale to
high dimensional signals such as point clouds with a large number of points.
The issue of computational cost has been addressed to a certain extent by the
deep decoder [21] and the DeepTensor [39], but they still need the signal to
be defined as a regular data grid such as a 2D matrix or 3D tensor.

#### Implicit representations.

INRs are continuous learned function approximators based on multilayer
perceptrons (MLPs). The continuous nature of INRs is particularly appealing
when dealing with irregularly sampled signals such as a point clouds. Since
its first widespread usage in novel view synthesis in graphics [29], INRs have
pervaded nearly all fields of vision and signal processing including rendering
[24], computational imaging [6, 12], medical imaging [51], and virtual reality
[16].

The popular choice of the ReLU nonlinearity in standard neural networks has
been empirically shown to result in poor approximation accuracy in INRs. This
has been remedied by several modifications to the MLP including the so-called
positional encoding [44, 30], as well as various choices of nonlinearity such
as the sinusoidal function [42] and the Gaussian function [34]. A closely
related work is the Gabor wavelet-based multiplicative filter networks (MFN),
where the output after each layer is multiplied by a Gabor filter. The output
then results in a combination of exponentially many Gabor wavelets, thereby
resulting in large capacity. Numerous architectural changes have also been
proposed that leverage multiscale properties of visual signals to accelerate
the INR training procedure including adaptive block decomposition [27], kilo-
NeRF [35], and predicting the Laplacian pyramid of the signal [40].

INRs can now train on signals nearly instantly [30] thanks to these numerous
advances. However, the high capacity of such INRs precludes robustness —
implying that the signal representation is brittle, resulting in overfitting
to both noise and signal equally. In this paper, we propose the complex Gabor
wavelet as a nonlinearity, which is uniquely well-suited to induce robustness
in INRs.

#### Wavelet transform and the Gabor wavelet.

The Fourier transform decomposes the signal as a sum of sinusoids with
infinite space support, implying that there is no notion of spatial
compactness. The wavelet transform remedies this by decomposing the signal as
a linear combination of translated and scaled versions of a short oscillating
pulse called a wavelet. Wavelets typically result in faster approximation
rates for signals and images than Fourier transform [17], and hence they are
often used for image compression [18, 41] and as a robust prior for inverse
problems of images [20] and videos [49]. In this paper, we show that wavelets
are a universally superior choice for the nonlinearity in INRs due to their
compact support in space and frequency and therefore faster approximation
rates.

##  3 Wavelet Implicit Representations

###  3.1 INR details

Consider an INR function
$F_{\theta}:\mathbb{R}^{D_{i}}\mapsto\mathbb{R}^{D_{o}}$ mapping $D_{i}$ input
dimensions to $D_{o}$ output dimensions, where $\theta$ represents the MLP’s
tunable parameters. The goal is to construct $F_{\theta}$ such that it
approximates a function $g(\mathbf{x})$ of interest, i.e.,
$g(\mathbf{x})\approx F_{\theta}(\mathbf{x})$. For example, $g(\mathbf{x})$
may simply be a ground truth image, represented as a function mapping
coordinates to pixel values. Modeling $F_{\theta}(\cdot)$ as an $M$-layer MLP,
the output at each layer is given by

$\displaystyle\mathbf{y}_{m}=\sigma(W_{m}\mathbf{y}_{m-1}+\mathbf{b}_{m}),$
(1)

where $\sigma$ is the nonlinearity (or nonlinear activation function);
$W_{m},\mathbf{b}_{m}$ are weights and biases for the $m^{\text{th}}$ layer;
$\mathbf{y}_{0}=\mathbf{x}\in\mathbb{R}^{D_{i}}$ is the input coordinate and
$\mathbf{y}_{M+1}=W_{M+1}\mathbf{y}_{M}+\mathbf{b}_{M+1}$ is the final output.

The nonlinear activation $\sigma$ plays a key role in the representation
capacity of the INR (see Fig. 1). Two leading choices include the periodic
$\sigma(x)=\sin(\omega_{0}x)$ used in SIREN [42], and the Gaussian
nonlinearity $\sigma(x)=e^{-(s_{0}x)^{2}}$ used by Ramasinghe et. al.
[34]—both result in significantly higher representation accuracy than ReLU.
However, their high representation capacity is also a drawback, since they can
represent noise with nearly equal accuracy as an image. Our goal is to propose
a nonlinearity $\sigma$ that is well-suited for visual signals such as images,
videos, and 3D volumes but poorly fits noise-like signals.

###  3.2 WIRE

Armed with the insight that a Gabor wavelet achieves optimal time-frequency
compactness, we propose the wavelet implicit representation (WIRE) that uses
the continuous complex Gabor wavelet $\psi$ for its activation nonlinearity:

$\displaystyle\sigma(x)=\psi(x;\omega_{0},s_{0})=e^{j\omega_{0}x}e^{-|s_{0}x|^{2}},$
(2)

where $\omega_{0}$ controls the frequency of the wavelet and $s_{0}$ controls
the spread (or width). The first layer activations have the form

$\displaystyle\mathbf{y}_{1}=\psi(W_{1}\mathbf{x}+\mathbf{b}_{1};\omega_{0},s_{0}),$
(3)

which are copies of the mother Gabor wavelet $\psi$ at scales and shifts
determined by $W_{1}$ and $\mathbf{b}_{1}$. Hence the building blocks of WIRE
are drawn from a dictionary of wavelet atoms. We let the weights of the INR as
well as the outputs be complex-valued to preserve phase relationships
throughout, and we represent real signals by simply taking the real part of
the output and discarding the imaginary part. Just as wavelets combine space
and frequency compactness, WIRE enjoys the advantages of periodic
nonlinearities such as SIREN due to the complex exponential term and the
spatial compactness from the Gaussian window term; recall Figure 1.
Additionally, unlike SIREN, WIRE does not require a carefully chosen set of
initial weights (see Fig. 3) due to the Gaussian window, which creates a
spatially compact output at each layer and produces high quality results with
the default neural network initialization of uniformly random weights
independent of the parameters $\omega_{0},s_{0}$.

(a) Implicit bias with NTK gradient flow

(b) Implicit bias in standard INRs.

Figure 2: Implicit bias in denoising (a) The empirical NTK of finite-width
INRs provides an insight into the implicit bias of INRs. Early trajectories of
NTK gradient flow show WIRE converging to the image faster than the noise,
outperforming all other nonlinearities. Bars indicate one standard deviation
over the dataset. (b) Early iterations of standard training are reflected well
by the relative performances of NTK gradient flow from part (a). Furthermore,
WIRE maintains its advantage against other nonlinearities throughout the
remainder of training.

###  3.3 Implicit bias of WIRE

#### Neural tangent kernel perspective.

As stated, we seek an INR that fits visual signals well but fits noise poorly
in comparison. Inspired by [53], who proposed to compare eigenfunctions of the
empirical neural tangent kernel (NTK) [22] of INRs to understand their
approximation properties, we compare the fitting of noisy natural images using
NTK gradient flow. The NTK gradient flow of INRs accurately captures the
behavior of early training of neural networks, and so in tasks such as
denoising where we regularize via early stopping, the early training behavior
determines the implicit bias. In the lazy training regime of wide neural
networks [25], the fit image at time $t\geq 0$ has value

$\displaystyle F_{\theta_{t}}(\mathbf{x})=[(I-e^{-tK})g](\mathbf{x}),$ (4)

where $I$ is the identity operator, $K$ is the NTK operator on the image’s
spatial domain, and $g$ is the image being fit.

In Fig. 2(a), we apply NTK gradient flow using the empirical finite-width NTK
to a denoising task, fitting the original image with $\mathcal{N}(0,0.05^{2})$
i.i.d. pixel-wise additive noise. Due to the computational intensity of
evaluating the NTK, we evaluate on $64\times 64\times 3$ images from Tiny
ImageNet [2]. Comparing WIRE to other INRs, we see that, as desired, WIRE
prefers to learn the signal in the image early in training rather than the
noise, converging orders of magnitude faster to essentially any given peak
signal-to-noise-ratio (PSNR).

#### Empirical evaluation.

We perform a denoising task analogous to the NTK-based analysis for real INRs
on the 24 $768\times 512\times 3$ images from the Kodak Lossless True Color
Image Suite [1], again with $\mathcal{N}(0,0.05^{2})$ additive noise, in Fig.
2(b). We apply the same INRs as in the NTK example, but train with ordinary
neural network gradient optimization instead of NTK gradient flow. Again, WIRE
drastically outperforms other INRs, converging an order of magnitude faster to
the same PSNR.

###  3.4 Choosing the parameters $\omega_{0},s_{0}$

WIRE’s performance is primarily decided by the constants $\omega_{0},s_{0}$
that control frequency of the sinusoid and width of the Gaussian,
respectively. WIRE outperforms both the SIREN and Gaussian nonlinearities
across a broad range of parameters. Figure 3 shows the approximation accuracy
achieved by WIRE for various parameters. We set the number of hidden layers to
three, and number of hidden features to 256. When $\omega_{0}=0$, we used a
Gaussian nonlinearity. When $s_{0}=0$, we used a sinusoidal nonlinearity. When
both parameters were zero, we used a ReLU nonlinearity. For the denoising
task, we added photon noise equivalent to a maximum of 50 photons per pixel.
We observe from Fig. 3 that WIRE outperforms SIREN, Gauss, and ReLU. Moreover,
the performance is superior for a large swath of values of $\omega_{0},s_{0}$
for both image representation and denoising. The reduced sensitivity to the
exact values of $\omega_{0},s_{0}$ implies that WIRE can be used without
precise information about image or noise statistics.

#### Alternate forms of WIRE.

For problems where complex weights are infeasible, WIRE can be instantiated as
the imaginary (or real) part of the complex Gabor wavelet,
$\psi(x;\omega_{0},s_{0})=\sin(\omega_{0}x)e^{-(s_{0}x)^{2}}$. Note that
setting $s_{0}=0$ results in the sine nonlinearity used in SIREN [42] and
$\omega_{0}=0$ results in Gaussian nonlinearity [34] implying WIRE inherits
the favorable properties of previously proposed nonlinearities. Another
embodiment of WIRE is a Constant-$Q$ Gabor wavelet where $\omega_{0}s_{0}=Q$,
which results in constant fractional bandwidth ($\omega/\delta\omega)$.
Constant-$Q$ Gabor wavelets are often used in music analysis [8, 47], wavelet
transforms [26], and the Laplace transform. Having only a single parameter
makes hyperparameter tuning simpler with a fixed $Q$.

Figure 3: WIRE is robust to the choice of parameters. The plot above shows
accuracy for image representation and denoising with various settings of
$\omega_{0}$ and $s_{0}$. The boxes show special cases with $\omega=0$
corresponding to Gaussian nonlinearity, and $s_{0}=0$ corresponding to SIREN.
WIRE achieves higher accuracy than both SIREN and Gauss on image
representation as well as image denoising tasks (marked by white cross).
Further, WIRE achieves super performance for a large choice of parameters
$\omega_{0},s_{0}$ implying that WIRE is not overly sensitive to the
hyperparameters.

###  3.5 Multidimensional localization

As defined above, WIRE applies the Gabor mother wavelet $\psi$ element-wise to
output of the linear transformation $W_{m}\mathbf{y}_{m-1}$. Hence, the output
of each unit is spatially localized only in the single direction determined by
the corresponding row of $W_{m}$ (see Fig. 11 top). While such highly
anisotropic spatial localization is well-suited for certain kinds of data,
many natural data (e.g., photographic images) are best represented using a
combination of atoms with isotropic and anisotropic spatial localization
(e.g., wavelets and curvelets [9] or wavelets and wedgelets [50]). To achieve
spatial localization along multiple directions, we augment the Gabor mother
wavelet with $D_{m}-1$ additional Gaussian windows:

$\displaystyle\mathbf{y}_{m}={}$
$\displaystyle\psi(W_{m}^{(1)}\mathbf{y}_{m-1}+\mathbf{b}_{m}^{(1)};w_{0},s_{0})$
(5) $\displaystyle~{}~{}~{}\cdot
e^{-\sum_{k=2}^{D_{m}}|s_{0}(W_{m}^{(k)}\mathbf{y}_{m-1}+\mathbf{b}_{m}^{(k)})|^{2}}.$

In two-dimensional settings, such as with natural image data, the resulting
first-layer activations will resemble a mixture of Gabor wavelets and
curvelets (see Fig. 11 bottom). As we will see below in Section 4.4, the more
diverse spatial localization of the resulting 2D WIRE representation
significantly benefits its performance (see Fig. 12 and Fig.13).

##  4 Experiments

WIRE learns representations for all signal classes faster than state-of-the-
art techniques. In addition, WIRE is well-suited to solve a large class of
inverse problems where the number of measurements is far fewer than the
dimensionality of the signal, or when the measurements are corrupted by noise.
For all the experiments below, we implemented the optimization procedure in
PyTorch [33] and used the Adam optimizer [23]. Code was executed on a system
unit equipped with 64GB RAM, and an Nvidia RTX 2080 Ti graphical processing
unit (GPU) with 8GB memory. Unless specified, we used an $\ell_{2}$ loss
function between the measurements and the outputs of INR. No other
regularization was used. We used a learning rate scheduler which decayed the
initial learning rate by $0.1$ at the end of training epochs.

(a) Image representation

(b) Volume representation

Figure 4: WIRE learns faster. The two plots above show representation accuracy
for an image (top row in Fig. 5) and an occupancy volume (bottom row in Fig.
5) over time. Owing to the high approximation capacity of Gabor wavelets for
visual signals, WIRE achieves high accuracy at a faster rate, making it an
appropriate choice for representing visual signals. Figure 5: WIRE has high
representation capacity. The results above show image representation in the
first row and meshes generated with occupancy volumes in the second row with
various nonlinearities. WIRE achieves highest representation accuracy for both
data, underlining its advantages as a signal model.

###  4.1 Signal representation

A common feature enabled by INRs is representation of signals. We evaluate two
tasks for this experiment: representing images and representing occupancy
volumes [28]. In both cases, we used an MLP with three hidden layers with a
width of 300 features for all nonlinearities. For WIRE, we reduced the number
of parameters by half to account for the doubling due to real and imaginary
parts. We did so by reducing the number of hidden features by $\sqrt{2}$. The
parameters for each nonlinearity and the learning rate were chosen to obtain
fastest approximation rate. Specifically, we chose $\omega_{0}=20,s_{0}=10$
for WIRE, $\omega_{0}=40$ for SIREN, and $s_{0}=30$ for Gaussian. We also
compare against multiplicative frequency networks (MFN) [19]. For the
occupancy volume, we sampled over a $512\times 512\times 512$ grid with each
voxel within the volume assigned a 1, and voxels outside the volume assigned a
0. We evaluated the PSNR and structural similarity (SSIM) [52] for images and
intersection over union (IOU) for the occupancy volumes.

Figure 4 shows the approximation accuracy as a function of time for an image
(Kodak dataset) and an occupancy volume (Thai statue). WIRE not only achieves
the highest accuracy, but it does so at a much faster rate than other
approaches. Figure 5 visualizes the final representation of the example image
after 1.6 minutes, and the 3D mesh of the Thai Statue constructed with
marching cubes after 30 minutes. WIRE achieves the highest accuracy both for
images (43.2dB) and for the occupancy volume (0.99), underlying our hypothesis
that INRs equipped with a Gabor nonlinearity have higher approximation
accuracy.

###  4.2 Solving inverse problems of 2D images

WIRE’s inductive bias favors images, and hence can be used for solving linear
inverse problems. To demonstrate the advantages of WIRE as a strong prior for
images, we showcase its performance on image denoising, single image super
resolution, and multiimage super resolution.

#### Image denoising.

To evaluate the robustness of INRs for representing noisy signals, we learned
a representation on a high resolution color image from the DIV2K dataset [3].
We simulated photon noise with an independently distributed Poisson random
variable at each pixel with a maximum mean photon count of $30$, and a readout
count of $2$, resulting in an input PSNR of 17.6 dB. We then learned a
representation on this noisy image with various nonlinearities. In all cases,
we chose an MLP with two hidden layers and 256 features per layer. We also
compared the denoising result with deep image prior (DIP) [48]. Figure 6
visualizes the final result for each nonlinearity along with metrices for each
result. WIRE produces the sharpest image with least amount of residual noise.
Qualitatively, WIRE’s result is similar to DIP’s, implying that WIRE enjoys
inductive biases that make it a good choice for inverse problems.

Figure 6: WIRE is robust to noise. A powerful feature uniquely enabled by WIRE
is the robustness to noisy data. Here, we show an image representation with
added shot noise, resulting in an input PSNR of 17.6dB. Among the various
approaches, WIRE results in the highest PSNR and SSIM of any representation,
thereby naturally resulting in denoising.  Figure 7: WIRE for single image
super resolution. The figure above shows results for a $4\times$ single image
super resolution with various approaches. Thanks to its strong implicit bias,
WIRE results in the sharpest reconstruction with quantitatively higher
reconstruction metrics. Figure 8: Multi-image super resolution. INRS are
particularly appealing for handling data on an irregular grid, such as images
captured with multiple sub-pixel shifts. The figure above shows $4\times$
super resolution with 4 images captured with varying sub-pixel shifts and
rotations. We then solved a joint inverse problem where the high resolution
image is modeled as the output of an INR. WIRE produces the best
reconstruction both quantitatively and qualitatively, implying that WIRE has
favorable interpolation properties for visual signals.

#### Image super resolution.

INRs function as interpolatants, and hence super resolution is expected to
benefit from INRs with good implicit biases. We evaluate this hypothesis by
implementing $4\times$ super resolution on a DIV2K image. The forward operator
can be cast as $\mathbf{y}=A_{4}\mathbf{x}$ where $A_{4}$ implements a
$4\times$ downsampling operator (without aliasing). We then solved for the
sharp image by modeling $\mathbf{x}$ as output of an INR. Figure 7 visualizes
the result on super resolution of image of a butterfly with various
approaches. WIRE produces the sharpest result with crisp details on the
butterfly’s antenna and on the wings. As with denoising, WIRE results are
similar to DIP, establishing the ubiquity of WIRE.

INRs are particularly advantageous when data interpolation needs to be
performed on an irregular grid. An example of such settings is multi-image
super resolution where the images are shifted and rotated with respect to each
other. Figure 21 shows an example of $4\times$ super resolution with four
images (and hence $25\%$ compression) from the Kodak dataset [1] simulated
with a small sub-pixel motion between them. The forward operator is then
$\mathbf{y}^{k}=A^{k}_{4}\mathbf{x}$ where $A^{k}_{4}$ encodes the
downsampling, and translation and rotation for the $k^{\text{th}}$ image. The
visualizations in the figure demonstrate that WIRE achieves the highest
accuracy and is qualitatively better at reconstructing high frequency
components. In contrast, the Gaussian nonlinearity leads to a blurry
reconstruction, while SIREN results in ringing artifacts.

Figure 9: Computed tomography reconstruction. Inverse problems with noisy
undersampled data require a strong signal prior for robust reconstruction.
Here, we show CT-based reconstruction with 100 angles for a $256\times 256$
image ($2.5\times$ compression) with various approaches. WIRE results in sharp
reconstruction, exposing features that are blurry, or with ringing artifacts
in reconstructions with other approaches. WIRE is hence a strong signal prior
for images, and can solve a large class of inverse problems. Figure 10: Novel-
view synthesis with neural radiance fields. INRs have shown most promise in
novel-view synthesis where the transmittance and color at each 3D voxel is
modeled as output of INRs. Here, we show that WIRE is well-suited for novel-
view synthesis with no additional positional encoding. WIRE not only achieves
higher accuracy (+0.2dB) with fewer epochs, but captures details that are
missed out by other nonlinearities, such as the rod connecting the ride cymbal
to its stand and the anisotropic reflections on the cymbals.

#### Computed tomography (CT) reconstruction.

Strong signal priors are critical for solving underconstrained problems, and
CT reconstruction is one such example. We emulated 100 CT measurements of a
$256\times 256$ x-ray colorectal image [13]. Figure 9 shows the final
reconstruction with various approaches. WIRE results in the sharpest
reconstruction with clearly pronounced features. SIREN performs the second
best but has striation artifacts that are expected from an unregularized
reconstruction. The Gaussian nonlinearity results in overly smooth results.
WIRE can hence be used as a robust prior for inverse problems with noisy and
undersampled measurements.

###  4.3 Learning neural radiance fields

INRs have been leveraged successfully for novel-view synthesis with neural
radiance fields (NeRF) [29]. Given images from a sparse set of view points,
the goal is to render an image from a different view point that is not in the
training set. NeRF achieves this by training a common INR that takes 3D
location and viewing directions as inputs (hence a 5D input), and produces
transmission and color at that location. Images are then produced by
integrating along lines that pass through each view’s lens (pinhole). The
simplest NeRF architecture consists of positional encoding, and two MLPs
equipped with ReLU for transmission and color values. We show that WIRE
without any positional encoding produces higher quality results within fewer
epochs.

Figure 11: First layer outputs with multi-dimensional WIRE. The figure above
shows outputs after first hidden layer with WIRE and 2D WIRE. We observe that
2D WIRE has spatially compact outputs due to the second Gaussian window, while
WIRE has elongated structures orthogonal to the Gaussian window.

We trained NeRFs for reconstruction on the synthetic drum dataset [29]. Each
image was downsampled to a resolution of $200\times 200$. To show the
advantages of WIRE, we trained the radiance field with only 25 images instead
of the default 100 images. We used the “torch-NGP” codebase [45] for training
the NeRF model. For all experiments, we chose a 4-layered MLP with a width of
128 features for each layer. Parameters and learning rate were chosen to
achieve fastest rate of increase of approximation accuracy on the validation
dataset. Figure 10 shows results with various nonlinearities. WIRE produces
highest accuracy (+0.2dB) with fastest rate of increase. WIRE learns features
absent in outputs of other nonlinearities such as the rod connecting the ride
cymbal to its stand and the anisotropic reflections on the cymbal.

###  4.4 Multi-dimensional WIRE comparisons

As we discussed in Section 3.5, WIRE can be instantiated as a multi-
dimensional non-linearity. One advantage of a two-dimensional WIRE is that its
activations tend to be compact along both spatial axes (see Fig. 11). This
enables a more accurate fit for signals that are composed of spatially compact
structures, such as many natural images. Figure 12 demonstrates the advantage
of 2D WIRE’s more general spatial localization for representing an image of
point singularities. 2D WIRE results in a sharper representation than 1D WIRE,
which blurs out some of the points.

(a) Ground truth

(b) 2D WIRE

(c) WIRE

(d) Accuracy vs. time

Figure 12: Multi-dimensional localization. The spatially compact nature of 2D
WIRE enables representing sparse images. In this example, we show
representation of a $256\times 256$ image with 256 non-zero values. 2D WIRE
represents each dot sharply while WIRE tends to blur the features along one of
the two axes. Both WIRE and 2D WIRE converge rapidly compared to other
approaches, as visualized in the plot in the bottom right corner.

The spatial localization of multi-dimensional WIRE is also of great advantage
for solving inverse problems more robustly. Figure 13 compares WIRE and 2D
WIRE on denoising, CT reconstruction, and 4$\times$ super-resolution tasks. In
all tasks, $\omega_{0}$ and $s_{0}$ of the mother wavelet were chosen for best
performance, and the total number of parameters were the same for WIRE and 2D
WIRE. Across the board, 2D WIRE achieves higher accuracy than WIRE. 2D WIRE
also learns sharper features for the CT reconstruction of lungs while WIRE’s
tend to be more blurry. Similarly, in the super-resolution task, 2D WIRE
learns improved high frequency features to represent the center of the flower.

Figure 13: Performance with multi-dimensional WIRE. The figure above shows
various linear inverse problems solved with INRs equipped with WIRE and 2D
WIRE. Across the board, 2D WIRE achieves higher performance in terms of PSNR
and SSIM. Visually, we observe that CT reconstruction and super resolution is
significantly sharper with 2D WIRE owing to the compact nature of activations.

##  5 Conclusions

We have proposed and validated the advantages of WIRE that equips INRs with a
complex Gabor wavelet activation nonlinearity. We have shown with an extensive
set of experiments that WIRE (a) has higher representation capacity, (b)
achieves higher accuracy at a faster rate, and (c) has strong inductive biases
that make it compelling for solving challenging inverse problems. Other
activation nonlinearities have largely complementary strengths: SIREN has high
representation capacity and trains fast, but is poor at regularizing inverse
problems. Positional encoding has lower capacity but is a good choice for
novel-view synthesis. Gaussian nonlinearity is more favorable for denoising
tasks. However, WIRE inherits the best properties of all of the above
activation nonlinearities, and hence is the current go-to INR solution for
signal representation and solving inverse problems.

##  6 Acknowledgements

This work was supported by NSF grants CCF-1911094, IIS-1838177, and
IIS-1730574; ONR grants N00014-18-12571, N00014-20-1-2534, and MURI
N00014-20-1-2787; AFOSR grant FA9550-22-1-0060; and a Vannevar Bush Faculty
Fellowship, ONR grant N00014-18-1-2047.

##  Appendix A Appendix 1: Experimental Details

###  A.1 WIRE initialization

INRs like SIREN [42] strongly depend on initialization to obtain accurate
representation. WIRE does not require any initialization except for the
default uniform weights. However, since WIRE consists of a complex sinusoidal
term, it marginally benefits from SIREN-like initialization. To understand the
dependence, we evaluated approximation accuracy for image representation (no
noise), and image denoising (20dB image noise). Here, a SIREN-like weight
initialization implies the first layer weights are drawn from
$\mathcal{U}(-1/N,1/N)$ and the weights of the rest of the layers are drawn
from $\mathcal{U}(-\sqrt{6/(\omega_{0}N)},\sqrt{6/(\omega_{0}N)})$, where $N$
is the number of input features and $\mathcal{U}(a,b)$ is a uniform
distribution over $[a,b]$. A normal weight initialization involves drawing
weights from $\mathcal{U}(-1/\sqrt{N},1/\sqrt{N})$ for all layers. Fig. 14
compares the representation accuracy for SIREN-like and standard
initialization. In both cases, we see that the trends are nearly similar;
SIREN-like initialization results in up to 1dB higher accuracy. Hence WIRE is
largely robust to initial parameters which enables easy tuning across a large
range of hyperparameters $\omega_{0},s_{0}$.

Figure 14: Effect of initialization. The plots show approximation accuracy for
image representation and image denoising (20dB input PSNR) across training
epochs with SIREN-like weight initialization and standard initialization. WIRE
is robust to the initial weights, but marginally benefits from a SIREN-like
initialization. Figure 15: Layer outputs for WIRE. The input image is a
Siemens star test image that contains all spatial frequencies and all angles.
The patches show outputs (same size as image) of each hidden feature in layer
one and two for a two hidden-layer MLP equipped with WIRE nonlinearity. WIRE
results in sparse images which enables high representational capacity for
images, as shown in Fig. 16. Figure 16: Visualization of hidden layer outputs.
The figure above visualizes outputs of hidden features in the two layers for
the Siemens sector test image shown in Fig. 15. WIRE uniquely results in
sparse images, which enables high accurate representation of high frequency
parts of the image (center of the sector).

###  A.2 WIRE layer visualizations

Gabor wavelets uniquely enable space–frequency localization, a property we
observe is inherited by WIRE. To evaluate this hypothesis, we visualized the
output WIRE composed of an MLP with two hidden layers and 181 hidden features
each. We then learned a representation for a Siemens star test image that
consists of all spatial frequencies and orientations. Fig. 15 visualizes the
input image real and imaginary outputs of 64 hidden features with least
variance. The output of each first layer feature consists of one-dimensional
Gabor wavelets at various orientations, while the outputs of second layer
consist of sparsely populated images.

Fig. 16 visualizes outputs at each layer for various nonlinearities and the
final approximated image for the Siemens star test image. The sparse outputs
of second layer are evidently unique to WIRE. Gauss has outputs that look less
spares, while SIREN and ReLU with positional encoding result in dense outputs.
This has a direct consequence on approximation capacity for high frequency
parts of the signal. The final result in the bottom row shows that the sparse
nature of outputs of WIRE enables high approximation accuracy with
qualitatively better features at the center of the image which consists of
highest spatial frequenices. Gauss follows next as it results in the second
most sparse outputs at each layer. SIREN and ReLU with positional encoding
alike produce blurry outputs at the center, primarily due to the non-compact
nature of outputs. WIRE’s ability to decompose images as a linear combination
of sparse images results in high representational capacity for the same number
of parameters, as we verify empirically in the next section.

###  A.3 Sensitivity to training parameters

WIRE is a promising INR model that achieves high representation accuracy and
is robust to a wide range of training parameters. We demonstrate the efficacy
of WIRE in this section with several sensitivity analyses.

#### Effect of learning rate.

WIRE performs well for a large range of learning rates. To understand the
performance trends, we learned an image representation with added noise (20dB
input PNSR) with various nonlinearities. We used a 2-layer MLP with 256 hidden
features per layer. Fig. 17 shows the maximum representation PSNR with varying
learning rates. WIRE has a stable and significantly higher accuracy compared
to other approaches. Interestingly, the highest accuracy is achieved at a high
learning rate of $2\times 10^{-2}$. This behavior is also observed with deep
image prior [48] where a larger learning rate enabled stronger regularization.
Such a similar behavior implies WIRE enjoys strong inductive biases and hence
is amenable to solve inverse problems.

Figure 17: Effect of learning rate. The plot above shows approximation
accuracy for representing a noisy image (input PSNR of 20dB) with various
nonlinearities. WIRE is robust to learning rate, and produces best results
with high learning rate of $2\times 10^{-2}$.

#### Effect of number of layers.

Fig. 18(a) shows a plot of representation accuracy of an image for varying
number of hidden layers with various nonlinearities. In each case, the number
of hidden features were set to 256. We reduced the learning rate with
increasing layers to avoid divergence. WIRE uniformly outperforms other
approaches (except with 0 hidden features), as is to be expected as Gabor
wavelets enable high approximation accuracies for images. Interestingly, for a
large number of hidden layers ($\geq 3$), WIRE performance is similar to SIREN
and Gaussian nonlinearity. This is to be expected as the network has a large
capacity with so many layers. However, a large number of layers is
computationally expensive and often results in an unstable learning regime.
WIRE therefore is a reliable choice for small to medium number of hidden
layers for most cases.

(a) PSNR vs. number of features

(b) PSNR vs. number of layers

Figure 18: Effect of number of parameters. The plot above shows approximation
accuracy for representing an image with varying number of (a) hidden features
and (b) hidden layers. WIRE outperforms other nonlinearities with 128 or more
hidden features, and one or more layers and is nearly the same as SIREN and
Gaussian nonlinearities for more than 3 layers.

(a) Ground truth image

(b) PSNR vs. number of prjections

Figure 19: CT with varying number of projections. (a) shows the $512\times
512$ ground truth x-ray image of lungs [5] we used for our CT experiments. We
denoised the original image to remove streak artifacts. (b) shows accuracy as
a function of number of measurements with various nonlinearities. Across the
board, WIRE outperforms all other approaches by a considerable margin.

#### Effect of number of features.

Fig. 18(b) shows approximation accuracy for image representation with varying
number of hidden features. In all cases, the number of hidden layers were
fixed to be two. The performance of WIRE is similar to other nonlinearities at
very low number of hidden features, where all models similarly lack sufficient
richness. For higher than 128 features, WIRE outperforms other approaches with
MFN [19] coming a close second.

Figure 20: Effect of number of projections on CT accuracy. The images above
visualize reconstruction for the lungs image shown in Fig. 19(a). WIRE
outperforms all other approaches even with 50 projections ($10\%$
measurements) and is visually pleasing. Figure 21: Multi-image super-
resolution. The figure above visualizes multi-frame super-resolution where
each sub-frame was simulated with a small known sub-pixel shift. WIRE achieves
highest reconstruction accuracy with qualitatively better reconstruction.

(a) Accuracy vs. epochs for 100 images.

(b) Accuracy vs. number of images.

Figure 22: NeRF accuracy on drums dataset. (a) shows training accuracy at each
epoch for various nonlinearities with neural radiance fields when trained with
100 images. WIRE achieves $0.4$dB higher than the next highest (SIREN) when
trained with 100 images and does so in a rapid manner. (b) shows accuracy as a
function of number of images with WIRE outperforming other approaches for all
number of images. Figure 23: Effect of number of images on NeRF accuracy. The
figure above visualizes rendered images with a neural radiance field for
various nonlinearities (across columns) and varying number of images (across
rows). WIRE achieves visually better reconstruction than all other methods for
all numbers of images, thanks to its strong inductive biases that enable
learning the high frequency features of the scene’s radiance field.

###  A.4 Inverse problems

#### Computed tomographic reconstruction.

We showed in Section 4 that computed tomography (CT) benefits from inductive
biases of INRs. Here, we study the effect of number of measurements. Fig.
19(a) shows the ground truth image we used in our experiments. We denoised the
original $512\times 512$ image [5] with BM3D [14] ($\sigma=0.1$) to remove
streak artifacts. We then simulated CT reconstruction with varying numbers of
projections. In each case, we used an MLP with three hidden layers and 256
hidden features per layer. We sampled the INR on a regular grid to first
generate the image, and then use Radon transform to obtain the sinogram. From
the accuracy plot in Fig. 19(b), we see that WIRE achieves higher PSNR than
any other nonlinearity. Fig. 20 visualizes the reconstruction with varying
number of projections for each nonlinearity. The reconstruction is visually
superior even with small number of projections, which is particularly
beneficial for reducing exposure to x-rays during capture.

#### Multi-image super-resolution.

We showed a result on multi-image super-resolution in Section 4. Here, we
provide more details about the experiment. Figure 21 shows the $512\times 768$
dimensional ground truth image from the Kodak dataset [1]. We simulated a
total of four low-resolution images by modifying each $4\times$ downsampled
image by a small translation and rotation, thereby resulting in sub-pixel
motion between the frames. We assumed the transformation $A_{k}$ between the
high-resolution frame $\mathbf{x}$ and each low-resolution frame
$\mathbf{y}_{k}$ was known. We represented the high resolution $\mathbf{x}$ as
output of an INR. In each case, the INR had three hidden layers with 256
hidden features. We then solved a linear inverse problem to estimate the high
resolution image. Figure 21 shows the reconstructed output for each
nonlinearity and their metrics. The inset shows reconstruction of spokes in
the motorcycle. Visually, WIRE generates the sharpest features without any
ringing artifacts. Moreover, WIRE results in 1dB or better reconstruction
accuracy, and $0.04$ higher SSIM.

###  A.5 Neural radiance fields

#### Implementation details.

For all experiments, we used the `torch-ngp` package [45] that implements a
wide variety of approaches for training neural radiance fields. The
architecture consists of two networks that predict transmittance (sigma) and
the color at each voxel respectively. Each of the two networks consisted of an
MLP with four hidden layers and 182 hidden features each. The color MLP took
position (x,y,z) and direction ($\theta,\phi$) as inputs, while the
transmittance MLP took only the position as input. As with all other
experiments, we used $182/\sqrt{2}=128$ hidden features for WIRE to account
for parameter doubling due to complex weights. We downsampled the images by
$4\times$ to ensure that the model and training data fit in the graphical
processing unit’s (GPU) memory. In the results shown in Fig. 10, we used a
total of 25 randomly chosen images to train the NeRF, and then validated it on
100 images. We used a learning rate of $4\times 10^{-4}$ for WIRE and
$2.5\times 10^{-4}$ for all other nonlinearities and reduced it to $0.1\times$
initial value over a total of 2500 training epochs. Except for ReLU, we did
not use any form of positional encoding with other nonlinearities as we wished
to demonstrate the capacity of each nonlinearity by itself.

#### Effect of number of images.

Fig. 22(a) shows accuracy vs. number of epochs for the drums dataset when
trained with all 100 images. WIRE results in highest accuracy within 2500
epochs and converges more rapidly than other approaches. Fig. 22(b) shows
accuracy as a function of number of training images. WIRE achieves 0.1dB
higher than the next competitor SIREN for 25, 50, and 75 images, and 0.4dB
higher when trained with 100 images.

Fig. 23 visualizes one of the reconstructed views for the drums. We varied the
number of images from 25 to 100 and then rendered the image from a novel view.
Visually, WIRE generated the most pleasing results including sharp features of
the cymbals and their stands, and the smooth membrane on the drum. In
contrast, Gaussian nonlinearity results in cloudy artifacts, while SIREN has
high frequency artifacts, especially at lower numbers of images.
ReLU+positional encoding requires all 100 images and considerably more than
2500 epochs to reconstruct the components. In all, WIRE is a a robust solution
for training radiance fields, even with a small number of training samples.

## References

  * [1] Kodak lossless true color image suite.  http://r0k.us/graphics/kodak/, 1999.  Accessed: 2022-11-09.
  * [2] Tiny imagenet visual recognition challenge.  https://www.kaggle.com/c/tiny-imagenet, 2015.  Accessed: 2022-11-03.
  * [3] Eirikur Agustsson and Radu Timofte.  NTIRE 2017 challenge on single image super-resolution: Dataset and study.  In IEEE Comp. Vision and Pattern Recognition (CVPR), July 2017.
  * [4] Michal Aharon, Michael Elad, and Alfred Bruckstein.  K-svd: An algorithm for designing overcomplete dictionaries for sparse representation.  IEEE Trans. Signal Processing, 54(11):4311–4322, 2006.
  * [5] Samuel G. Armato III, Geoffrey McLennan, Luc Bidaut, Michael F. McNitt-Gray, Charles R. Meyer, Anthony P. Reeves, Binsheng Zhao, Denise R. Aberle, Claudia I. Henschke, Eric A. Hoffman, Ella A. Kazerooni, Heber MacMahon, Edwin J. R. van Beek, David Yankelevitz, Alberto M. Biancardi, Peyton H. Bland, Matthew S. Brown, Roger M. Engelmann, Gary E. Laderach, Daniel Max, Richard C. Pais, David P.-Y. Qing, Rachael Y. Roberts, Amanda R. Smith, Adam Starkey, Poonam Batra, Philip Caligiuri, Ali Farooqi, Gregory W. Gladish, C. Matilda Jude, Reginald F. Munden, Iva Petkovska, Leslie E. Quint, Lawrence H. Schwartz, Baskaran Sundaram, Lori E. Dodd, Charles Fenimore, David Gur, Nicholas Petrick, John Freymann, Justin Kirby, Brian Hughes, Alessi Vande Casteele, Sangeeta Gupte, Maha Sallam, Michael D. Heath, Michael H. Kuhn, Ekta Dharaiya, Richard Burns, David S. Fryd, Marcos Salganicoff, Vikram Anand, Uri Shreter, Stephen Vastagh, Barbara Y. Croft, and Laurence P. Clarke.  The lung image database consortium (lidc) and image database resource initiative (idri): A completed reference database of lung nodules on ct scans.  Medical Physics, 38(2):915–931, 2011.
  * [6] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O’Toole.  Törf: Time-of-flight radiance fields for dynamic scene view synthesis.  Adv. Neural Info. Processing Systems, 34:26289–26301, 2021.
  * [7] Richard G Baraniuk, Volkan Cevher, Marco F Duarte, and Chinmay Hegde.  Model-based compressive sensing.  IEEE Trans. Info. Theory, 56(4):1982–2001, 2010.
  * [8] Judith C Brown.  Calculation of a constant q spectral transform.  The J. Acoustical Society of America, 89(1):425–434, 1991.
  * [9] Emmanuel J Candès and David L Donoho.  New tight frames of curvelets and optimal representations of objects with piecewise c2 singularities.  Comm. Pure and Applied Mathematics, 57(2):219–266, 2004.
  * [10] Antonin Chambolle.  An algorithm for total variation minimization and applications.  J. Mathematical Imaging and Vision, 20(1):89–97, 2004.
  * [11] Yinbo Chen, Sifei Liu, and Xiaolong Wang.  Learning continuous image representation with local implicit image function.  In IEEE Comp. Vision and Pattern Recognition (CVPR), 2021.
  * [12] Chiun-Hong Chien and Jake K Aggarwal.  Volume/surface octrees for the representation of three-dimensional objects.  Computer Vision, Graphics, and Image Processing, 36(1):100–113, 1986.
  * [13] Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al.  The cancer imaging archive (tcia): maintaining and operating a public information repository.  J. Digital Imaging, 26(6):1045–1057, 2013.
  * [14] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space.  In IEEE Intl. Conf. Image Processing (ICIP), 2007.
  * [15] Mohammad Zalbagi Darestani and Reinhard Heckel.  Accelerated MRI with un-trained neural networks.  IEEE Trans. Computational Imaging, 7:724–733, 2021.
  * [16] Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun.  Fov-nerf: Foveated neural radiance fields for virtual reality.  IEEE Trans. Visualization and Computer Graphics, 28(11):3854–3864, 2022.
  * [17] Ronald A. DeVore.  Nonlinear approximation.  Acta Numerica, 7:51–150, 1998.
  * [18] Ronald A DeVore, Björn Jawerth, and Bradley J Lucier.  Image compression through wavelet transform coding.  IEEE Trans. Info. Theory, 38(2):719–746, 1992.
  * [19] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter.  Multiplicative filter networks.  In Intl. Conf. Learning Representations, 2020.
  * [20] Lihan He and Lawrence Carin.  Exploiting structure in wavelet-based bayesian compressive sensing.  IEEE Trans. Signal Processing, 57(9):3488–3497, 2009.
  * [21] Reinhard Heckel and Paul Hand.  Deep decoder: Concise image representations from untrained non-convolutional networks.  In Intl. Conf. Learning Representations, 2018.
  * [22] Arthur Jacot, Franck Gabriel, and Clement Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Adv. Neural Info. Processing Systems, 2018.
  * [23] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Intl. Conf. Learning Representations, 2015.
  * [24] Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi.  NeuMIP: Multi-resolution neural materials.  ACM Trans. Graphics, 40(4):1–13, 2021.
  * [25] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington.  Wide neural networks of any depth evolve as linear models under gradient descent.  In Adv. Neural Info. Processing Systems, 2019.
  * [26] Stéphane Mallat.  A wavelet tour of signal processing.  Elsevier, 1999.
  * [27] Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein.  Acorn: Adaptive coordinate networks for neural scene representation.  arXiv preprint arXiv:2105.02788, 2021.
  * [28] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In IEEE Comp. Vision and Pattern Recognition (CVPR), 2019.
  * [29] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In IEEE European Conf. Computer Vision (ECCV), 2020.
  * [30] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, 2022.
  * [31] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  In IEEE Comp. Vision and Pattern Recognition (CVPR), 2017.
  * [32] Bruno A Olshausen and David J Field.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images.  Nature, 381(6583):607–609, 1996.
  * [33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Adv. Neural Info. Processing Systems. 2019.
  * [34] Sameera Ramasinghe and Simon Lucey.  Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps.  In IEEE European Conf. Computer Vision (ECCV), 2021.
  * [35] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  arXiv preprint arXiv:2103.13744, 2021.
  * [36] JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankaranarayanan.  One network to solve them all–solving linear inverse problems using deep projection models.  In IEEE Intl. Conf. Computer Vision (ICCV), 2017.
  * [37] Yaniv Romano, Michael Elad, and Peyman Milanfar.  The little engine that could: Regularization by denoising (red).  SIAM J. Imaging Sciences, 10(4):1804–1844, 2017.
  * [38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Intl. Conf. Medical Image Computing and Computer-Assisted Intervention, 2015.
  * [39] Vishwanath Saragadam, Randall Balestriero, Ashok Veeraraghavan, and Richard G Baraniuk.  Deeptensor: Low-rank tensor decomposition with deep network priors.  arXiv preprint arXiv:2204.03145, 2022.
  * [40] Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard Baraniuk, and Ashok Veeraraghavan.  Miner: Multiscale implicit neural representations.  In European Conf. Computer Vision, 2022.
  * [41] Jerome M Shapiro.  Embedded image coding using zerotrees of wavelet coefficients.  IEEE Trans. Signal Processing, 41(12):3445–3462, 1993.
  * [42] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Adv. Neural Info. Processing Systems, 2020.
  * [43] Yu Sun, Jiaming Liu, Mingyang Xie, Brendt Wohlberg, and Ulugbek S Kamilov.  Coil: Coordinate-based internal learning for imaging inverse problems.  arXiv preprint arXiv:2102.05181, 2021.
  * [44] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Adv. Neural Info. Processing Systems, 2020.
  * [45] Jiaxiang Tang.  Torch-ngp: a pytorch implementation of instant-ngp, 2022.  https://github.com/ashawkey/torch-ngp.
  * [46] Robert Tibshirani.  Regression shrinkage and selection via the lasso.  J. Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996.
  * [47] Massimiliano Todisco, Héctor Delgado, and Nicholas Evans.  Constant q cepstral coefficients: A spoofing countermeasure for automatic speaker verification.  Computer Speech & Language, 45:516–535, 2017.
  * [48] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In IEEE Comp. Vision and Pattern Recognition (CVPR), 2018.
  * [49] Michael B. Wakin, Jason N. Laska, Marco F. Duarte, Dror Baron, Shriram Sarvotham, Dharmpal Takhar, Kevin F. Kelly, and Richard G. Baraniuk.  Compressive imaging for video representation and coding.  In PCS Proc., 2006.
  * [50] Michael B Wakin, Justin K Romberg, Hyeokho Choi, and Richard G Baraniuk.  Wavelet-domain approximation and compression of piecewise smooth images.  IEEE Trans. Image Processing, 15(5):1071–1087, 2006.
  * [51] Yuehao Wang, Yonghao Long, Siu Hin Fan, and Qi Dou.  Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery.  In Intl. Conf. Medical Image Computing and Computer-Assisted Intervention, 2022.
  * [52] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Trans. Image Processing, 13(4):600–612, 2004.
  * [53] Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard.  A structured dictionary perspective on implicit neural representations.  In IEEE Comp. Vision and Pattern Recognition (CVPR), 2022.