#  Order from chaos: Interplay of development and learning in recurrent
networks of structured neurons

Laura Kriener∗,†,1, Kristin Völk∗,1,2,
Ben v. Hünerbein1, Federico Benitez1, Walter Senn1, Mihai A. Petrovici†,1
∗ These authors contributed equally
†Corresponding authors {laura.kriener<EMAIL_ADDRESS>
1Department of Physiology, University of Bern, 3012 Bern, Switzerland.
2Catlab Engineering GmbH, 82284 Grafrath, Germany.

_Behavior can be described as a temporal sequence of actions driven by neural
activity. To learn complex sequential patterns in neural networks, memories of
past activities need to persist on significantly longer timescales than
relaxation times of single-neuron activity. While recurrent networks can
produce such long transients, training these networks in a biologically
plausible way is challenging. One approach has been reservoir computing, where
only weights from a recurrent network to a readout are learned. Other models
achieve learning of recurrent synaptic weights using propagated errors.
However, their biological plausibility typically suffers from issues with
locality, resource allocation or parameter scales and tuning. We suggest that
many of these issues can be alleviated by considering dendritic information
storage and computation. By applying a fully local, always-on plasticity rule
we are able to learn complex sequences in a recurrent network comprised of two
populations. Importantly, our model is resource-efficient, enabling the
learning of complex sequences using only a small number of neurons. We
demonstrate these features in a mock-up of birdsong learning, in which our
networks first learn a long, non-Markovian sequence that they can then
reproduce robustly despite external disturbances._

## Introduction

Adaptation to a complex world requires to evolve complex and flexible
behaviors, which are themselves grounded in neural activity. In order for a
neuronal network to learn a complex sequential pattern, memories of past
activities need to persist on behavioral time scales—significantly longer than
the characteristic relaxation times of single-neuron activity. Typically,
network recurrence is assumed to produce such long transients, but learning in
recurring networks remains a challenging problem. Commonly used approaches can
be divided into two groups. First, one can simply forego recurrent plasticity
and only learn the weights of a “readout” layer (reservoir computing [1]).
Second, the recurrent weights can be learned as well, with the most successful
algorithms using propagated errors to update synaptic weights. However, their
biological plausibility suffers from issues with locality (BPTT [2]), resource
allocation and scaling (RTRL and its approximations [3]), or parameter scales
and tuning (FORCE [4], E-PROP [5]).

We suggest that many of these issues can be alleviated through dendritic
information storage and computation. Specifically, we consider the interplay
between two populations of cortical neurons (for example, in motor and
premotor areas) that undergo two phases of network evolution. During early
development, first, neurons form a sparse, random scaffold of somato-somatic
connections. During later learning, this scaffold is then used to transmit
errors that induce the growth and tuning of new synapses along dendritic
trees. This fully local, always-on plasticity ultimately carves out strong
attractors that generate robust activation sequences. Importantly, our model
makes efficient use of its neuronal resources, allowing the learning of
complex sequences with only a small number of neurons. We demonstrate these
features in a mock-up of birdsong learning, in which our networks learn a
long, non-Markovian sequence (a sample of Beethoven’s “Für Elise”) that they
can reproduce robustly despite severe external disturbances.

Figure 1:  (a) Development and learning in a two-population model of latent
and output neurons. Following neurogenesis (a1), axons (orange) extend to form
a sparse scaffold of somato-somatic connections between pyramidal neurons. Its
structure (phenotype) is controlled by parameters $p$ and $q$ (genotype) (a2).
Inhibition in this scaffold is mediated by interneurons (a3). Following the
development phase, somato-dendritic synapses (green) evolve according to a
three-factor plasticity rule (a4). (b) Evolution of network activity during
learning. Output neurons are first nudged towards a particular target and then
released in order to observe the network’s spontaneous activity (as demarcated
by the red line). Snapshots during early (left), intermediate (middle) and
final (right) stages of training. (c) Accuracy and stability of sequence
replay, measured by the correlation between generated and target activity.
Comparison between our model (capable of recurrent learning) and an equivalent
reservoir (where only latent-to-output connections are learned). Left:
Accuracy during training. Right: Accuracy after repeated replay cycles.

## Network model

The network is comprised of structured pyramidal neurons with a somatic ($u$)
and a dendritic ($v$) compartment and leaky-integrator dynamics:

$\textstyle C_{\text{m,u}}\dot{u}$
$\textstyle=-g_{\text{l}}u+g_{\text{den}}\left(v-u\right)+\sum\nolimits_{j}w^{\text{som}}_{ij}r_{j}\left(E^{\text{rev}}-u\right)\;,$
$\textstyle C_{\text{m,v}}\dot{v}$
$\textstyle=-g_{\text{l}}v+\sum\nolimits_{j}w^{\text{den}}_{ij}r_{j}\;,$

where $C_{\text{m}}$ is the capacitance of the compartment, $g_{\text{l}}$ is
the leak conductance, $w$ the synaptic weights, $E^{\text{rev}}$ the reversal
potential and $g_{\text{den}}$ the coupling between the compartments. To
account for the electrophysiology of signal propagation, somatic compartments
receive weak conductance-based input (nudging), while the effect of dendritic
synapses is current-based. The output of a neuron is its firing rate
$r=\varphi(u)$, which is a non-linear function of the somatic potential $u$.

We model the interplay between an output and a latent population of neurons.
While our model is more general, in a motor context these populations would
respectively be found in motor and premotor areas of cortex. During
development, a sparse scaffold of somato-somatic connections is formed (Fig.
1a). These connections are static and induce random delays. Their sparsity and
uniformity is controlled by the affinity of pyramidal neurons to form new
output connections $p$ and to accept new input connections $q$. After
development, the network learns an output pattern by adapting somato-dendritic
synapses, which also induce transmission delays. A teacher nudges output
neurons towards a target pattern, and the somato-somatic scaffold transports
this nudging signal throughout the network. The somato-dendritc synapses use
these signals to adapt based a local error-correcting learning rule [6]

$\displaystyle\Delta
w^{\text{den}}_{ij}=\eta\,\left[\varphi(u_{i})-\varphi\left(\frac{g_{\text{L}}E_{\text{L}}+g_{\text{D}}v_{i}}{g_{\text{l}}+g_{\text{den}}}\right)\right]\bar{r}_{j}$

where $\bar{r}_{j}$ is the lowpass-filtered presynaptic firing rate. The error
used for updating the weights is locally stored in the postsynaptic neuron as
the difference between the somatic firing rate $\varphi(u)$ and the firing
rate that the soma would produce if it were only driven by the dendrite.
During the learning phase, the somato-dendritic connections form a new
scaffold that extracts increasingly useful information from the nudging
signals. This, in turn, gives rise to a robust dynamical attractor that learns
to generate the correct output pattern even without the external teacher.

Figure 2:  (a, b) Robustness of pattern replay. As in Fig. 1, external nudging
(teacher input) ends at the red line. A strong disturbance is introduced by
voltage-clamping the output layer such that it produces very low (a) and very
high (b) firing rates. After the disturbance, the network quickly recovers the
correct sequence. (c) Efficient allocation of latent resources. Smaller latent
populations require longer training times, but are still capable of producing
the desired output. The results are robust across different levels of scaffold
sparsity ($p\in[0.1,0.5]$) and uniformity ($q\in[0.05,0.25]$). (d) Robustness
to different levels of output feedback. The strength of this feedback is
determined by the output-to-latent somato-dendritic innervation ratio $r$
(only somato-dendritic synapses can learn large weights during training). The
ability of the network to learn this output feedback ($r>0$) strongly
influences its learning speed, but even in the complete absence of this
capability ($r=0$), the network can learn robust latent attractors that give
rise to accurate sequence replays.

## Results

We demonstrate our model’s ability to learn complex, non-Markovian sequences
by exposing it repeatedly to a sample of Beethoven’s “Für Elise” (Fig. 1b).
Immediately after development, the latent population is only weakly active.
During learning, dendritic connections evolve to shape the latent activity
into an increasingly structured and regular attractor, allowing the network to
replay the desired pattern independently of the teacher. Notably, our model is
superior to a reservoir of neurons of the same size, both with regard to its
ability to learn the pattern during training and to sustain it during replay
(Fig. 1c).

For a biologically plausible model, robustness is of utmost importance. A
model that only performs well in an idealized (disturbance-free) setting or
for finely tuned and tightly controlled network parameters is not realistic.
We therefore demonstrate our model’s resilience to multiple disturbances and
its functionality in large ranges of the network parameter space. Despite
severe temporary disruptions of the output population activity during pattern
replay, the network is able to recover a correct replay of the learned pattern
(Fig. 2a,b). Additionally, we demonstrate successful pattern learning across a
range of network sizes and somatic scaffold parameters $p$ and $q$ (Fig. 2c).
Finally, we demonstrate convergence to the target behavior even when the
network is stripped of the capability to learn direct feedback from its output
population, albeit at a slower pace (Fig. 2d).

## Conclusion

In this work we propose a framework that enables fully local learning of
sequences in recurrent networks of structured neurons. As a model of sequence
learning in cortex, our method shows many advantages: It is resource-
efficient, biologically plausible, and very robust. We demonstrate that our
approach, compared to reservoir computation, is capable of both higher
accuracy during training and sustained replay of the target pattern.
Additionally, our model is not reliant on optimal initial conditions and
retains its capability to learn and replay sequences across a wide range of
parametrizations. Finally, we show that the attractor dynamics imprinted into
our latent population via our local learning rule are able to withstand strong
external disturbances. Our model starts with only a sparse, random connection
scaffold generating weak and unstructured activity. We show that this is
enough for local plasticity to extract useful information in order to imprint
strong attractor dynamics. Thus, the combination of a developmental scaffold
together with dendritic computation undergirds a robust and biologically
plausible model of sequence learning in the brain.

## References

  * [1] Mantas Lukoševičius and Herbert Jaeger.  Reservoir computing approaches to recurrent neural network training.  Computer science review, 3(3):127–149, 2009.
  * [2] Paul J Werbos.  Backpropagation through time: what it does and how to do it.  Proceedings of the IEEE, 78(10):1550–1560, 1990.
  * [3] Owen Marschall, Kyunghyun Cho, and Cristina Savin.  A unified framework of online learning algorithms for training recurrent neural networks.  The Journal of Machine Learning Research, 21(1):5320–5353, 2020\.
  * [4] David Sussillo and Larry F Abbott.  Generating coherent patterns of activity from chaotic neural networks.  Neuron, 63(4):544–557, 2009.
  * [5] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass.  A solution to the learning dilemma for recurrent networks of spiking neurons.  Nature communications, 11(1):3625, 2020.
  * [6] Robert Urbanczik and Walter Senn.  Learning by the dendritic prediction of somatic spiking.  Neuron, 81(3):521–528, 2014.

## Acknowledgment

We gratefully acknowledge funding from the European Union for the Human Brain
Project under the grant agreement 945539, the Manfred Stärk Foundation and the
Swiss National Science Foundation (SNSF) for the “Prospective coding with
pyramidal neurons”-grant (310030L_156863) and the Sinergia Grant “Neural
Processing of Distinct Prediction Errors: Theory, Mechanisms & Interventions”
(CRSII5_180316). Our work has greatly benefitted from access to the Fenix
Infrastructure resources, which are partially funded from the European Union’s
Horizon 2020 research and innovation programme through the ICEI project under
the grant agreement No. 800858.

## Author contributions

KV, MAP and WS designed the model. KV wrote the initial code for the software
simulation which was extended and adapted by LK. The experiments were jointly
designed by LK, KV, BvH, FB, WS and MAP and conducted by LK. All authors
contributed to the writing of the manuscript.

## Code availability

The code for the simulations is available at https://github.com/unibe-cns/
temporal_sequence_learning.

## Competing Interests statement

The authors declare no competing interests.


# Regularize implicit neural representation by itself

Zhemin Li1 Hongxia Wang 1 Deyu Meng2,3
1 National University of Defense Technology 2 Xi’an Jiaotong University
3 Macau University of Science and Technology
lizhemin@nudt.edu.cn,wanghongxia<EMAIL_ADDRESS>This work
was supported by the National Key Research ,Development Program
(2020YFA0713504), the National Natural Science Foundation of China (61977065)
and the Macao Science and Technology Development Fund (061/2020/A2).

###### Abstract

This paper proposes a regularizer called Implicit Neural Representation
Regularizer (INRR) to improve the generalization ability of the Implicit
Neural Representation (INR). The INR is a fully connected network that can
represent signals with details not restricted by grid resolution. However, its
generalization ability could be improved, especially with non-uniformly
sampled data. The proposed INRR is based on learned Dirichlet Energy (DE) that
measures similarities between rows/columns of the matrix. The smoothness of
the Laplacian matrix is further integrated by parameterizing DE with a tiny
INR. INRR improves the generalization of INR in signal representation by
perfectly integrating the signal’s self-similarity with the smoothness of the
Laplacian matrix. Through well-designed numerical experiments, the paper also
reveals a series of properties derived from INRR, including momentum methods
like convergence trajectory and multi-scale similarity. Moreover, the proposed
method could improve the performance of other signal representation methods.

##  1 Introduction

Figure 1: Overview of proposed improve scheme for INR. (a) INR is a fully connected neural network which maps from coordinate to pixel value. (b) INRR is a regularization term represented by an INR which can capture the self-similarity. (c) INR-Z improve the performance of INR by combining the neighbor pixels with coordinate together as the input of another INR. |  |
---|---|---
(a) Sampling | (b) INR (18.1 dB) | (c) INRR (23.3 dB)
Figure 2: Image fitting results. All the methods are based on the SIREN to fit
an $256\times 256$ Baboon with the sampling data in (a). (b) trained with a
vanilla SIREN while (c) trained with proposed INRR.

INR uses a fully connected network (FCN)
$\phi_{\bm{\theta}}(\mathbf{x}):\mathbb{R}^{d}\mapsto\mathbb{R}^{o}$ to
approximate the explicit solution of an implicit function
$F\left(\mathbf{x},\phi_{\bm{\theta}},\nabla_{\mathbf{x}}\phi_{\bm{\theta}},\nabla_{\mathbf{x}}^{2}\phi_{\bm{\theta}},\ldots\right)=0$.
For an example, we can represent a gray-scale image
$\mathbf{X}\in\mathbb{R}^{m\times n}$ with an INR
$\phi_{\bm{\theta}}(\mathbf{x}):\mathbb{R}^{2}\mapsto\mathbb{R}$ which
satisfied
$\phi_{\bm{\theta}}(\frac{i}{m},\frac{j}{n})=\mathbf{X}_{ij},i\in\left\\{1,\ldots,m\right\\},j\in\left\\{1,\ldots,n\right\\}$.
Compared with traditional grid representation $\mathbf{X}$, INR’s
representation ability to details is not restricted by grid resolution $m,n$
as INR can predict the pixel value at any location $(x,y)\in\mathbb{R}^{2}$
even not equals to $(\frac{i}{m},\frac{j}{n})$.

Besides the representation ability of INR, generalization ability is critical
for a neural network. We explore the empirical generalization ability via a
$256\times 256$ gray-scale non-uniformly sampled image inpainting task as
Figure 2(a) shows. Although INR fits training data perfectly in Figure 2(b),
its prediction outside training data is unreasonable. Theoretical analysis of
INR illustrates that a hyper-parameter controls the smoothness degree of
$\phi_{\bm{\theta}}(\mathbf{x})$. Moreover, the experiments show that the best
hyper-parameter varies with the missing rate (the percentage of unsampled
pixels) as Figure 3 shows. Adjusting this hyper-parameter cannot make the non-
uniformly missing case perform best, as different locations might have
different missing rates.

A carefully designed regularizer is proposed to improve the generalization
ability of INR. It is based on Adaptive and Implicit Regularization (AIR)
which is a learned Dirichlet Energy (DE)[13] that measures similarities or
correlations between rows/columns of $\mathbf{X}$. The smoothness of the
Laplacian matrix is further integrated by parameterizing DE with a tiny INR.
The structure of the proposed implicit neural representation regularizer
(INRR) is shown in Figure 1(b). Because a smooth Laplacian matrix represents
non-local prior and large-scale local prior in vision data, INRR can improve
the generalization of INR in image representation. Numerous numerical
experiments show that INRR outperforms various classical regularizers,
including total variation (TV), $L_{2}$ energy, and so on. As a regularizer
both in a new form and with new meaning, INRR can be combined with other
signal representation methods, such as deep matrix factorization (DMF) [1].

To summarize, the contributions of our work include the following:

  * •

Neural Tangent Kernel (NTK) [1] theoretically analyzes the generalization
ability of INR and why INR performs poorly with nonuniform sampling is given.

  * •

A tiny INR parameterized regularizer named INRR is proposed based on DE, which
perfectly integrates the image’s self-similarity with the smoothness of the
Laplacian matrix.

  * •

A series of properties derived from INRR, including momentum methods, multi-
scale similarity, and generalization ability, are revealed by well-designed
numerical experiments.

##  2 Related Work

Implicit neural representation. Recently, INR has shown outstanding potential
in representing vision data, including font, images, and videos [24, 26]. It
has been applied in novel view synthesis [22, 19, 32, 11], signal compression
[35, 28, 5, 7, 23], and classification[6, 21].

In these latter years, a series of works have systematically studied and
advanced the representation capabilities of INR. Tancik et al. discuss why an
INR with ReLU activation function can not represent the high-frequency
components well and introduce a Fourier feature encode that significantly
improves the representation ability of INR [29]. Furthermore, Stizmann et al.
replace ReLU with a sinuous activation function and propose a specific
initialization scheme. The corresponding network is named sinusoidal
representation network (SIREN) [26]. Then Fathony et al. propose filter neural
networks with the Fourier and Gabor as basis activation [8]. Furthermore,
Band-limited Coordinate Networks (BACON) introduces the ability of multiscale
INR representation [15]. Apart from fitting the training set, the
generalization ability of INR is more critical in many applications.

Regularization. Improving the generalization of NN with regularization
techniques such as $L_{1}$-norm, $L_{2}$-norm, and the Dropout technique has a
long history [27]. These regularizations take the images or other signals as
input. Recently, there has been a class of NN that use a whole NN to represent
a signal, such as Deep Image Prior (DIP), Deep Matrix Factorization (DMF), and
INR [30, 1, 26]. In this case, the classical signal regularization technique
can be applied to the signal represented NN [16, 20, 13, 14]. Significantly,
the learnable regularizer is better than those not learned [20, 13]. To our
knowledge, no effort has been made to regularize INR using a learnable
regularizer based on the characteristics of INR’s data representation.

##  3 Theoretical analysis of INR

Figure 3: Fitting a $256\times 256$ Cameraman which random missing $50\%$
pixels with (a) SIREN and (b) NTK, respectively. $\omega_{0}$ and $\delta$ are
the hyper-parameters of models. (c,d) shows the PSNR value change with
$\omega_{0},\delta$ at different random missing rate, respectively.

As Figure 2(b) shows, INR’s generalization ability is not as well as its
representation ability. We analyze INR theoretically with a proxy model NTK to
answer when and why INR generalizes badly.

Implicit neural representation. INR uses a FCN
$\phi_{\bm{\theta}}(\mathbf{x}):\mathbb{R}^{d}\mapsto\mathbb{R}^{o}$ to
approximate the explicit representation of an implicit function
$F\left(\mathbf{x},\phi_{\bm{\theta}},\nabla_{\mathbf{x}}\phi_{\bm{\theta}},\nabla_{\mathbf{x}}^{2}\phi_{\bm{\theta}},\ldots\right)=0$,
where the FCN has $L$ hidden layers defined as follows,

$\displaystyle\mathbf{z}^{(\ell)}=\mathbf{W}^{(\ell)}\mathbf{x}^{(\ell-1)}+\mathbf{b}^{(\ell)}$
$\displaystyle,\quad\mathbf{x}^{(\ell)}=\sigma\left(\mathbf{z}^{(\ell)}\right),$
(1) $\displaystyle\phi_{\bm{\theta}}(\mathbf{x})=\mathbf{z}^{(L+1)}$
$\displaystyle,\quad\ell=1,2,\ldots,L+1,$

with $\sigma(\cdot)$ an element-wise activation function and
$\mathbf{x}^{(0)}=\mathbf{x}$,
$\bm{\theta}=\left\\{\mathbf{W}^{(\ell)},\mathbf{b}^{(\ell)}\mid\ell=1,2,\ldots,L+1\right\\}\sim\mathcal{D}$
at initialization, $\mathbf{W}^{(\ell)}\in\mathbb{R}^{n_{\ell}\times
n_{\ell-1}}$, and $n_{\ell}$ is the width of $\ell$-th layer with $n_{0}=d$,
$n_{L+1}=o$.

For simplicity, we focus on a special case of INR with
$F(\bm{\theta},\mathbf{x},z)=\left\|\phi_{\bm{\theta}}(\mathbf{x})-z(\mathbf{x})\right\|_{2}^{2}=0$
and $\phi_{\bm{\theta}}(\mathbf{x}):\mathbb{R}^{2}\mapsto\mathbb{R}$, where
$z(\mathbf{x}):\mathbb{R}^{2}\mapsto\mathbb{R}$ is the gray-scale image we
want to represent. All the results in this paper can be easily extended to a
higher dimension. The vanilla INR is formulated as

$\bm{\theta}^{*}=\arg\min_{\bm{\theta}}\left\\{\mathcal{L}(\bm{\theta},\mathcal{X},\mathcal{Z})=\sum_{(\mathbf{x}_{i},z_{i})\in\mathcal{X}\times\mathcal{Z}}F\left(\bm{\theta},\mathbf{x}_{i},z_{i}\right)\right\\},$
(2)

where
$\mathcal{X}\times\mathcal{Z}=\left\\{(\mathbf{x}_{i},z_{i})\right\\}_{i=1}^{N}$
is the training set, and $\mathbf{x}_{i}=(x_{i},y_{i})$ is the coordinate. The
training set is sampled from the grid of matrix
$\mathbf{X}\in\mathbb{R}^{m\times n}$. For example, we can use
$(\frac{i}{m},\frac{j}{n})$ as input and $\mathbf{X}_{ij}$ as the
corresponding output of INR. After training, $z(\mathbf{x})$ is predicted by
$\phi_{\bm{\theta}}(\mathbf{x})$ at any location $\mathbf{x}=(x,y)$ even when
$\mathbf{x}\not\in\mathcal{G}=\left\\{(\frac{i}{m},\frac{j}{n})\mid
i\in\left\\{1,\ldots,m\right\\},j\in\left\\{1,\ldots,n\right\\}\right\\}$.

Kernel regression approximate neural networks. Jacot et al. show that with
infinity width of the layers in $\phi_{\bm{\theta}}$ and small learning rate,
the function $\phi_{\bm{\theta}}$ converges to the kernel regression

$\phi_{\text{NTK}}(\mathbf{x})=\sum_{i=1}^{N}\left(\mathbf{K}^{-1}\mathbf{z}\right)_{i}k_{\text{NTK}}\left(\mathbf{x}_{i},\mathbf{x}\right),$

where $\mathbf{K}$ is an $N\times N$ kernel matrix dubbed neural tangent
kernel (NTK) [10, 9, 34] with entries defined as

$\mathbf{K}_{ij}=k_{\text{NTK}}(\mathbf{x}_{i},\mathbf{x}_{j})=\mathbb{E}_{\bm{\theta}\sim\mathcal{D}}\left<\frac{\partial\phi_{\bm{\theta}}(\mathbf{x}_{i})}{\partial\bm{\theta}},\frac{\partial\phi_{\bm{\theta}}(\mathbf{x}_{j})}{\partial\bm{\theta}}\right>.$

In this paper, we consider INR
$\phi^{\prime}:\mathbb{R}^{2D}\mapsto\mathbb{R}$ with a feature map
$\gamma(\mathbf{x})=\frac{1}{\sqrt{D}}[\cos\mathbf{Bx}^{\top},\sin\mathbf{Bx}^{\top}]^{\top}:\mathbb{R}^{d}\mapsto\mathbb{R}^{2D}$
as its input, where $\mathbf{x}\in\mathbb{R}^{d}$,
$\mathbf{B}\in\mathbb{R}^{D\times d}$, and
$\mathbf{B}_{ij}\sim\mathcal{N}(0,\delta)$. Then
$\phi_{\text{NTK}}^{\prime}(\gamma(\mathbf{x}))$ is shift-invariant thus more
suitable for image representation.

Now we analyze how INR predicts the data outside of the training set. Theorem
1 illustrates that the smoothness of represented signal is controlled by the
hyper-parameter $\delta$ globally. Especially when $\delta$ tends to infinity,
the prediction of $\phi_{\text{NTK}}^{\prime}(\cdot)$ outside the training set
all tends to the same weighted average of the training set according to
Corollary 1.

###### Theorem 1

Given a FCN
$\phi_{\bm{\theta}}^{\prime}(\cdot):\mathbb{R}^{2D}\mapsto\mathbb{R}$ with
$\mathbf{x}\in\mathbb{R}^{d}$, $\gamma(\mathbf{x})\in\mathbb{R}^{2D}$, and the
feature map
$\gamma(\mathbf{x})=\frac{1}{\sqrt{D}}[\cos\mathbf{Bx}^{\top},\sin\mathbf{Bx}^{\top}]^{\top}$
with $\mathbf{B}\in\mathbb{R}^{D\times d}$ and
$\mathbf{B}_{ij}\sim\mathcal{N}(0,\delta)$. Denote the corresponding composed
NTK as
$k_{D}(\mathbf{x}_{i},\mathbf{x}_{j})=h_{\text{NTK}}\left(\frac{1}{D}\mathbf{1}_{D}^{\top}\cos\left(\mathbf{B}(\mathbf{x}_{i}-\mathbf{x}_{j})\right)\right)$,
then we have

$\lim_{D\rightarrow\infty}k_{D}(\mathbf{x}_{j},\mathbf{x}_{j})=h_{\text{NTK}}(e^{-\delta^{2}\left\|\mathbf{x}_{i}-\mathbf{x}_{j}\right\|^{2}}).$

###### Corollary 1

Assume the $h_{\text{NTK}}$ in Theorem 1 satisfies $h_{\text{NTK}}(1)\neq
h_{\text{NTK}}(0)$ and $h_{\text{NTK}}(1)\neq 0$, then

$\lim_{\delta\rightarrow\infty}\phi_{\text{NTK}}^{\prime}(\gamma(\mathbf{x}))=\left\\{\begin{array}[]{cc}z_{l},\qquad\quad\mathbf{x}=\mathbf{x}_{l}\in\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N},\\\
\frac{h(0)\mathbf{1}_{N}^{\top}\mathbf{z}}{(N-1)h(0)+h(1)},\mathbf{x}\not\in\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}.\\\
\end{array}\right.$

INR needs to be regularized. We validate Corollary 1 by exploring the
performance of $\phi_{\text{NTK}}^{\prime}(\gamma(\mathbf{x}))$ in image
inpainting task with different missing rates. As Figure 3(b) shows, when
$\delta=900$, $\phi_{\text{NTK}}^{\prime}(\gamma(\mathbf{x}))$ at the location
of outside of sampled data has the same value. Furthermore, Figure 3(a) shows
that the latest SIREN [26], which represents signals without a feature map of
input, is also controlled by the hyper-parameter $\omega_{0}$ in the first
layer as $\sin\left(\omega_{0}\mathbf{W}\mathbf{x}+\mathbf{b}\right)$.

Based on the numerical result, the optimal $\omega_{0}$ or $\delta$ is
required so that INR generalizes the best. However, finding an optimal
$\omega_{0}$ or $\delta$ with non-uniformly sampled training data is
impossible. Figure 3(c,d) illustrates that the optimal $\omega_{0}$ or
$\delta$ varies considerably according to the missing rate. It decreases with
the increase of missing rate, which is consistent with the theoretical results
that the sparser sampling needs a smoother fitting. As to the case with
nonuniform missing, note that different locations might have different missing
rate; it is tough to make INR performs well by choosing an optimal hyper-
parameter.

Furthermore, the results above all based on the loss function
$\mathcal{L}=\sum_{(\mathbf{x}_{i},z_{i})\in\mathcal{X}\times\mathcal{Z}}\left\|\phi_{\bm{\theta}}(\mathbf{x}_{i})-z_{i}\right\|_{2}^{2}$
which is a fidelity term measured on the training data. Enforcing additional
constraints on the predicted data is profitable to improve the generalization
ability of INR. In the next section, we add constraints by a newly proposed
regularizer named INRR.

##  4 Methods to regularize INR

This section presents a regularized model
$\mathcal{L}(\bm{\theta},\mathcal{X},\mathcal{Z})+\lambda\mathcal{R}(\bm{\theta},\mathcal{X},\mathcal{Z})$,
where $\lambda$ is a parameter that balances the loss of training data and the
regularizer $\mathcal{R}$.

Now consider the priors of images on a larger scale. Since the vanilla INR’s
loss function is pixel-by-pixel, it ignores the structural features of images.
Specifically, these features include the relationship between rows, columns,
or blocks. Low rank is a well-known prior that describes the correlation
between rows and columns. However, a low-rank matrix cannot express the
details of a signal well because these details are located in the subspaces
corresponding to the small singular value of the image.

So we turn to self-similarity, which is quite common in large and fine scales
of an image. As a simple example, smooth $\mathbf{X}$ implies local similarity
between adjacent rows and columns of $\mathbf{X}$. Furthermore, the non-local
self-similarity of an image, which refers to the similarity between non-
adjacent rows, columns, or blocks, is also very universal and valuable. In
this paper, we choose Dirichlet Energy (DE) to describe images’ local and non-
local self-similarity. Our method is not restricted to DE.

###  4.1 Dirichlet Energy

Given a matrix $\mathbf{X}\in\mathbb{R}^{m\times n}$, DE is formulated as
follows

$\mathcal{R}_{\text{DE}}=tr\left(\mathbf{X}^{\top}\mathbf{L}\mathbf{X}\right)=\sum_{1\leq
i,j\leq
m}\mathbf{A}_{ij}\left\|\mathbf{X}_{i,:}-\mathbf{X}_{j,:}\right\|^{2},$

where $\mathbf{A}\in\mathbb{R}^{m\times m}$ is a weighted adjacency matrix
along rows of $\mathbf{X}$, and $\mathbf{L}=\mathbf{D}-\mathbf{A}$ with
$\mathbf{D}_{ii}=\sum_{j=1}^{m}\mathbf{A}_{ij}$ and $\mathbf{D}_{ij}=0$ if
$i\neq j$. As $\mathbf{A}_{ij}$ measures the similarity of rows
$\mathbf{X}_{i,:}$ and $\mathbf{X}_{j,:}$, DE is a non-local self-similarity
measure of $\mathbf{X}$.

However, there are two main issues in using DE: (a) $\mathbf{L}$ or
$\mathbf{A}$ is unknown under the incomplete sampling of $\mathbf{X}$; (b) DE
only encodes the similarity between two rows, other large-scale similarities
such as block similarity cannot be captured. To solve these problems, we
parameterize $\mathbf{L}$ with another tiny INR and learn it during training
$\phi_{\bm{\theta}}(\mathbf{x})$.

###  4.2 INRR

Learning $\mathbf{L}$ during training is naive thinking when $\mathbf{L}$ is
unknown. Nevertheless, we need to sufficiently extract the properties of
$\mathbf{L}$ to make it meaningful and practical. There are two mathematical
properties that $\mathbf{L}$ needs to satisfy: (a) positive semi-definite, (b)
the sum of each row equals zero. Specially, we find the $\mathbf{L}$ of
natural images has some extra priors. The natural images are usually piecewise
smooth, so $\mathbf{L}$, which measures the similarity of the rows of
$\mathbf{X}$, should also be nearly smooth.

Therefore, we propose an implicit neural representation regularization (INRR)
which is expressed as follows:

$\left\\{\begin{array}[]{c}\begin{aligned}
\mathcal{R}(\bm{\theta})&=tr\left(\left[\mathcal{T}(\mathbf{X})\right]^{\top}\mathbf{L}(\bm{\theta})\mathcal{T}(\mathbf{X})\right)\\\
\mathbf{L}(\bm{\theta})&=\mathbf{A}(\bm{\theta})\cdot\mathbf{1}_{m^{\prime}\times
m^{\prime}}\odot\mathbf{I}_{m^{\prime}}-\mathbf{A}(\bm{\theta})\\\
\mathbf{A}(\bm{\theta})&=\frac{\text{exp}\left(g^{\top}(\bm{\theta};\mathbf{u})g(\bm{\theta};\mathbf{u})\right)}{\mathbf{1}_{m^{\prime}}^{\top}\text{exp}\left(g^{\top}(\bm{\theta};\mathbf{u})g(\bm{\theta};\mathbf{u})\right)\mathbf{1}_{m^{\prime}}}\end{aligned}\end{array}\right.,$

where $\mathcal{T}(\cdot):\mathbb{R}^{m\times
n}\mapsto\mathbb{R}^{m^{\prime}\times n^{\prime}}$ aims to capture self-
similarity in $\mathbf{X}$, $\mathbf{L}(\bm{\theta})$ measures the similarity
between rows of $\mathcal{T}(\mathbf{X})$.
$g(\bm{\theta};\cdot):\mathbb{R}\mapsto\mathbb{R}^{r}$ is a tiny INR,
$g(\bm{\theta};\mathbf{u})\in\mathbb{R}^{r\times m^{\prime}}$,
$g(\bm{\theta};\mathbf{u})_{i}=g(\bm{\theta};\mathbf{u}_{i}),i=1,2,\ldots,m^{\prime}$.
And $\mathbf{u}_{i}$ is coordinate of sampled matrix $\mathcal{T}(\mathbf{X})$
with
$\mathcal{T}(\mathbf{X})_{ij}=\phi_{\bm{\theta}}\left(\mathbf{u}_{i},\mathbf{v}_{j}\right)$
and
$\mathbf{u}=\left[\frac{1}{m^{\prime}},\frac{2}{m^{\prime}},\ldots,\frac{m^{\prime}}{m^{\prime}}\right]^{\top}$,
$\mathbf{v}=\left[\frac{1}{n^{\prime}},\frac{2}{n^{\prime}},\ldots,\frac{n^{\prime}}{n^{\prime}}\right]^{\top}$.
It is not difficult to verify that the parameterized Laplacian matrix keeps
properties (a) and (b). Furthermore, $g(\bm{\theta};\cdot)$ introduces the
smoothness of $\mathbf{L}$ implicitly, and $r$ restricts the rank of
$\mathbf{L}$.

Take the relations between columns into account simultaneously. The whole
regularized model is formulated as

$\textrm{minimize}_{\bm{\theta},\bm{\theta}_{r},\bm{\theta}_{c}}\left\\{\mathcal{L}(\bm{\theta},\mathcal{X},\mathcal{Z})+\lambda_{r}\mathcal{R}(\bm{\theta}_{r})+\lambda_{c}\mathcal{R}(\bm{\theta}_{c})\right\\},$

where $\mathcal{R}(\bm{\theta}_{r})$ and $\mathcal{R}(\bm{\theta}_{c})$ are
row and column regularizers respectively. $\mathcal{T}(\mathbf{X})=\mathbf{X}$
in $\mathcal{R}(\bm{\theta}_{r})$, and
$\mathcal{T}(\mathbf{X})=\mathbf{X}^{\top}$ in $\mathcal{R}(\bm{\theta}_{c})$.
$\lambda_{r},\lambda_{c}$ are used to balance the fidelity and regularization
terms.

As the self-similarity which is represented by $g(\bm{\theta}_{r};\cdot)$ or
$g(\bm{\theta}_{c};\cdot)$ are much simpler than the image, so the parameter
number of $g(\bm{\theta}_{r};\cdot)$ or $g(\bm{\theta}_{c};\cdot)$ are much
lesser than the one of $\phi_{\bm{\theta}}(\cdot)$, which called tiny INR.

##  5 Experiments

###  5.1 Experimental setting

Table 1: PSNR (dB) of recovered images by INR based models with different missing patters include random missing, patch missing, and textural missing. Four images are tested including Baboon, Barbara, Man and Boats. |  | INR | INR-Z | TV | $L_{2}$ | AIR | INRR |  | INR | INR-Z | TV | $L_{2}$ | AIR | INRR
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Random | Baboon | 21.0 | 21.5 | 23.8 | 23.1 | 24.2 | 24.7 | Barbara | 28.6 | 28.8 | 30.2 | 30.5 | 30.3 | 30.7
Patch | 25.3 | 27.3 | 24.5 | 26.3 | 31.6 | 33.8 | 26.1 | 27.1 | 28.8 | 28.5 | 28.9 | 29.5
Textural | 20.8 | 23.5 | 21.4 | 23.9 | 27.2 | 28.3 | 26.5 | 27.9 | 26.5 | 27.9 | 29.0 | 29.4
Random | Man | 23.5 | 22.8 | 25.6 | 25.6 | 25.7 | 25.9 | Boats | 28.1 | 27.6 | 29.5 | 29.7 | 29.4 | 29.9
Patch | 25.6 | 26.5 | 22.7 | 26.9 | 31.3 | 32.1 | 27.6 | 28.5 | 27.8 | 29.5 | 33.4 | 34.4
Textural | 22.6 | 22.6 | 24.2 | 24.9 | 25.3 | 26.4 | 24.1 | 27.0 | 25.2 | 28.1 | 27.9 | 28.9
Figure 4: Residual of image inpainting, i.e., $\left|\mathbf{X}-\mathbf{X}^{*}\right|$ with three types of missing data by different regularized INR including (b) INR without regularization, (c) with TV, (d) $L_{2}$, (e) AIR, and (f) INRR. The hyper-parameters of benchmark models and algorithms are adopted from the original paper. Table 2: PSNR of recovered images by five data representation models without regularization and with INRR under mixture missing. The hyper-parameters of benchmark models and algorithms are adopted from the original paper. The value in parentheses represents the increment of the PSNR value after using the INRR. | DMF | DMF+ | ReLU | ReLU+ | Siren | Siren+ | Garbor | Garbor+ | Fourier | Fourier+
---|---|---|---|---|---|---|---|---|---|---
Baboon | 8.2 | 22.1 (13.9) | 20.9 | 21.4 (0.6) | 17.6 | 22.7 (5.1) | 17.5 | 22.8 (5.4) | 14.3 | 22.8 (8.5)
Man | 8.4 | 21.7 (13.3) | 22.2 | 22.5 (0.3) | 18.9 | 22.8 (3.8) | 17.9 | 22.0 (4.1) | 15.3 | 22.1 (6.9)
Barbara | 8.8 | 25.0 (16.3) | 25.2 | 26.4 (1.2) | 23.3 | 25.6 (2.3) | 19.4 | 25.0 (5.5) | 17.4 | 25.0 (7.7)
Boats | 10.9 | 24.4 (13.4) | 24.0 | 25.2 (1.2) | 22.2 | 25.7 (3.4) | 17.6 | 24.6 (7.0) | 14.9 | 24.6 (9.7)
Cameraman | 6.9 | 24.5 (17.6) | 25.7 | 25.8 (0.1) | 23.9 | 25.7 (1.8) | 18.2 | 24.7 (6.5) | 15.2 | 24.7 (9.4)

Data types and missing patterns. We consider five gray-scale benchmark images
of size $m\times n=256\times 256$, including Baboon, Man, Barbara, Boats, and
Cameraman. Moreover, we study matrix completion with three different missing
patterns: random missing, patch missing, and textural missing, which is shown
as different parts in Figure 1(a). The default missing rate is $50\%$.

Network settings. In this section, the INR defaults to SIREN when not
otherwise specified [26]. The INR network is organized in five hidden layers
SIREN whose widths are all the same as 256. As to INRR, five hidden layers
SIREN is chosen with the same width 32, and the output dimension
$r=\max(m,n)$. We use Adam with default settings in [12] to train all the
networks.

Peered methods The peered methods include

  1. 1.

TV:
$\mathcal{R}_{\text{TV}}=\sum_{\mathbf{x}_{i}=(x_{i},y_{i})\in\mathcal{G}}\left\|\nabla_{\mathbf{x}}\phi_{\bm{\theta}}(\mathbf{x}_{i})\right\|_{1}$,
$\mathcal{G}=\left\\{\frac{1}{m},\ldots,\frac{m}{m}\right\\}\times\left\\{\frac{1}{n},\ldots,\frac{n}{n}\right\\}$,
which is the discrete version on $\mathcal{G}$.

  2. 2.

$L_{2}$:
$\mathcal{R}_{L_{2}}=\sum_{\ell=1}^{L+1}\left\|\mathbf{W}^{(\ell)}\right\|_{2}$
which is a common regularizer which is used to regularize NN.

  3. 3.

INR-Z: Combining the neighbor of the input with coordinate
$[x,y,f(N(x,y))]\in\mathbb{R}^{(N_{0}+2)\times 1}$ as the input of a new INR
$h(\cdot):\mathbb{R}^{N_{0}+2}\mapsto\mathbb{R}$ as Figure 1(c) shows.

  4. 4.

AIR: Adaptive and implicit regularization [13]

  5. 5.

INRR: Implicit neural representation regularization proposed in this paper.

###  5.2 Image representation with various missing patterns

We apply INRR for matrix completion (or image inpainting) on three types of
missing patterns. A few related models are also used for comparison.

Adaptive to training data. We compare vanilla INRR with several improved
models in the following experiments, including TV, $L_{2}$, AIR, INRR, and
INR-Z. Table 1 lists the PSNRs of recovered images using the aforementioned
improved models for different data with different missing patterns. The
results show that the non-local regularization methods, including AIR and
INRR, significantly outperform the vanilla INR. Furthermore, INRR is much
better than AIR since INRR integrates the smoothness of Laplacian matrix into
the DE regularizer. The residual of recovered images
$\left|\mathbf{X}-\mathbf{X}^{*}\right|$ corresponding to Table 1 are shown in
Figure 4. Unlike other INR-regularized methods that perform well for random
missing cases but poorly for other missing patterns, INRR consistently gives
visually appealing results. To conclude, INRR achieves decent results
qualitatively and quantitatively independent of sampling mode of training
data.

Adaptive to data representation. To distinguish the effect of INRR regularizer
from the vanilla INR model, Table 2 lists the PSNRs of recovered images by
several data representations which INRR regularizes. The data representation
includes deep matrix factorization (DMF) [1], FCN with ReLU activation
function, SIREN [26], the filter neural network with Gabor and Fourier filter,
respectively [8]. The INRR regularized models are denoted by ’.+’ in Table 2.
The mixture missing pattern is shown in Figure 2(a). The results shown in
Table 2 and Figure 2(b)(c) both illustrate that INRR significantly improves
the performance of recently proposed data representation methods without
regularization. Overall, INRR is a general regularizer not limited to being
combined with a particular data representation model.

##  6 Why INRR performs better

Now we have shown that INRR achieves excellent performance in image
representation (image inpainting as an example) under different missing
patterns. In this section, the reasons why INRR performs better than other
peered methods are analyzed carefully. Firstly, the smoothness of $\mathbf{L}$
learned by INRR is demonstrated by experiments. Then a heuristic connection
between INRR, implicit bias, and the momentum method is built.

###  6.1 Tiny INR smooths Laplacian matrix implicitly

|
---|---
(a) AIR | (b) INRR
Figure 5: $\mathbf{L}$ at $t=2000$ learned by (a) AIR and (b) INRR,
respectively. All the methods are based on the SIREN to super-resolution
Baboon, which is down-sampled from $256\times 256$ to $128\times 128$.

Parameterizing DE with a tiny INR is the key of INRR. In this section, we
focus on illustrating the benefit of this parameterization. A $256\times 256$
Baboon is down-sampled to $128\times 128$, and then AIR and INRR are used to
regularize INR to recover the original image based on the sampled data. Figure
5 shows the Laplacian matrix $\mathbf{L}$ learned by AIR and INRR,
respectively. The $\mathbf{L}$ learned by AIR (Figure 5(a)) is discontinuous
with high probability at those locations that are not sampled, while the
$\mathbf{L}$ learned by INRR (Figure 5(b)) is much more continuous. The
continuous $\mathbf{L}$ introduced by the tiny INR is more consistent with
practice.

###  6.2 INRR behaves like a momentum

|  |
---|---|---
(a) Random | (b) Textural | (c) Patch
Figure 6: Trajectories of training INR model without regularization, with TV,
$L_{2}$, AIR and INRR regularization for inpainting the Baboon image with
three types of missing types of missing pixels: (a) randomly missing $50\%$,
(b) textural missing, and (c) patch missing. The dot point indicates
$\mathbf{X}(0)$ and the diamond shape of different color indicate the MSE on
training data achieve $10^{-3}$.

We connect INRR with the momentum method in this subsection. As Figure 8
shows, INRR tends to vanish during training. Then INR with INRR converges to
the vanilla INR model. First, compare INRs with and without INRR by the
optimization trajectory. In Figure 6, we plot the MSE’s trajectory during
training. At the beginning of training, the observed and unobserved MSEs of
the five models drop similarly. However, these five models perform
dramatically differently near the convergence. When the observed MSE becomes
smaller, the model learns details in observed elements. The unobserved MSE
increased during the observed MSE decrease in the vanilla INR, INR+TV, and
INR+$L_{2}$ cases; we name this phenomenon over-fitting. INRR and AIR keep the
decaying trend for both observed and unobserved MSEs. Significantly, the
proposed INRR keeps the decaying trend better than AIR due to the extra
smoothness introduced by a tiny INR.

Looking back into the training process of INRR, the update of
$\mathbf{X}(t+1)$ involves both $\mathbf{X}(t)$ and $\mathbf{L}(t)$, and the
update of $\mathbf{L}(t)$ depends on $\mathbf{X}(t-1)$. To understand the
training dynamics, we consider the following simplified model:

$\textrm{minimize}_{\mathbf{X},\mathbf{L}}\left\\{\mathcal{L}+\lambda
tr(\mathbf{X}^{\top}\mathbf{L}\mathbf{X})\right\\},$

and we have

$\nabla_{\mathbf{X}(t)}\mathbf{L}=\nabla_{\mathbf{X}(t)}\mathcal{L}+2\lambda\mathbf{L}(t)\mathbf{X}(t),$

where $\mathcal{L}$ is the fidelity term, $\mathbf{L}(t)$ is the function of
$\left\\{\mathbf{X}(t_{0})\mid t_{0}<t\right\\}$ as $\mathbf{L}(t)$ is updated
based on $\mathbf{X}(t-1)$. Therefore, every iteration step of INRR leverages
all the previously learned information $\left\\{\mathbf{X}(t_{0})\mid
t_{0}<t\right\\}$. Note that the update of both vanilla INR, INR+TV, and
INR+$L_{2}$ only depend on $\mathbf{X}(t)$. From this viewpoint, INRR shares a
similar spirit as the momentum method, which leverages history to improve
performance.

###  6.3 INRR connects implicit bias with multi-scale self-similarity

Figure 7: Fitting a $256\times 256$ synthetic data which randomly missing
$50\%$ pixels at different training step with (a) single matrix (DMF with only
one factor), (b) DMF with three factors, (c) ReLU neural network, and (d)
SIREN. The effective rank of the fitted matrix is shown in (e). Figure 8:
Learned $\mathbf{L}_{r}(t)$ and $\mathbf{L}_{c}(t)$ during training. (a):
first and second rows depict the Baboon image and its rotation. (b)-(d):
first/second row shows the heatmap of $\mathbf{L}_{r}/\mathbf{L}_{c}$ at
different $t$. A darker color indicates a stronger similarity captured by the
adaptive regularizer. The $(i,j)$-th element in the heatmap of
$\mathbf{L}_{r}(t)$ has a darker color than the $i,j^{\prime}$-th element
indicates that the $i$-th row is more related to the $j$-th row compared with
the $j^{\prime}$-th row.
---
(a) Random
(b) Patch
(c) Textural
Figure 9: Contrasting the adaptive regularizer with the fixed regularizer for
Baboon image recovery. We consider inpainting the Baboon image with three
types of missing pixels: (a) randomly missing $50\%$ pixels, (b) patch
missing, and (c) textural missing. The red lines plot the PSNR during the
training of the vanilla INRR. The remaining three lines in each figure
indicate replacing $\mathbf{L}_{r}$ and $\mathbf{L}_{c}$ with
$\mathbf{L}_{r}(t)$ and $\mathbf{L}_{c}(t)$ at $t=3000,6000$ and $10000$,
respectively.

Implicit bias of NN. We then demonstrate other properties of INRR by
connecting implicit bias with multi-scale self-similarity. The implicit bias
of NN is used to explain the generalization ability of NN in recent years [33,
17, 3, 2, 1, 31, 36]. As Figure 7 shows, we fit synthetic data with DMF with
one factor, DMF with three factors, ReLU FCN, and SIREN, respectively. The
synthetic data is sampled from function
$s(x,y)=\sin\left(25\pi\sin\left(\frac{\pi}{3}\cdot\sqrt{x^{2}+y^{2}}\right)\right)$,
where $\\{(x_{i},y_{j})|i,j\\}$ is a uniform $256\times 256$ grid on
$[-1,1]\times[-1,1]$, where the local frequency of the synthetic data
increases from boundary to center. All the networks except DMF with one factor
evolve from a low complexity pattern to a high complexity one. ReLU FCN and
SIREN first fit the low-frequency components and then gradually fit the high-
frequency components [33, 17, 3, 2]. More specifically, the effective rank of
DMF with three factors, SIREN, and ReLU FCN, increases gradually as the line
plot, where the effective rank can measure the effective dimension of the
matrix with more accuracy than discrete rank [25, 1].

Multi-scale similarity captured by INRR. Then we turn to explain the multi-
scale similarity seized by INRR. Due to the implicit bias of fidelity term,
INRR can capture different scales of data similarity. The heatmaps of
Laplacian matrices $\mathbf{L}_{r}(t)$ and $\mathbf{L}_{c}(t)$ for Baboon are
shown in Figure 8. A few large blocks appear in $\mathbf{L}_{r}(500)$ and
$\mathbf{L}_{c}(500)$ Figure 8(b), which reflect the similarity in a large
scale. Then the size of blocks becomes smaller while the number of blocks
increases at $t=2500$ in Figure 8(c), which reflects the similarity on a
smaller scale. The values in these blocks reflect the substantial similarity
of the corresponding highlighted patches of the original Baboon, which echoes
our intuition. Moreover, as the training goes further, both
$\mathbf{L}_{r}(t)$ and $\mathbf{L}_{c}(t)$ focus on reflecting the similarity
of the neighbor at $t=20000$ in Figure 8(d), which is similar to the TV.

The importance of learned INRR. The results confirm that INRR captures the
similarity from large to small. Next, we experimentally illustrate that the
learned $\mathbf{L}_{c}$ and $\mathbf{L}_{c}$ by INRR are crucial for image
representation. Fix $\mathbf{L}_{r}$ and $\mathbf{L}_{c}$ at a specific
training step for INRR, and then compare INRR with the overall adaptive
$\mathbf{L}_{r}$ and $\mathbf{L}_{c}$.

We contrast the vanilla INRR and INRR with fixed Laplacian matrices (let
$t=3000,6000$ and $10000$ respectively) for Baboon image inpainting. Figure 9
shows how the PSNR changes during training. INRR, which continuously updates
the regularization during training, performs best for all missing patterns.
Fixing Laplacian matrices helps reduce the computation costs during training.
However, as the optimal $t^{*}$ is varied with missing patterns, the learned
Laplacian matrices are more applicable.

##  7 Conclusion

This paper proposes a novel regularizer named INRR, which significantly
improves INR’s representation performance, especially when the training data
is sampled arbitrarily. INRR parameterizes the Laplacian matrix in DE by a
tiny INR and then adaptively learns the non-local similarities hidden in image
data. INRR is a generic framework for integrating multiple prior into a single
regularizer, decreasing the redundancy of the regularizer. The connection
among INRR, momentum term, implicit bias, and multi-scale self-similarity
deserve further theoretical analysis.

## References

  * [1] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo.  Implicit regularization in deep matrix factorization.  NeurIPS, 2019.
  * [2] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan.  Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks.  Nature Communications, 2021.
  * [3] Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu.  Towards understanding the spectral bias of deep learning.  In IJCAI, 2021.
  * [4] Bin Dong, Haocheng Ju, Yiping Lu, and Zuoqiang Shi.  Cure: Curvature regularization for missing data recovery.  Siam Journal on Imaging Sciences, 2019.
  * [5] Emilien Dupont, Adam Goli’nski, Milad Alizadeh, Yee Whye Teh, and A. Doucet.  Coin: Compression with implicit neural representations.  ArXiv, abs/2103.03123, 2021.
  * [6] Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, and Dan Rosenbaum.  From data to functa: Your data point is a function and you should treat it like one.  ICML, 2022.
  * [7] Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goli’nski, Yee Whye Teh, and A. Doucet.  Coin++: Data agnostic neural compression.  ArXiv, abs/2201.12904, 2022.
  * [8] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J. Zico Kolter.  Multiplicative filter networks.  ICLR, 2021.
  * [9] Eugene Golikov, Eduard Pokonechnyy, and Vladimir Korviakov.  Neural tangent kernel: A survey.  ArXiv, abs/2208.13614, 2022.
  * [10] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  NeurIPS, 2018.
  * [11] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  ICCV, 2021.
  * [12] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  CoRR, abs/1412.6980, 2015.
  * [13] Zhemin Li, Tao Sun, Hongxia Wang, and Bao Wang.  Adaptive and implicit regularization for matrix completion.  SIAM Journal on Imaging Sciences, 15(4):2000–2022, 2022.
  * [14] Zhemin Li, Zhi-Qin John Xu, Tao Luo, and Hongxia Wang.  A regularized deep matrix factorized model of matrix completion for image restoration.  IET Image Process., 16:3212–3224, 2022.
  * [15] David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein.  Bacon: Band-limited coordinate networks for multiscale scene representation.  CVPR, 2022.
  * [16] Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S. Kamilov.  Image restoration using total variation regularized deep image prior.  ICASSP, pages 7715–7719, 2019.
  * [17] Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang.  Theory of the frequency principle for general deep neural networks.  CSIAM Transactions on Applied Mathematics, 2021.
  * [18] David Martin, Charless C. Fowlkes, D. Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  International Conference on Computer Vision, 2001.
  * [19] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  CVPR, 2020.
  * [20] Gary Mataev, Peyman Milanfar, and Michael Elad.  Deepred: Deep image prior powered by red.  ICCV, 2019.
  * [21] Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker.  Modulated periodic activations for generalizable local functional representations.  arXiv: Computer Vision and Pattern Recognition, 2021.
  * [22] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  ECCV, 2020.
  * [23] Francesca Pistilli, Diego Valsesia, Giulia Fracastoro, Enrico Magli Politecnico, and Di Torino.  Signal compression via neural implicit representations.  ICASSP, 2022.
  * [24] Pradyumna Reddy, Zhifei Zhang, Matthew Fisher, Hailin Jin, Zhaowen Wang, and Niloy J. Mitra.  A multi-implicit neural representation for fonts.  ICML, 2021.
  * [25] Olivier Roy and Martin Vetterli.  The effective rank: A measure of effective dimensionality.  ECCV, 2007.
  * [26] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  NeurIPS, 2020.
  * [27] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15:1929–1958, 2014.
  * [28] Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari.  Implicit neural representations for image compression.  In ECCV, 2022.
  * [29] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  NeurIPS, 2020.
  * [30] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Deep image prior.  IJCV, 128:1867–1888, 2018.
  * [31] Tomas Va${\bm{s}}$kevi${\bm{c}}$ius, Varun Kanade, and Patrick Rebeschini.  Implicit regularization for optimal sparse recovery.  NeurIPS, 2019.
  * [32] Krishna Wadhwani and Tamaki Kojima.  Squeezenerf: Further factorized fastnerf for memory-efficient inference.  CVPR, 2022.
  * [33] Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma.  Frequency principle: Fourier analysis sheds light on deep neural networks.  Communications in Computational Physics, 2019.
  * [34] Greg Yang.  Tensor programs ii: Neural tangent kernel for any architecture.  ArXiv, abs/2006.14548, 2020.
  * [35] Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen.  Implicit neural video compression.  ArXiv, abs/2112.11312, 2021.
  * [36] Peng Zhao, Yun Yang, and Qiao-Chu He.  Implicit regularization via hadamard product over-parametrization in high-dimensional linear regression.  arXiv: Statistics Theory, 2019.

##  Appendix A Theoretical analysis

###  A.1 Fourier feature map induce a shift-invariant kernel regression

Notice that $\phi_{\text{NTK}}(\mathbf{x})$ should be shift-invariant, i.e.,
if we shift the training data
$\left\\{(\mathbf{x}_{i},z_{i})\right\\}_{i=1}^{N}$ to
$\left\\{(\mathbf{x}_{i}+\Delta\mathbf{x},z_{i})\right\\}_{i=1}^{N}$, and the
corresponding kernel regression is $\phi_{\text{NTK}}^{+}(\mathbf{x})$, we
look forward
$\phi_{\text{NTK}}^{+}(\mathbf{x}+\Delta\mathbf{x})=\phi_{\text{NTK}}(\mathbf{x})$.
Researchers encode the shift-invariant property by a Fourier feature map
$\gamma(\mathbf{x})=\frac{1}{\sqrt{D}}[\cos\mathbf{Bx}^{\top},\sin\mathbf{Bx}^{\top}]^{\top}:\mathbb{R}^{d}\mapsto\mathbb{R}^{2D}$
as input, where $\mathbf{x}\in\mathbb{R}^{d}$,
$\mathbf{B}\in\mathbb{R}^{D\times d}$, and
$\mathbf{B}_{ij}\sim\mathcal{N}(0,\delta)$ [29]. The NTK can be written as
$h_{\text{NTK}}(\mathbf{x}_{i}^{\top}\mathbf{x}_{j})$,
$h_{\text{NTK}}:\mathbb{R}\mapsto\mathbb{R}$ when $\mathbf{x}_{i}$ on a
hypersphere, so NTK with feature map can be composed as
$h_{\text{NTK}}\left(\gamma(\mathbf{x}_{i})^{\top}\gamma(\mathbf{x}_{j})\right)=h_{\text{NTK}}\left(\frac{1}{D}\mathbf{1}_{D}^{\top}\cos\left(\mathbf{B}(\mathbf{x}_{i}-\mathbf{x}_{j})\right)\right)$
which is shift-invariant. Then
$\phi_{\text{NTK}}(\gamma(\mathbf{x}+\Delta\mathbf{x}))=\phi_{\text{NTK}}(\gamma(\mathbf{x}))$,
and

$\phi_{\text{NTK}}^{\prime}(\mathbf{x})=\phi_{\text{NTK}}(\gamma(\mathbf{x}))=\sum_{i=1}^{N}(\mathbf{H}^{-1}\mathbf{z})_{i}h_{\text{NTK}}(\mathbf{x}_{i},\mathbf{x}),$

where $\mathbf{H}$ is an $n\times n$ PSD matrix with entries
$\mathbf{H}_{ij}=h_{\text{NTK}}(\mathbf{x}_{i},\mathbf{x}_{j})$.

###  A.2 Proof of main theorem

######  Proof 1 (Proof of Theorem 1)

$\displaystyle k_{D}(\mathbf{x}_{i},\mathbf{x}_{j})$
$\displaystyle=h_{\text{NTK}}\left(\frac{1}{D}\mathbf{1}_{D}^{\top}\cos\left(\mathbf{B}(\mathbf{x}_{i}-\mathbf{x}_{j})\right)\right)$
$\displaystyle=h_{\text{NTK}}\left(\sum_{l=1}^{D}\frac{1}{D}\cos\left(\mathbf{B}_{l,:}(\mathbf{x}_{i}-\mathbf{x}_{j})\right)\right),$

where ${\bm{B}}_{l,:}$ is the $l$-th row of ${\bm{B}}$.

Therefore,

$\lim_{D\rightarrow\infty}k_{D}(\mathbf{x}_{i},\mathbf{x}_{j})=h_{\text{NTK}}\left(\mathbb{E}_{\mathbf{b}_{l}\sim\mathcal{N}(0,\delta)}\cos\left(\mathbf{b}^{\top}(\mathbf{x}_{i}-\mathbf{x}_{j})\right)\right),$

where $\mathbf{b}\in\mathbb{R}^{d}$. Furthermore,
$\lim_{D\rightarrow\infty}k_{D}(\mathbf{x}_{i},\mathbf{x}_{j})=h_{\text{NTK}}(e^{-\delta^{2}\left\|\mathbf{x}_{i}-\mathbf{x}_{j}\right\|_{2}^{2}})$.
$\hfill\blacksquare$

######  Proof 2 (Proof of Corollary 1)

As

$\lim_{\begin{subarray}{c}\delta\rightarrow\infty\\\
D\rightarrow\infty\end{subarray}}k_{D}(\mathbf{x},\mathbf{X})=\left\\{\begin{array}[]{cc}h(1)\mathbf{e}_{i}^{\top}&\mathbf{x}\in\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N},\\\
h(0)\bm{1}_{N}^{\top}&\mathbf{x}\not\in\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N},\end{array}\right.$

then,

$\lim_{\begin{subarray}{c}\delta\rightarrow\infty\\\
D\rightarrow\infty\end{subarray}}k_{D}(\mathbf{X},\mathbf{X})=h(0)\bm{1}_{N}\bm{1}_{N}^{\top}+\left(h(1)-h(0)\right){\bm{I}}_{N}.$

That is, the singular value of
$\lim_{\begin{subarray}{c}\delta\rightarrow\infty\\\
D\rightarrow\infty\end{subarray}}k_{D}(\mathbf{X},\mathbf{X})$ are
$h(1),h(1)-h(0),\ldots,h(1)-h(0)$. It’s evidence that
$k_{\infty}(\mathbf{X},\mathbf{X})=\lim_{\begin{subarray}{c}\delta\rightarrow\infty\\\
D\rightarrow\infty\end{subarray}}k_{D}(\mathbf{X},\mathbf{X})$ is invertible
when $h(1)\neq h(0)$ and $h(1)\neq 0$. Then

$\Phi_{\text{NTK}}^{\prime}(\mathbf{X})=k_{\infty}(\mathbf{X},\mathbf{X})k_{\infty}(\mathbf{X},\mathbf{X})^{-1}\mathbf{z}=\mathbf{z},$

that is $\phi_{\text{NTK}}^{\prime}(\mathbf{x}_{l})=z_{l}$.

As for $\mathbf{x}\not\in\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$,

$\displaystyle\Phi_{\text{NTK}}^{\prime}(\mathbf{x})$
$\displaystyle=k_{\infty}(\mathbf{x},\mathbf{X})\cdot
k^{-1}_{\infty}(\mathbf{X},\mathbf{X})\mathbf{z}$ (3)
$\displaystyle=h(0)\bm{1}_{N}^{\top}k^{-1}_{\infty}(\mathbf{X},\mathbf{X})\cdot\mathbf{z}.$

Note that $k_{\infty}(\mathbf{X},\mathbf{X})$ is a particular matrix which has
same column summation that is

$\bm{1}_{N}^{\top}\cdot
k_{\infty}(\mathbf{X},\mathbf{X})=\left((N-1)h(0)+h(1)\right)\bm{1}_{N}^{\top},$

therefore its corresponding eigenvalue is $(N-1)h(0)+h(1)$. Furthermore, as

$\frac{1}{(N-1)h(0)+h(1)}\bm{1}_{N}^{\top}=\bm{1}_{N}^{\top}\cdot
k_{\infty}^{-1}(\mathbf{X},\mathbf{X}),$

we have, $\bm{1}_{N}^{\top}$ is the left eigenvector of
$k_{\infty}(\mathbf{X},\mathbf{X})^{-1}$ and the corresponding eigenvalue is
$\frac{1}{(N-1)h(0)+h(1)}$. Then bring it back to Eq.3, we have

$\Phi_{\text{NTK}}^{\prime}(\mathbf{x})=\frac{h(0)}{(N-1)h(0)+h(1)}\bm{1}_{N}^{\top}\mathbf{z}.$

$\hfill\blacksquare$

##  Appendix B Explain the proposed method step-by-step

For simplicity, we focus on a gray-scale image inpainting task to illustrate
the workflow of our method.

Task: Given a partially observed image $\mathbf{X}$ on $\mathcal{X}$, where
$\mathcal{X}\subseteq\mathcal{G}=\left\\{(\frac{i}{m},\frac{j}{n})|i\in\\{1,2,\cdots,m\\},j\in\\{1,2,\cdots,n\\}\right\\}$,
i.e.,
$\mathcal{Z}=\left\\{\mathbf{X}_{i,j}\mid(\frac{i}{m},\frac{j}{n})\in\mathcal{X}\right\\}$,
find $\mathbf{X}$ on unobserved $\mathcal{G}\backslash\mathcal{X}$.

Input: Training set $\mathcal{X}\times\mathcal{Z}$; initial network parameters
$\bm{\theta}_{a}(0)=\\{\bm{\theta}(0),\bm{\theta}_{r}(0),\bm{\theta}_{c}(0)\\}$;
super-parameters $\lambda_{r},\lambda_{c}$; iteration step $t=0$.

Step 1: Calculate loss function. Loss function is
$\mathcal{L}_{a}(\bm{\theta}_{a})=\mathcal{L}(\bm{\theta}(t),\mathcal{X},\mathcal{Z})+\lambda_{r}\mathcal{R}(\bm{\theta}_{r}(t))+\lambda_{c}\mathcal{R}(\bm{\theta}_{c}(t))$,
where
$\mathcal{L}=\sum_{(\mathbf{x}_{i},z_{i})\in\mathcal{X}\times\mathcal{Z}}\left\|\phi_{\bm{\theta}(t)}(\mathbf{x}_{i})-z_{i}\right\|_{2}^{2}$,
$\mathcal{R}(\bm{\theta}_{r}(t))=\text{tr}\left(\mathbf{X}^{\top}\mathbf{L}(\bm{\theta}_{r}(t))\mathbf{X}\right)$
and
$\mathcal{R}(\bm{\theta}_{c}(t))=\text{tr}\left(\mathbf{X}\mathbf{L}(\bm{\theta}_{c}(t))\mathbf{X}^{\top}\right)$
measure the similarity between rows and columns in image respectively.

Step 2: Update parameters. Minimize $\mathcal{L}_{a}$ by updating parameters
$\bm{\theta}_{a}(t)$ with optimization algorithm such as Adam.

Iteration stops at $t=T$ when $\mathcal{L}_{a}(\bm{\theta}_{a}(T))$ is smaller
than some given precision.

Step 3: Output estimation. The pixel value of $\mathbf{X}_{ij}$ on
$\mathcal{G}\backslash\mathcal{X}$ is predicted by
$\phi_{\bm{\theta}(T)}(\frac{i}{m},\frac{j}{n})$.

For high-dimensional data such as video, the regularizer captures the
similarity between the vectorized frames.

##  Appendix C Additional experiments

As a general image representation model, our method can be readily applied to
other image tasks, including those higher dimensional ones. Table 3 shows that
INRR outperforms INR in image denoising under variant noise types. Besides, we
have also verified that INRR shows its power in video frame interpolation and
RGB image inpainting.

Table 3: PSNR (dB) of denoised images by INR and INRR on (a) Baboon, (b) Man, (c) Barbara, (d) Boats and (e) Cameraman. Noise Type | Method | (a) | (b) | (c) | (d) | (e)
---|---|---|---|---|---|---
Gaussian ($\sigma=10$) | INR | 28.5 | 29.3 | 30.5 | 30.8 | 33.1
INRR | 29.0 | 29.7 | 30.6 | 31.2 | 33.3
Salt & Peper ($r=0.95$) | INR | 21.9 | 23.0 | 25.4 | 24.7 | 25.9
INRR | 22.0 | 23.1 | 25.8 | 24.9 | 25.9
Poisson ($\lambda=50$) | INR | 23.2 | 24.9 | 27.0 | 25.9 | 27.7
INRR | 23.6 | 25.2 | 27.1 | 26.3 | 28.1

Here we test INRR on video interpolation, where the data is represented by
$\phi_{\bm{\theta}}(x,y,t)=[r,g,b]:\mathbb{R}^{3}\mapsto\mathbb{R}^{3}$. Each
frame is an RGB image, as shown in Figure 10 (a). The tested video is a scene
of water droplets that owns 202 frames. We sample 21 frames uniformly as
training data; the rest are test data. Figure 10 (b) shows that INRR can
capture the non-local self-similarity between different frames. The average
PSNR of INRR is 37.5 dB, while vanilla INR is 36.8 dB. Furthermore, we have
validated the inpainting performance on a dataset BSD100 [18] which includes
100 RGB images. The sampling mode is the same as in Figure 2(a) in the paper.
The average PSNR of INRR is 28.8 dB, while vanilla INR is 27.1 dB.

Figure 10: The similarity between different frames (a) can be captured by
$\mathbf{L}$ (L392) in INRR (b) which is helpful to frame interpolation.

##  Appendix D Smoothness of Laplacian matrix

Figure 11: The first row show five different $256\times 256$ gray-scale
images. The second row shows the column covariance matrix, and the third row
shows the row covariance matrix.

We first calculate the covariance matrix of the columns and the rows in Figure
11. The covariance matrix of $\mathbf{X}$ is $\mathbf{C}(\mathbf{X})$, where
$\mathbf{C}_{ij}=\mathbb{E}[\mathbf{X}_{:,i}-\mathbb{E}(\mathbf{X}_{:,i})][\mathbf{X}_{:,j}-\mathbb{E}(\mathbf{X}_{:,j})]$,
which measures the similarity among the columns. While the similarity among
rows is $\mathbf{C}(\mathbf{X}^{\top})$. As we can see, all the covariance
matrices of various images are locally smoothly. Dong et al. proposed to
utilize the smoothness of the Laplacian matrix by an extra regularizer [4]. We
use an INR to encode the smoothness in such a Laplacian matrix implicitly;
that is, our proposed INRR combines the self-similarity and smoothness of the
Laplacian matrix at the same time.

Because $g(\bm{\theta};\mathbf{u})$ is an INR which is a smooth FCN about
$\mathbf{u}$,
$\mathbf{A}(\bm{\theta})=\frac{\text{exp}\left(g^{\top}(\bm{\theta};\mathbf{u})g(\bm{\theta};\mathbf{u})\right)}{\mathbf{1}^{\top}\text{exp}\left(g^{\top}(\bm{\theta};\mathbf{u})g(\bm{\theta};\mathbf{u})\right)\mathbf{1}}$
is smooth according to its expression. It means that a slight change of
$\mathbf{u}$ generally leads to a slight change of
$\mathbf{L}(\bm{\theta})=\mathbf{D}(\bm{\theta})-\mathbf{A}(\bm{\theta})$,
which can be controlled by a Lipschitz constant. So we conclude that
$\mathbf{A}(\bm{\theta})$ smoothes $\mathbf{L}(\bm{\theta})$, which is
different from the vanilla $\mathbf{L}$. Furthermore, we can deduce a
conclusion similar to Theorem 1 that the smoothness of
$\mathbf{L}(\bm{\theta})$ is controlled by the $\delta$ of
$g(\bm{\theta};\cdot)$. Smaller $\delta$ leads to a smoother result. Moreover,
INRR degenerates to AIR when $\delta\rightarrow\infty$.

##  Appendix E Implicit bias

Figure 12: Fitting a $256\times 256$ synthetic data with a combine missing at
different training step with (a) single matrix (DMF with only one factor), (b)
DMF with three factors, (c) ReLU neural network, and (d) SIREN. The effective
rank of the fitted matrix is shown in (e). Figure 13: Fitting a $256\times
256$ synthetic data with Gaussian noise at different training step with (a)
single matrix (DMF with only one factor), (b) DMF with three factors, (c) ReLU
neural network, and (d) SIREN. The effective rank of the fitted matrix is
shown in (e).

The neural network tends to converge to a good solution and may suffer from
over-fitting with the training goes. Researchers explain this phenomenon by
the implicit bias of neural networks. We show the implicit bias by fitting the
synthetic data, which is sampled from function
$s(x,y)=\sin\left(25\pi\sin\left(\frac{\pi}{3}\cdot\sqrt{x^{2}+y^{2}}\right)\right)$,
where $\\{(x_{i},y_{j})|i,j\\}$ is a uniform $256\times 256$ grid on
$[-1,1]\times[-1,1]$, where the local frequency of the synthetic data
increases from boundary to center. We show two tasks on such synthetic data:
fitting the incomplete data and the noisy data in Figure 12 and Figure 13,
respectively.

As Figure 12(a) shows, the single layer DMF fits all pixels without bias, and
the pixels of the fitted image increase gradually. While Figure 12(b) shows
the low-rank bias of the three-layer DMF fitting the synthetic data from low-
rank to high-rank. Figure 12(c,d) shows that a fully connected neural
network’s bias is related to the data frequency and sampling rate. With the
bias mentioned above, it is possible to complete an image without an extra
explicit regularizer. Similarly, these neural network has similar phenomenon
when fitting the noisy data.

##  Appendix F Recovered image

Figure 14: Result of image inpainting with three types of missing data by
different regularized INR including (b) INR without regularization, (c) with
TV, (d) $L_{2}$, (e) AIR, (f) INRR, and (g) original image. The hyper-
parameters of benchmark models and algorithms are adopted from the original
paper.

We show the recovered image in Figure 14.