# Adversarial Generation of Continuous Images

Ivan Skorokhodov1 Savva Ignatyev2 Mohamed Elhoseiny1
1King Abdullah University of Science and Technology (KAUST)
2Skolkovo Institute of Science and Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

In most existing learning systems, images are typically viewed as 2D pixel
arrays. However, in another paradigm gaining popularity, a 2D image is
represented as an implicit neural representation (INR) — an MLP that predicts
an RGB pixel value given its $(x,y)$ coordinate. In this paper, we propose two
novel architectural techniques for building INR-based image decoders:
factorized multiplicative modulation and multi-scale INRs, and use them to
build a state-of-the-art continuous image GAN. Previous attempts to adapt INRs
for image generation were limited to MNIST-like datasets and do not scale to
complex real-world data. Our proposed INR-GAN architecture improves the
performance of continuous image generators by several times, greatly reducing
the gap between continuous image GANs and pixel-based ones. Apart from that,
we explore several exciting properties of the INR-based decoders, like out-of-
the-box superresolution, meaningful image-space interpolation, accelerated
inference of low-resolution images, an ability to extrapolate outside of image
boundaries, and strong geometric prior. The project page is located at
https://universome.github.io/inr-gan.

##  1 Introduction

In deep learning, images are typically represented as 2D arrays of pixels.
However, there is another paradigm which views an image as a continuous
function $F(\bm{p})=\bm{v}$ that maps a pixel’s 2D coordinate
$\bm{p}=(x,y)\in\mathbb{R}^{2}$ to its RGB value
$\bm{v}=(r,g,b)\in\mathbb{R}^{3}$. The advantage of such a representation is
that it gives a true continuous version of the underlying 2D signal instead of
its cropped quantized counterpart like pixel-based representations do. In
practice, we almost never know the underlying function $F(\bm{p})$ and thus
have to work with its approximations. The most popular and expressive way to
approximate $F(\bm{p})$ is through a neural network $\mathsf{F}_{\bm{\theta}}$
[80, 73]. It is called an implicit neural representation (INR) and is
especially popular in 3D deep learning where working with voxels directly
(i.e. pixels defined on a 3D grid) is too expensive [50, 11, 66, 54].

Figure 1: Comparison between a traditional convolutional generator (left) and
an INR-based one (right). A traditional generator directly generates a pixel-
based image representation given its latent code $\bm{z}$. The INR-based one
produces parameters of an MLP. The corresponding pixel-based representation is
obtained by evaluating the INR at each coordinate location $(x,y)$ of a
specified grid.

Building such a decoder has two severe difficulties: 1) since it is a
hypernetwork, i.e. a network that produces parameters for another network
[26], it is unstable to train and requires too many parameters in general [9];
and 2) it is too costly to evaluate INRs for a dense high-resolution
coordinates grid limiting their application to low-resolution images only. To
alleviate these issues, we design two principled architectural techniques:
factorized multiplicative modulation (FMM) layer for hypernetworks and multi-
scale INRs. We use these techniques to build INR-GAN: a state-of-the-art
continuous image generator that generates pictures in their INR
representations. Previous attempts to build such a model were only conducted
on small MNIST-like datasets [11, 3, 50] and do not scale to complex real-
world data. In our case, we managed to achieve FID [28] scores of 5.09, 4.96
and 16.32 on LSUN Churches $256^{2}$, LSUN Bedrooms $256^{2}$ and FFHQ
$1024^{2}$, respectively, greatly reducing the gap between continuous image
GANs and their pixel-based analogs. In their contemporary work, [1] achieved
even better results by employing a large-scale INR-based decoder with
learnable coordinate embeddings.

In our paper, we also shed light on many interesting properties of the INR-
based decoders:

  * •

Extrapolating outside of image boundaries (see fig:oob-generalization): an
ability to generate a “zoomed-out” version of an image without being trained
explicitly to do this.

  * •

Geometric prior (see fig:keypoints-prediction): better encoding of geometric
properties of a dataset in the latent space.

  * •

Accelerated low-resolution inference (see fig:fast-lowres-generation): an
ability to generate a low-resolution version of an image in shorter time than
an image of the corresponding training-time resolution.

  * •

Meaningful image space interpolation (see fig:inr-space-interpolation).

  * •

Out-of-the-box superresolution (see fig:superresolution): an ability to
produce a higher-resolution version of an image without being trained for this
task at all.

We emphasize that these features come naturally to INR-based decoders and do
not require any additional training.

To summarize, our contributions are the following:

  1. 1.

We propose a novel factorized multiplicative modulation (FMM) layer for
hypernetworks. It makes it possible to generate INRs with a large number of
parameters and stabilizes hypernetwork training.

  2. 2.

We propose a novel multi-scale INR architecture. It makes it possible to
represent high-resolution images in the INR-based form in a very efficient
way.

  3. 3.

Using the above two techniques we build INR-GAN: an INR-based GAN model that
outperforms existing continuous image generators by several times on large
real-world datasets.

  4. 4.

We explore several exciting properties of INR-based decoders, like out-of-the-
box superresolution, meaningful image-space interpolation, accelerated
inference of low-resolution images, an ability to extrapolate outside of image
boundaries, and geometric prior.

Figure 2: Extrapolating outside of image boundaries. After training our INR-
based GAN on LSUN $256^{2}$, we tried to evaluate it on a wider grid. During
training, we used coordinates from $[0,1]^{2}$ square, and here we evaluate it
on coordinates from $[-0.3,1.3]^{2}$ square. To our surprise, the model can
produce meaningful extrapolation and generate the picture beyond the
coordinates area it was trained on. Blue bounding box denotes $[0,1]^{2}$
coordinates area — an image area the model was trained on. Very similar
results were previously shown by [48].

##  2 Related work

Implicit Neural Representations. The original idea of augmenting neural
networks with coordinates information was proposed in CPPN [77] which is a
neuroevolution-based model that is trained to represent a 2D image. After
that, there were several works that use coordinates as an additional source of
information to neural networks [51, 86, 85, 67, 76], but the largest
popularity of implicit neural representations (INRs) is observed in 3D deep
learning, where it provides a cheap and continuous way to represent a 3D shape
compared to mesh/voxel/pointcloud-based ones [54, 66, 50, 21, 20, 74]. They
have also been used for other tasks, like representing textures [62], 3D
shapes flow [59], scenes [56, 75], audios and differential equations [73],
human grasps [40] and other information [55, 13]. Occupancy Networks [54, 68,
13, 35] model a probability function of a voxel being occupied by a 3D shape
and typically employ a coordinate-based decoder that operates on top of
single-view images. They use the multi-resolution surface extraction method,
which is similar in nature to our proposed multi-scale INR. However, in our
case, we share computation between neighboring pixels while they use surface
extraction to find regions to refine the predictions on. In this way, they
conduct the full inference for each low-resolution coordinate which is the
opposite of what we try to achieve with multi-scale INRs. DeepSDF [66] models
a signed distance function instead of the occupancy function and they
additionally have an encoder, which transforms an image into a latent code.
IM-NET [11] proposes to train a generative model on top of these latent codes
and conduct experiments not only on ShapeNet objects [8], but on MNIST images
as well. DeepMeta [50] models the occupancy function and predicts decoder
parameters instead of the latent codes. A vital branch of research on INRs is
concerned about the most efficient way to encode coordinates positions [83,
19]. Recent works show that using Fourier features greatly improves INR
expressiveness [80, 73], which we observe in our experiments as well.

(a) Nearest (19.24)

(b) Bilinear (29.51)

(c) Bicubic (30.3)

(d) INR-based (11.81)

Figure 3: Out-of-the-box superresolution properties of the INR-based decoder.
We train our INR-GAN on LSUN $128\times 128$ and perform upsampling at test
time to $256\times 256$ resolution with either classical interpolation
techniques (first 3 columns) or with “natural” INR-based upsampling by
evaluating $\mathsf{F}_{\bm{\theta}}$ on a denser coordinates grid. INR-based
superresolution is much crisper and obtained without any additional
supervision. Numbers in parentheses denote the corresponding UpsampledFID
scores.

GANs. State-of-the-art results in image generation are held by generative
adversarial networks (GANs) [23]. Two key challenges in GAN training are
instability and mode collapse, that is why a big part of research in recent
years was devoted to finding stronger objective formulation [2, 52, 47],
regularizers [25, 53, 57] and architectural designs [37, 38, 39, 36] that
would encourage stability, diversity and expressivity of the GAN-based models.

Generative models + coordinates. There were attempts to combine generative
modeling and INR-based representations prior to our work. IM-NET [11] trains a
generative model on top of latent codes of an occupancy autoencoder. In our
case, instead of feeding a latent code to a coordinate-based decoder, we
produce its parameters with a hypernetwork-based generator. Besides, their
image generation experiments were limited to small-scale MNIST-like datasets
[45] only. CoordConv GAN [51] concatenates coordinates to each representation
in the DCGAN model, which endows it with geometric translating behavior during
latent space interpolation. [90] use CoordConv layers to perform
superresolution. SBD [87] augments a VAE [42] decoder with coordinates
information. SpatialVAE [3] additionally models rotation and translation
separately from the latent codes COCO-GAN [48] generates images by patches
given their spatial information and then assembles them into a single image.
[92] uses coordinates-based convolutions for sky replacement and video
harmonization. In their contemporary work, [1] builds an INR-based generator
with learnable coordinate embeddings and achieves state-of-the-art results on
several large-scale datasets.

(a) StyleGAN2

(b) INR-GAN

Figure 4: Predicting keypoints from latent codes. We train a linear model to
predict face keypoints directly from the corresponding latent codes. Its
performance shows how much geometric information is contained in a latent code
and how accessible it is. For this benchmark, our model easily outperforms
StyleGAN2 despite it being trained with the additional PPL loss [39] that
forces better latent codes conditioning in the decoder. That demonstrates
better geometric prior of our model. The corresponding scores are presented in
Table 3.

Hypernetworks. Hypernetworks or meta-models are models that generate
parameters for other models [26, 49]. Such parametrization provides higher
expressivity [17, 18] and compression due to weight sharing through a meta-
model [26]. Our factorized multiplicative modulation (FMM) is closely related
to the squeeze-and-excitation mechanism [30]. But in contrast to it, we
modulate weights instead of hidden representations, similar to [10], where
authors condition kernel weights on an input via attention. Hypernetworks are
known to be unstable to train [82] and [9] proposed a principled
initialization scheme to remedy the issue. In our case, we found it to be
unnecessary since our FMM module successfully reduces the influence of
hypernetwork initialization on signal propagation inside an INR. Hypernetworks
found many applications in other areas like few-shot learning [4], continual
learning [84], architecture search [91] and others. In the case of generative
modeling, [79] built a character-level language model, and [70] proposed a
generative hypernetwork to produce parameters of neural classifiers. Similar
to our FMM, [78] proposed a low-rank modulation by parametrizing a target
model’s weight matrix as $\bm{W}=\bm{W}_{s}\odot(\bm{A}\times\bm{B})$, where
$\bm{W}_{s}$ is a shared component and $\bm{A},\bm{B}$ are rectangular
matrices produced by a hypernetwork.

Hypernetworks + generative modeling. Combining hypernetworks and generative
models is not new. In [70] and [27], authors built a GAN model to generate
parameters of a neural network that solves a regression or classification task
and demonstrate its favorable performance for uncertainty estimation. HyperVAE
[58] is designated to encode any target distribution by producing generative
model parameters given distribution samples. HCNAF [63] is a hypernetwork that
produces parameters for a conditional autoregressive flow model [41, 65, 31].
[7] trains a generator to produce a 3D object in the INR form and uses
mFiLM[69, 14] to compress the output space.

Hypernetworks + INRs. There are also works that combine hypernetworks and
INRs. [75] represents a 3D scene as an INR, rendered by differentiable ray-
marching, and trains a hypernetwork to learn the space of such 3D scenes. [44]
proposes to represent an image dataset using a hypernetwork and perform super-
resolution by passing denser coordinate grid into it. DeepMeta [50] builds an
encoder that takes a single-view 3D shape image as input and outputs
parameters of an INR. Authors also trained a model to encode a MNIST image
into a temporal sequence of digits.

Computationally efficient models. Among other things, we demonstrate that our
INR-based decoder enjoys faster inference speed compared to classical
convolutional ones. Each INR layer can be seen as a $1\times 1$ convolution
[29, 72], i.e. a convolution with the kernel size of 1. The core difference is
that layer’s weights are produced with a hypernetwork $\mathsf{G}$. Since INR
may have a lot of parameters, we factorize them with low-rank factorization
[89]. There is a vast literature on using low-rank matrix approximations to
compress deep models or accelerate them [60, 12, 33, 15]. In our case, we
found it sufficient just to output a weight matrix as a product of two low-
rank matrices $\bm{W}=\bm{A}\times\bm{B}$ without using any specialized
techniques. SENet [30] proposed to apply squeeze-and-excitation mechanism on
the hidden representations, and we apply them on the INR weights to make the
model more stable to train and faster to converge.

##  3 Image Meta Generation

###  3.1 Model overview

We adopt the standard GAN training setup and replace a convolutional generator
with our INR-based one, illustrated in Figure 1. We build upon the StyleGAN2
framework and keep every other component except for the generator untouched,
including the discriminator, losses, optimizers, and the hyperparameters. The
details are in Appendix A.

Our generator $\mathsf{G}$ is hypernetwork-based: it takes latent code
$\bm{z}\sim\mathcal{N}(\bm{0},\bm{I})$ as input and generates parameters
$\bm{\theta}$ for an INR model $\mathsf{F}_{\bm{\theta}}$. To produce an
actual image, we evaluate $\mathsf{F}_{\bm{\theta}}$ at all the locations of a
predefined coordinates grid, which size is determined by the dataset
resolution. For example, for LSUN bedroom $256^{2}$ we compute pixel values at
$256^{2}=65536$ grid locations of $[0,1]^{2}$ square, that are positioned
uniformly. We use recently proposed Fourier features to embed the $(x,y)$
coordinates [80, 73]. From the implementation perspective, these Fourier
features is just a simple linear layer $\bm{u}=\sin(\gamma\bm{U}\bm{p})$ with
sine (or cosine) activation that maps a raw coordinate vector $\bm{p}=(x,y)$
into a feature vector $\bm{u}\in\mathbb{R}^{d_{f}}$. Note that our embedding
matrix $\bm{U}$ is not kept fixed and shared but predicted by the generator
from $\bm{z}$. This gives the model flexibility to select feature frequencies
that are the most appropriate for a given image.

(a) Image interpolation in the pixel-based form.

(b) Image interpolation in the INR-based form.

Figure 5: Images have meaningful interpolation when represented in the INR-
based form. To interpolate between $\mathsf{F}_{\bm{\theta_{1}}}$ and
$\mathsf{F}_{\bm{\theta_{2}}}$, we compute interpolation parameters
$\bm{\theta}=\alpha\bm{\theta}_{1}+(1-\alpha)\bm{\theta}_{2}$ and evaluate
$\mathsf{F}_{\bm{\theta}}$ for the provided coordinates grid.

###  3.2 Factorized Multiplicative Modulation (FMM)

(a) FMM Linear layer.

(b) INR-based generator with FMM.

Figure 6: (a) FMM linear layer for inputs $\bm{x}$, shared matrix $\bm{W}_{s}$
and output $\bm{y}$; (b) INR-based generator with the FMM mechanism: its
parameters are split into $\bm{\theta}_{s}$ (shared) and $\bm{\theta}$
(predicted by the hypernetwork). This mechanism makes our architecture be
similar to StyleGAN2 [39]: the hypernetwork becomes a mapping network and the
INR $\mathsf{F}_{\bm{\theta}}$ becomes the synthesis network (decoder).

A hypernetwork is a model that generates parameters for another model. In our
case, we want to generate INR’s parameters given the noise vector $\bm{z}$.
Imagine, that we need to produce the weights
$\bm{W}^{\ell}\in\mathbb{R}^{n_{\text{in}}\times
n_{\text{out}}},\bm{b}^{\ell}\in\mathbb{R}^{n_{\text{out}}}$ of the $\ell$-th
linear layer of $\mathsf{F}_{\bm{\theta}}$. A naive implementation would
output them directly, but this is extremely inefficient: if the hypernetwork
has the hidden dimensionality of size $h$, then its output projection matrix
will have the size $h\times(n_{\text{in}}\cdot
n_{\text{out}}+n_{\text{out}})$. Even for small $n_{\text{in}},n_{\text{out}}$
this is prohibitively expensive.

The main problem lies in generating $\bm{W}^{\ell}$ since it contains most of
the weights, thus factorizing it via low-rank matrix decomposition
$\bm{W}^{\ell}=\bm{A}^{\ell}\times\bm{B}^{\ell}$ might seem like a reasonable
idea. However, our preliminary experiments showed that it severely decreases
the performance because having low-rank weight matrices is equivalent to
having a lot of zero singular values, leading to severe instabilities in GAN
training [5, 57]. That is why we change the parametrization so that the full
rank is preserved while the hypernetwork output is still factorized.

Namely, we first define a shared parameters matrix
$\bm{W}_{s}^{\ell}\in\mathbb{R}^{n_{\text{out}}^{\ell}\times
n_{\text{in}}^{\ell}}$, which is learnable and shared across all samples. Our
hypernetwork produces two rectangular matrices
$\bm{A}^{\ell}\in\mathbb{R}^{n_{\text{out}}^{\ell}\times r}$ and
$\bm{B}^{\ell}\in\mathbb{R}^{r\times n_{\text{in}}^{\ell}}$, which are
multiplied together to obtain a low-rank modulating matrix
$\bm{W}_{h}^{\ell}=\bm{A}^{\ell}\times\bm{B}^{\ell}$. We compute the final
weight matrix $\bm{W}^{\ell}$ as
$\bm{W}^{\ell}=\bm{W}_{s}^{\ell}\odot\sigma(\bm{W}_{h}^{\ell})$ where $\sigma$
is sigmoid function. Bias vector $\bm{b}^{\ell}$ is produced directly since it
is small. In all the experiments, we set $r=10$ for all the layers of
$\mathsf{F}_{\bm{\theta}}$ except the first and the last ones which are small
enough to be generated directly. We illustrate FMM in Figure 6.

The above modulation is very similar to the one proposed in [78] with the
difference that [78] does not use the sigmoid activation. However, as our
experiments in Section 4 demonstrate, using the activation is crucial since it
bounds the activations and makes the training more stable. Note that the FMM-
based INR-GAN becomes very close architecturally to StyleGAN2 [39], which uses
a mapping network to predict the style vector used to modulate the
convolutional weights of its decoder via multiplication.

###  3.3 Multi-scale INRs

Scaling a traditional INR-based decoder to large image resolutions is too
expensive: to produce a $1024^{2}$ image, we have to input ${\approx}10^{6}$
coordinates into $\mathsf{F}_{\bm{\theta}}$. Such a huge batch size makes it
impossible to use large hidden layers’ sizes due to excessive computation and
memory consumption. To circumvent the issue, we propose a multi-scale INR
architecture: we split $\mathsf{F}_{\bm{\theta}}$ into $K$ blocks, where each
block operates on its own resolution and only the final block operates on the
target one. Earlier blocks compute low-res features that are then replicated
and passed to the next level. This process is illustrated on fig:multi-scale-
inr and is equivalent to using different grid sizes depending on the
resolution and using the nearest neighbor interpolation for the hidden
representations. For the multi-scale INRs, we use more neurons for lower
resolutions and fewer ones for the more expensive high-resolution blocks. The
use of multi-scale INRs makes our architecture very similar to classical
convolutional decoders, which grow the resolution progressively with depth
[61]. In our case, one of the additional benefits of having the multi-scale
architecture is that neighboring pixels get conditioned on the common context
computed at a previous resolution.

We start with the resolution of $64^{2}$ for the first block and increase it
by 2 for each next one until we reach the target resolution. Each block
contains 2-4 layers, and Fourier coordinates features are concatenated to a
hidden representation at the beginning of each block. Replicating low-
resolution features for each high-dimensional pixel is equivalent to
upsampling the inner grid with nearest neighbors interpolation, and this is
the way we implement it in practice. Further implementation details can be
found in the supplementary material or the attached source code.

Figure 7: Multi-Scale INR-based GAN (without FMM). Each block operates on a
different resolution, determined by the granularity of an input grid. We
increase the granularity with depth: this allows to share computation between
neighbouring pixels and condition them on a common context. We depict the
multi-scale mechanism without FMM not to clutter the illustration. In
practice, we use both FMM and the multi-scale architecture for our INR-GAN.

##  4 Experiments

###  4.1 Standard GAN training

Table 1: We start with the INR-based decoder conditioned on the latent code
[11, 3] and progressively improve it. O/M denotes “out-of-memory” error: we
couldn’t train the model even for a batch size of 1 on a 32GB NVidia V100 GPU.

Decoder type | LSUN $128^{2}$ | LSUN $256^{2}$ | FFHQ $1024^{2}$
---|---|---|---
GMACs | #params | FID | GMACs | #params | FID | GMACs | #params | FID
Latent-code conditioned INR decoder [43, 3] | 30.09 | 7.1M | 229.9 | 120.33 | 7.1M | 253.3 | 1925.22 | 7.1M | O/M
\+ Hypernetwork-based decoder [50] | 23.54 | 2055.5M | 28.83 | 88.01 | 2055.5M | O/M | 1377.52 | 2055.5M | O/M
\+ Fourier embeddings from [73, 80] (ours) | 28.02 | 2312.02M | 23.07 | 105.34 | 2312.02M | O/M | 1651.52 | 2312.02M | O/M
\+ Factorized Multiplicative Modulation (ours) | 25.87 | 108.2M | 11.51 | 103.19 | 108.2M | 15.68 | 1649.37 | 108.2M | O/M
\+ Multi-Scale INR (ours) | 21.58 | 107.03M | 5.69 | 38.76 | 107.03M | 6.27 | 47.35 | 117.3M | 16.32
StyleGAN2 generator [38] | - | - | - | 84.36 | 30.03M | 2.65 | 143.18 | 30.37M | 4.41

Datasets. We conduct experiments on four datasets: LSUN Bedrooms $128^{2}$,
LSUN Bedrooms $256^{2}$, LSUN Churches $256^{2}$ and FFHQ $1024^{2}$. LSUN
Bedrooms and LSUN Churches consist on 3M and 125k images of in-the-wild
bedrooms and churches [88], respectively. FFHQ is a high-resolution dataset of
70k human faces [38]. For all the datasets, we apply random horizontal flip
for data augmentation.

Evaluation metrics. We evaluate the model using Frechet Inception Distance
(FID) [28] metric using 50k images to compute the statistics. We also compute
the number of parameters for a given model and the amount of multiply-
accumulate operations (MACs) — a standard measure for accessing the model’s
computational efficiency [29, 16].

Models and training details. All our models have equivalent training settings
and differ only in the generator architecture. We use two existing
architectures as our baseline:

  1. 1.

A non-hypernetwork-based generator, which has shared parameters for all INRs
and each INR is conditioned on latent code $\bm{w}=\mathsf{G}(\bm{z})$ [11, 3,
66]. I.e., instead of producing parameters for $\mathsf{F}_{\bm{\theta}}$, we
pass $\bm{w}$ into it as an additional input
$\bm{v}=\mathsf{F}_{\bm{\theta}}(x,y,\bm{w})$.

  2. 2.

A hypernetwork-based generator, which produces parameters $\bm{\theta}$ for
$\mathsf{F}_{\bm{\theta}}$ but does not use any factorization techniques to
reduce the output matrix size [50].

We build upon the above baselines by incorporating Fourier positional
embeddings of the coordinates [80, 73], incorporating our FMM layer and
incorporating our multi-scale INR architecture. In all the experiments,
$\mathsf{G}$ is a 4-layer MLP with residual connections that takes
$\bm{z}\in\mathbb{R}^{512}$ as input and produces INR parameters $\bm{\theta}$
as output. For the first baseline, it produces the transformed latent codes
instead. For hypernetwork-based models, we additionally apply a learned linear
layer to $\bm{w}$ to obtained INR parameters $\bm{\theta}$. In all the
experiments, a ResNet-based discriminator from StyleGAN2 [39] is used.
However, since beating the scores is not the goal of the paper, we used its
“small” version (corresponding to config-e). $R_{1}$-regularization [53] with
the penalty weight of 10 is used. All the models are trained for 800k
iterations on 4 NVidia V100 32GB GPUs.

Results. The results are reported in Table 1. As we can see, the latent code
conditioned baseline cannot fit training data at all since it lacks
expressivity and cannot capture the whole image representation in the latent
code. Its hypernetworks-based counterpart achieves much higher performance but
is still not competitive. An attempt to fit the baselines on FFHQ $1024^{2}$
resulted in out-of-memory errors even for a batch size of 1 for a 32GB GPU.
The non-factorized hypernetwork-based decoder [50] was too expensive even for
$256^{2}$ resolution.

Our INR-based decoder with FMM layer and multi-scale INR architecture achieves
competitive FID scores and greatly reduces the gap between continuous and
pixel-based image generation. It is still inferior to StyleGAN2 [39] in terms
of performance, but is three times more efficient. One should also note that
we didn’t use any of the numerous training tricks employed by StyleGAN2 and
that our discriminator architecture is smaller (it corresponds to config-e
[39]). Besides, as we show in Section 4.2, our model naturally gives rise to a
lot of additional properties that convolutional decoders lack. Our model also
has three times more parameters than its convolution-based counterpart. The
reason for it is the huge size of the output projection matrix of
$\mathsf{G}$, which has the dimensionality of $\approx 10^{3}\times 10^{5}$
occupying 90-99% of parameters of the entire model. Compressing this matrix is
an ongoing hypernetworks research topic [26, 15] and we leave this for future
work. We emphasize that despite its enormous size, the output layer’s
inference time is almost negligible due to highly optimized matrix-matrix
multiplications on modern GPUs.

Ablating FMM rank. In all the experiments, we use an FMM rank of 10. We ablate
over its importance for LSUN bedroom $256^{2}$ and report the resulting
convergence plots on fig:fmm-ranks-ablation. As one can see, the model
benefits from the increased rank, but the advantage is diminishing for further
rank increase. These results show that vanilla or ”full-rank” hypernetworks
are heavily overparameterized and there is no need to predict each parameter
separately. On the other that is the evidence that there is much potential for
”going further” than squeeze-and-excitation [30] and AdaIN [32, 38]
approaches, i.e. predicting more than a single modulating value for each
neuron.

Figure 8: FID scores (in log scale) on LSUN Bedroom $256^{2}$ for different
rank values of the proposed FMM modulation. Increasing the rank beyond 1 gives
a clear advantage, but it diminishes for the subsequent rank increase. “kimg”
denotes the number of images seen by $\mathsf{D}$. Table 2: FID scores for
additional ablations of our multi-scale INR-based GAN (INR-GAN). Removing
$\sigma(x)$ from FMM worsens the scores. Incorporating StyleGAN2’s
architecture and bilinear upsampling allows the INR-based generator to rival
its convolution-based counterpart. Img/sec of $\mathsf{G}$ is measured on 1
NVidia V100 32GB.

Decoder type | Churches $\downarrow$ | Bedrooms $\downarrow$ | img/sec $\uparrow$
---|---|---|---
INR-GAN w/o FMM activation | 10.35 | 11.73 | 267.3
INR-GAN | 7.12 | 6.27 | 267.1
\+ StyleGAN2 architecture | 5.09 | 4.96 | 265.1
\+ bilinear upsampling | 3.12 | 3.41 | 203.4
StyleGAN2 | 3.86 | 2.65 | 85.5

Additional ablations. All the above experiments were conducted on top of a
vanilla INR-based decoder which uses nearest neighbour interpolation (to make
neighboring pixels be computed independently as the vanilla INR does). It also
lacks important StyleGAN2’s techniques which improve the performance, like
equalized learning rate [37], style mixing, pixel normalization [37] and noise
injection [38]. In all our experiments, we also used a small-size version of
StyleGAN2’s discriminator to make the training run faster. Incorporating the
above design choices and giving up the nearest neighbour upsampling for the
bilinear one (which might be reasonable for some applications) significantly
improves our generator’s performance. We conduct those experiments on LSUN
Churches $256^{2}$ and LSUN Bedrooms $256^{2}$ and report the results in Table
2. For the INR-based decoder with bilinear upsampling, we even surpass
StyleGAN2’s performance on Churches.

Another critical question is how important the activation in FMM is. As
discussed in Section 3.2, it allows to stabilize training by bounding the
magnitudes of the weights. We ablate its influence by replacing $\sigma$ with
the identity mapping and report the resulted FID in Table 2. The drop in
performance demonstrates that it carries a crucial influence on the image
quality.

Performance on multi-class datasets. To test if the proposed architectural
techniques improve the performance on diverse multi-class datasets, we also
conduct experiments on LSUN-10 $256^{2}$ and MiniImageNet $128^{2}$. We
provide the details of these experiments in Appendix F.

###  4.2 Exploring the properties

Extrapolating outside of image boundaries. At test time, we sample pixels
beyond the coordinates grid that the model was trained on and present the
results on fig:oob-generalization. It shows that an INR-based decoder is
capable of generating meaningful content outside of the grid boundaries, which
is equivalent to a zooming-out operation. It is a surprising quality
indicating that the coordinates features produced by the generator are
exploited by an INR in a generalizable manner instead of just being simply
memorized.

Meaningful interpolation. Image space interpolation is known for its poor
behavior [22]. However, when images are represented in the INR-based form and
not the pixel-based one, the interpolation becomes reasonable. We illustrate
the difference on fig:inr-space-interpolation.

Keypoints prediction. Direct access to coordinates provides more fine-grained
control over the geometrical properties of images during the generation
process. Thus one can expect the INR-based decoder to better embed the
geometric structure of an image into the latent space [51]. To test this
hypothesis, we perform the following experiment. We take 10k samples from our
model trained on FFHQ $256^{2}$ and extract face keypoints with a pre-trained
model [6]. After that, we fit a simple linear regression model to predict
these keypoints coordinates given the corresponding latent code. We compute
this model’s test loss on the real-world images from FFHQ $256^{2}$ and coin
this metric Keypoints Prediction Loss (KPL). The above procedure is formally
described in Algorithm 1 in Appendix B. The corresponding latent codes for the
real-world images are obtained by projecting an image into the corresponding
latent space through gradient descent optimization. We use the standard
protocol from [39] to do this for both models. KPL is computed for both
$\mathcal{Z}$-space and $\mathcal{W}$-space. For our model, this corresponds
to $\mathsf{G}$ input noise vectors $\bm{z}$ and penultimate hidden
representations (i.e. hidden representations before the output projection into
INR parameter space). For StyleGAN2, this corresponds to the mapping network’s
input and output space. We also compute KPL on top of random vectors to check
that a better performance in keypoints prediction is not due to their reduced
variability. The results of this experiment are provided in Table 3.

Table 3: Keypoints Prediction Loss (KPL) metric for our INR-based decoder and
StyleGAN2 generator computed on FFHQ $256\times 256$ dataset for different
latent spaces. The qualitative difference is illustrated on fig:keypoints-
prediction.

Generator | KPL $(\mathcal{Z})$ | KPL $(\mathcal{W})$ | KPL (random)
---|---|---|---
StyleGAN2 | $2.1\cdot 10^{-4}$ | $7.6\cdot 10^{-5}$ | $4.6\cdot 10^{-4}$
INR-GAN | $6.0\cdot 10^{-5}$ | $5.2\cdot 10^{-5}$ | $4.7\cdot 10^{-4}$

Accelerated inference of lower-resolution images. Our INR-GAN has a natural
capability of generating a low-resolution sample faster because it can be
directly evaluated at a low-resolution coordinates grid without performing the
full generation. However, traditional convolutional decoders lack this
property: to produce a lower-resolution image, one would need to perform the
full inference and then downsample the resulted image with standard
interpolation techniques. To state the claim rigorously, we compute an amount
of multiply-accumulate (MAC) operations for our INR-based generator and
StyleGAN2 generator trained on FFHQ $1024^{2}$ for different lower-resolution
image sizes. To produce the low-resolution image with StyleGAN2, we first
produce the full-resolution image and them downsample it with nearest
neighbour interpolation. For our model, we just evaluate it on a grid of the
given resolution. The results are reported on fig:fast-lowres-generation. To
the best of our knowledge, our work is the first one that explores the
accelerated generation of lower-resolution images — an analog of early-exit
strategies [81] for classifier models for image generation task.

Figure 9: Accelerated low-resolution image generation. We measure a decoder’s
efficiency in terms of #MACs on generating an image of lower resolution
compared to what it has been trained on. Since INR can do this by evaluating
on a sparser grid, this allows it to save a lot of computation. Traditional
convolutional decoders require performing a full inference first and then
downsampling the produced image.

Out-of-the-box superresolution. Our INR-based decoder is able to produce
images of higher resolution than it was trained on. For this, we evaluate our
model on a denser coordinates grid. To measure this quantitatively, we propose
UpsampledFID: a variant of FID score where fake data statistics are computed
based on upsampled low-resolution images. We compare our UpsampledFID score to
three standard upsampling techniques: nearest neighbor, bilinear, and bicubic
interpolation. fig:superresolution demonstrates that INR-upsampling improves
the score by up to 50%.

##  5 Additional potential of INR-based decoders

In Section 4.2, we explored several exciting properties of INR-based decoders
and tested them in practice. Here, we highlight their additional potential and
leave its exploration for future work.

An ability to backpropagate through pixels positions. Since coordinate
positions $(x,y)$ are transformed via well-differentiable operations, it gives
a possibility to back-propagate through coordinate positions. This opens a
large range of possible work like using spatial transformer layers [34] in the
generator and not only discriminator. Or producing a coordinates grid with the
discriminator for zooming-in into specific parts of an image.

Faster inference speed. Since INRs do not use local context information during
inference, it does not spend computation on aggregating it as convolutions do.
This is equivalent to using convolutions with a kernel size of 1 and thus
works much faster [29]. Table 1 demonstrates that our model uses three times
fewer MACs for its inference process, which is due to ignoring context
information.

Parallel pixel computation Non-autoregressive models like Parallel WaveNet
[64, 24] are valued by their ability to generate an arbitrarily long sequence
in parallel, so their inference speed decreases linearly with more compute
being added. INR-based decoders also have this property due to their
independent pixel generation nature. Convolutional decoders, in contrast, are
forced to use local context during the inference process, which limits their
parallel inference.

Universal decoder architecture One can employ the same decoder architecture
for training a decoder model in any domain: 2D images, 3D shapes, video,
audio, etc. They would only differ at what coordinates are being passed as
input to the corresponding INR. For 2D images, these would be
$(x,y)\mathbb{R}^{2}$ coordinates, for 2D video, this would be
$(x,y,t)\mathbb{R}^{2}\times\mathbb{R}_{+}$ with the additional timestep
$t\in\mathbb{R}_{+}$ input, for 3D shapes — $(x,y,z)\in\mathbb{R}^{3}$
coordinates, etc. That has already been partially explored in [50, 73].

Biological plausbility. While convolutional encoders have a very intimate
connection to how a human eye works [46], convolutional decoders do not have
much resemblance to any human brain mechanism. In contrast, [71] argue that
hypernetworks have a very close relation to how a prefrontal cortex influences
other brain parts. It does so by modulating the activity in several different
areas at once, precisely how our hypernetwork-based generator applies
modulation to different INR layers.

##  6 Limitations

The core limitation of INR-based decoders comes from their strengths: not
using spatially common context between neighboring pixels. Multi-Scale INR
architecture partially alleviates this issue by grounding them on the same
low-resolution representation, but as our experiment with adding bilinear
interpolation demonstrates (see Table 2), it is not enough. Also, we noticed
that the INR-based decoders might become too sensitive to high-frequency
coordinates features. This may produce “texture” artifacts: a generated image
having a random transparent texture spanned on it which becomes more
noticeable for higher resolutions.

##  7 Conclusion

This paper explored the adversarial generation of continuous images
represented in the implicit neural representations (INRs) form. We proposed
two principled architectural techniques: factorized multiplicative modulation
and multi-scale INRs, which allowed us to obtain solid state-of-the-art
results in continuous image generation. We explored several attractive
properties of INR-based decoders and discussed their future potential and
limitations.

## References

  * [1] Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov.  Image generators with conditionally-independent pixel synthesis.  arXiv preprint arXiv:2011.13775, 2020.
  * [2] Martín Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In ICML, 2017.
  * [3] Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger.  Explicitly disentangling image content from translation and rotation with spatial-vae.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 15409–15419. Curran Associates, Inc., 2019.
  * [4] Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi.  Learning feed-forward one-shot learners.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 523–531. Curran Associates, Inc., 2016.
  * [5] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2018.
  * [6] Adrian Bulat and Georgios Tzimiropoulos.  Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans.  arXiv preprint arXiv:1712.02765, 2017.
  * [7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  arXiv preprint arXiv:2012.00926, 2020.
  * [8] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [9] Oscar Chang, Lampros Flokas, and Hod Lipson.  Principled weight initialization for hypernetworks.  In International Conference on Learning Representations, 2020.
  * [10] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu.  Dynamic convolution: Attention over convolution kernels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [11] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [12] Flavio Chierichetti, Sreenivas Gollapudi, Ravi Kumar, Silvio Lattanzi, Rina Panigrahy, and David P Woodruff.  Algorithms for $\ell\\_p$ low rank approximation.  arXiv preprint arXiv:1705.06730, 2017.
  * [13] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In International conference on medical image computing and computer-assisted intervention, pages 424–432. Springer, 2016.
  * [14] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio.  Feature-wise transformations.  Distill, 2018.  https://distill.pub/2018/feature-wise-transformations.
  * [15] Elad Eban, Yair Movshovitz-Attias, Hao Wu, Mark Sandler, Andrew Poon, Yerlan Idelbayev, and Miguel A. Carreira-Perpinan.  Structured multi-hashing for model compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [16] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE international conference on computer vision, pages 6202–6211, 2019.
  * [17] Tomer Galanti and Lior Wolf.  Comparing the parameter complexity of hypernetworks and the embedding-based alternative.  arXiv preprint arXiv:2002.10006, 2020.
  * [18] Tomer Galanti and Lior Wolf.  On the modularity of hypernetworks.  Advances in Neural Information Processing Systems, 33, 2020.
  * [19] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243–1252, 2017.
  * [20] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Deep structured implicit functions.  arXiv preprint arXiv:1912.06126, 2019.
  * [21] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.  Learning shape templates with structured implicit functions.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7154–7164, 2019.
  * [22] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * [23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [24] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher.  Non-autoregressive neural machine translation.  arXiv preprint arXiv:1711.02281, 2017.
  * [25] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In Advances in neural information processing systems, pages 5767–5777, 2017.
  * [26] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  arXiv preprint arXiv:1609.09106, 2016.
  * [27] Christian Henning, Johannes von Oswald, João Sacramento, Simone Carlo Surace, Jean-Pascal Pfister, and Benjamin F Grewe.  Approximating the predictive distribution via adversarially-trained hypernetworks.  2018 NeurIPS Workshop on Bayesian Deep Learning, 2018.
  * [28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [29] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [30] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [31] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville.  Neural autoregressive flows.  In International Conference on Machine Learning, pages 2078–2087, 2018.
  * [32] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510, 2017.
  * [33] Yerlan Idelbayev and Miguel A. Carreira-Perpinan.  Low-rank compression of neural nets: Learning the rank of each layer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [34] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In Advances in neural information processing systems, pages 2017–2025, 2015.
  * [35] Timothy Jeruzalski, Boyang Deng, Mohammad Norouzi, John P Lewis, Geoffrey Hinton, and Andrea Tagliasacchi.  Nasa: Neural articulated shape approximation.  arXiv preprint arXiv:1912.03207, 2019.
  * [36] Animesh Karnewar and Oliver Wang.  Msg-gan: Multi-scale gradients for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7799–7808, 2020.
  * [37] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [38] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [39] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [40] Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, and Siyu Tang.  Grasping field: Learning implicit representations for human grasps.  arXiv preprint arXiv:2008.04451, 2020.
  * [41] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  In Advances in neural information processing systems, pages 4743–4751, 2016.
  * [42] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [43] Marian Kleineberg, Matthias Fey, and Frank Weichert.  Adversarial generation of continuous implicit shape representations.  arXiv preprint arXiv:2002.00349, 2020.
  * [44] Sylwester Klocek, Łukasz Maziarka, Maciej Wołczyk, Jacek Tabor, Jakub Nowak, and Marek Śmieja.  Hypernetwork functional image representation.  In International Conference on Artificial Neural Networks, pages 496–510. Springer, 2019.
  * [45] Y. LECUN.  The mnist database of handwritten digits.  http://yann.lecun.com/exdb/mnist/, 1998.
  * [46] Yann LeCun, Yoshua Bengio, et al.  Convolutional networks for images, speech, and time series.  The handbook of brain theory and neural networks, 1995.
  * [47] Jae Hyun Lim and Jong Chul Ye.  Geometric gan.  arXiv preprint arXiv:1705.02894, 2017.
  * [48] Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chen.  Coco-gan: Generation by parts via conditional coordinating.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [49] Etai Littwin, Tomer Galanti, and Lior Wolf.  On the optimization dynamics of wide hypernetworks.  arXiv preprint arXiv:2003.12193, 2020.
  * [50] Gidi Littwin and Lior Wolf.  Deep meta functionals for shape representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [51] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski.  An intriguing failing of convolutional neural networks and the coordconv solution.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9605–9616. Curran Associates, Inc., 2018.
  * [52] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [53] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  Which training methods for gans do actually converge?  In International Conference on Machine Learning (ICML), 2018.
  * [54] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [55] Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [56] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  arXiv preprint arXiv:2003.08934, 2020.
  * [57] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In International Conference on Learning Representations, 2018.
  * [58] Phuoc Nguyen, Truyen Tran, Sunil Gupta, Santu Rana, Hieu-Chi Dam, and Svetha Venkatesh.  Hypervae: A minimum description length variational hyper-encoding network.  arXiv preprint arXiv:2005.08482, 2020.
  * [59] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5379–5389, 2019.
  * [60] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov.  Tensorizing neural networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 442–450. Curran Associates, Inc., 2015.
  * [61] Augustus Odena, Vincent Dumoulin, and Chris Olah.  Deconvolution and checkerboard artifacts.  Distill, 2016.
  * [62] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4531–4540, 2019.
  * [63] Geunseob Oh and Jean-Sebastien Valois.  Hcnaf: Hyper-conditioned neural autoregressive flow and its application for probabilistic occupancy map forecasting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [64] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al.  Parallel wavenet: Fast high-fidelity speech synthesis.  In International conference on machine learning, pages 3918–3926. PMLR, 2018.
  * [65] George Papamakarios, Theo Pavlakou, and Iain Murray.  Masked autoregressive flow for density estimation.  In Advances in Neural Information Processing Systems, pages 2338–2347, 2017.
  * [66] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  * [67] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [68] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In European Conference on Computer Vision (ECCV), 2020.
  * [69] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [70] Neale Ratzlaff and Li Fuxin.  HyperGAN: A generative model for diverse, performant neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, pages 5361–5369, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  * [71] Jacob Russin, Randall O’Reilly, and Yoshua Bengio.  Deep learning needs a prefrontal cortex.  Bridging AI and Cognitive Science ICLR 2020 Workshop, 2020.
  * [72] Laurent Sifre and Stéphane Mallat.  Rigid-motion scattering for image classification.  P.h.D. thesis, 2014.
  * [73] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In Proc. NeurIPS, 2020.
  * [74] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  * [75] Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 1121–1132. Curran Associates, Inc., 2019.
  * [76] Chen Song, Jiaru Song, and Qixing Huang.  Hybridpose: 6d object pose estimation under hybrid representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [77] Kenneth O Stanley.  Compositional pattern producing networks: A novel abstraction of development.  Genetic programming and evolvable machines, 8(2):131–162, 2007\.
  * [78] Joseph Suarez.  Character-level language modeling with recurrent highway hypernetworks.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 3269–3278, 2017.
  * [79] Joseph Suarez.  Language modeling with recurrent highway hypernetworks.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3267–3276. Curran Associates, Inc., 2017.
  * [80] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  NeurIPS, 2020.
  * [81] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung.  Branchynet: Fast inference via early exiting from deep neural networks.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469. IEEE, 2016.
  * [82] Kenya Ukai, Takashi Matsubara, and Kuniaki Uehara.  Hypernetwork-based implicit posterior estimation and model averaging of cnn.  In Asian Conference on Machine Learning, pages 176–191, 2018.
  * [83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [84] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe.  Continual learning with hypernetworks.  In International Conference on Learning Representations, 2020.
  * [85] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.  Solov2: Dynamic, faster and stronger.  arXiv preprint arXiv:2003.10152, 2020.
  * [86] Zhenyi Wang and Olga Veksler.  Location augmentation for cnn.  arXiv preprint arXiv:1807.07044, 2018.
  * [87] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner.  Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes.  arXiv preprint arXiv:1901.07017, 2019.
  * [88] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [89] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao.  On compressing deep models by low rank and sparse decomposition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7370–7379, 2017.
  * [90] K. Zafeirouli, A. Dimou, A. Axenopoulos, and P. Daras.  Efficient, lightweight, coordinate-based network for image super resolution.  In 2019 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC), pages 1–9, 2019.
  * [91] Chris Zhang, Mengye Ren, and Raquel Urtasun.  Graph hypernetworks for neural architecture search.  In International Conference on Learning Representations, 2019.
  * [92] Zhengxia Zou.  Castle in the sky: Dynamic sky replacement and harmonization in videos.  arXiv preprint arXiv:2010.11800, 2020.

##  Appendix A Additional implementation details

We build on top of the StyleGAN2 framework [39] and change only its generator.
All other settings, including the discriminator architecture $\mathsf{D}$,
optimizers, losses, training settings and other hyperparameters are kept
untouched. This means that we use non-saturating logistic loss for training
[23] and use Adam optimizers with the parameters
$\beta_{1}=0.0,\beta_{2}=0.98$ and $\epsilon=1e-8$. Our $\mathsf{D}$ is a
small version of StyleGAN2 discriminator (i.e. config-e from the paper [39]),
regularized with zero-centered $R_{1}$ gradient penalty [53] with weight
$\gamma=10$. We apply the regularization on each iteration instead of using
the lazy setup, as done by StyleGAN2[39] who applies it only each 16-th
iteration. We use the learning rate of 0.00001 for $\mathsf{G}$, 0.0005 for
the shared parameters of an INR (which is a part of $\mathsf{G}$) and 0.003
for $\mathsf{D}$. We also employ skip-connections for coordinates inside each
multi-scale INR block [36]. We apply them by concatenating the coordinates to
inner representations.

For the main experiments, we didn’t employ any ProGAN [37], StyleGAN [38] or
StyleGAN2 [39] training tricks, like path regularization, progressive growing,
equalized learning rate, noise injection, pixel normalization, style mixing,
etc. For our additional ablations, we reimplemented our INR-based decoder on
top of the StyleGAN2’s generator and employed style mixing, equalized learning
rate and pixel normalization for it. In terms of implementation, it was
equivalent to replacing StyleGAN2’s weight modulation-demodulation with our
FMM mechanism, reducing kernel size from 3 to 1, concatenating coordinates
information at each block and replacing its upfirdn2d upsampling with the
nearest neighbour one. We found that for the nearest neighbour upsampling, the
model learns to ignore spatial noise injection (by setting noise strengths to
zero), because pixels cannot communicate the noise information between each
other, making it meaningless and harmful to the generation process.

For our $256\times 256$ experiments (for both LSUN and FFHQ), we use 2 multi-
scale INR blocks of resolutions 128 and 256. Each block contains 4 layers of
512 dimensions each. For FFHQ $1024\times 1024$, we used 4 multi-scale INR
blocks of resolutions 128, 256, 512 and 1024. First 2 blocks had 3 layers of
dimensionality of 512, 512 resolution had 2 layers of resolution 128, the
final block had 2 layers of dimensionality of 32.

Our $\mathsf{G}$ architecture is the same for all the experiments and consists
on 3 non-linear layers with the hidden dimension of 1024 and residual
connections. Our noise vector $\bm{z}$ has the dimensionality of 512.

For additional implementation details, we refer a reader to the accompanying
source code.

In all the experiments, we compute FID scores based on 50k images using the
tensorflow script from BigGAN pytorch
repo111https://github.com/ajbrock/BigGAN-
PyTorch/blob/master/inception_tf13.py.

##  Appendix B Experiments details

###  B.1 Zooming out

On fig:zoom:extra we present additional examples of our zooming operation, but
evaluating the model on $[-1.5,1.5]^{2}$ grid instead of $[-0.3,1.3]^{2}$ like
we did for fig:oob-generalization in the main body. This is done to
demonstrate the extent to which the model can extrapolate.

Figure 10: After training our INR-based GAN on LSUN $256\times 256$ dataset,
we feed larger coordinates grid $[-0.5,1.5]^{2}$ into it. This is larger than
on Figure 2 to demonstrate the extent to which the model extrapolates. As one
can see, extending the grid outside of $[-0.4,1.4]^{2}$ makes the quality to
decrease rapidly.

###  B.2 Keypoints prediction

As being said, we train a linear model to predict the keypoints from the
latent codes. To train such a model, we first generate $n=10^{4}$ latent codes
$\bm{w}_{1},...,\bm{w}_{n}$ for each model, then we decode them into images
$\bm{x}_{1},...,\bm{x}_{n}$. After that, we predict keypoints vector
$\bm{y}_{i}$ for each image with Super-FAN model [6]. Then we fit a linear
regression model to predict $\bm{y}_{i}$ from $\bm{w}_{i}$.

Measuring quality based on the synthesized images may be unfair since the
variability in keypoints of each model can be different: imagine a generator
that always produces a face with the same keypoints. This is why we compute
the test quality by embedding real FFHQ images into each model. But to
additionally demonstrate that the variability is equal for the both models, we
fit a linear regression model on randomly permuted latent codes and check its
score: if the prediction accuracy is high, than the variability in keypoints
is low and the prediction task is much easier. We call this metric KPL
(random) and depict the corresponding values on Table 3. It clearly
demonstrates that the both models have equal variability in terms of
keypoints.

We project FFHQ images into a latent space using the latent space projection
procedure from the official
repo222https://github.com/NVlabs/stylegan2/blob/master/projector.py. We used
default hyperparameters except for it, except that we didn’t optimize the
injected noise since this would take away some of the information that is
better to be stored in the latent code. We depict additional qualitative
results for random samples of FFHQ on fig:keypoints-extra.

We provide the algorithm on KPL computation in Algorithm 1. We use
$N_{\text{tr}}=10^{4}$ and $N_{\text{ts}}=256$.

(a) StyleGAN2 generator

(b) Our INR-based generator

Figure 11: Predicting keypoints from latent codes for random FFHQ images. The
corresponding scores are presented in Table 3.

(a) INR-GAN latent space projections.

(b) StyleGAN2 latent space projections.

Figure 12: Projection results by projection images from 11. We use the
original StyleGAN2’s projection procedure [39] to project FFHQ dataset images
into the latent space of a generator. All low-frequency details, together with
the keypoints are reconstructed well. In our case, the reconstruction quality
is lower because we do not optimize for spatial noise as StyleGAN2 does
because our vanilla INR-GAN architecture does not use spatial noise injection.

Input : Keypoints extractor
$\mathcal{K}:\bm{x}\mapsto\bm{k}\in\mathbb{R}^{d_{k}}$.

Input : Generator model $\mathsf{G}:\bm{w}\mapsto\bm{x}$

Input : Embedding procedure $\mathcal{E}:\bm{x}\mapsto\bm{w}$.

Input : Collection of real face images
$X_{\text{test}}=\\{\bm{x}_{i}\\}_{i=1}^{n}$ of size $N_{\text{ts}}$.

Output : KPL score $s\in[0,+\infty)$.

Generate $N_{\text{tr}}$ latent codes
$W_{\text{train}}=\\{\bm{w}_{1},...,\bm{w}_{N_{\text{tr}}}\\}$;

Generate a dataset of synthetic images
$X_{\text{train}}=\\{\mathsf{G}(\bm{w}_{i})|\bm{w}_{i}\in
W_{\text{train}}\\}$;

Extract keypoints
$K_{\text{train}}=\\{\mathcal{K}(\bm{x}_{i})~{}|~{}\bm{x}_{i}\in
X_{\text{train}}\\}$ and
$K_{\text{test}}=\\{\mathcal{K}(\bm{x}_{j})~{}|~{}\bm{x}_{j}\in
X_{\text{test}}\\}$;

Embed real images
$W_{\text{test}}=\\{\mathcal{E}(\bm{x}_{j})~{}|~{}\bm{x}_{j}\in
X_{\text{test}}\\}$;

Train a linear keypoints estimator
$(\bm{A}^{*},\bm{b}^{*})=\arg\min_{\bm{A},\bm{b}}\sum_{i=1}^{N_{\text{tr}}}\|(\bm{A}\bm{w}_{i}+\bm{b})-\bm{k}_{i}\|_{2}^{2}$;

Evaluate its performance on the test set:
$s=\sum_{i=1}^{N_{\text{ts}}}\|(\bm{A}\bm{w}_{j}+\bm{b})-\bm{k}_{j}\|_{2}^{2}$;

Return $s$;

Algorithm 1 Compute Keypoints Prediction Loss (KPL).

###  B.3 Additional samples

On fig:ffhq1024-samples, we present additional samples with the truncation
factor of 0.9 from our INR-based model trained on FFHQ1024. We perform the
truncation in similar nature to StyleGAN2 [39] by linearly interpolating an
inner representation inside $\mathsf{G}$ to its averaged value. On
fig:ffhq1024-artifacts, we present common artifacts found in the produced
images. On fig:superresolution:extra, we present additional superresolution
samples from our model trained on LSUN $128^{2}$. On fig:lsun256, we present
additional uncurated samples of our model trained on LSUN bedroom $256^{2}$.

Figure 13: Random (uncurated) samples from our model trained on FFHQ
$1024\times 1024$ dataset with the truncation factor of 0.9. FID: 16.32

Figure 14: Common artifacts found in our model’s samples when sampling without
truncation. As one can see, the most severe ones are “stains” and patterned
texture.

(a) Original

(b) Nearest neighbour

(c) Bilinear

(d) Bicubic

(e) INR-based

Figure 16: Additional samples from our model to show superresolution
properties. We trained the model on LSUN $128\times 128$ and upsampled to
$256\times 256$. Figure 17: Random samples of our model on LSUN bedroom
$256^{2}$ dataset. FID: 6.27.

##  Appendix C Positional encoding of coordinates

Recent works [73, 80] demonstrate that using positional embeddings [83] like
Fourier features greatly increases the expressivity of a model, allowing to
fit more complex data. Our positional encoding of coordinates follows [80]
design and consists on a linear matrix $\bm{W}\in\mathbb{R}^{n\times 2}$
applied to raw coordinates vector $\bm{p}=(x,y)$ and followed by sine/cosine
non-linearities and concatenated:

$\begin{split}\bm{e}(\bm{p})=\begin{bmatrix}\sin(\bm{W}\bm{p})\\\
\cos(\bm{W}\bm{p})^{\top}\end{bmatrix}\end{split}$ (1)

Matrix $\bm{W}$ is produced by our generator $\mathsf{G}$ without any
factorization since it has only 2 columns. Each row of this matrix corresponds
to the parameters of the Fourier transform. The norm of a row corresponds to
the frequency of the corresponding wave. We depict frequencies distribution
learned by our generator on Figure 18.

Figure 18: Frequencies distributions for different multi-scale INR blocks of
our INR-based GAN trained on FFHQ $1024^{2}$. To produce the plot, we sample
128 images in an INR-based form and computed the norms of the positional
encoding layers, i.e. those layers which take raw coordinates as an input. As
one can see, the model tries to use more high-frequent positional embeddings
for the last layer since they are more important for drawing fine-grained
details. Early layers determine the structure of an image and hence use
smaller frequencies to operate on a larger scale.

##  Appendix D Geometric prior

Adding coordinates to network input induces powerful prior on the geometric
shapes, since now different pixels, otherwise created equal are ordered
through the euclidean (or any other) coordinate system. It is a well-known
fact that complex geometric shapes could be compactly represented with the use
of euclidean coordinates (for example one can write down an ellipse equation
as $\frac{(x-x_{0})^{2}}{a^{2}}+\frac{(y-y_{0})^{2}}{b^{2}}=1$ ). On the other
hand without any form of prior one would hope to fit complex patterns with
dedicated filters which would potentially consume much more parameters.

It is worthy to discuss the synergy between the coordinate representations and
hypernetworks. Lets imagine a learnable system consisting of sequentially
connected linear layer $W\in\mathbb{R}^{2\times 2}$, which is modulated by a
hypernetwork, and INR, taking Euclidean coordinates as an input
$f(X),X\in\mathbb{R}^{2\times 1}$ The whole model then could be written as
$f(W(z)\times X)$. Now, it could be seen, that introducing the hypernetwork to
the pipeline allows to apply linear transformation to the coordinates,
rotating and zooming the image encoded by INR $f$. Thus, hypernetworks allow
to easily perform transformations non-trivial for traditional deep learning
systems.

Finer control over form and placement Recent studies show that convolutional
NN struggle with such simple and crucial tasks as accurately predicting the
coordinates of a drawn point [51] (and vice versa, drawing a point given the
coordinates). One should expect, that such an important skill is necessary for
the generative model for accurate placement of the different object parts and
for precise representation of the object proportions.

##  Appendix E FMM as a generalizaton of the common weight modulation schemes

In this section we show that Factorized Matrix Multiplication could be seen as
a general framework for weight modulation, with Squeeze-and-Excitation, AdaIN
and ”vanilla” hypernetworks as its particular cases.

Squeeze-and-Excitation Let us look at the $l$-th FMM layer of our network with
the effective rank of 1. In this case $\bm{A}^{l}$ and $\bm{B}^{l}$ are
matrices (actually vectors) of the sizes $n_{in}\times 1$ and $1\times
n_{out}$ respectively. Thus, following the rules of matrix multiplication, we
get that $\bm{W}_{h\,i,j}^{l}=\bm{A}^{l}_{i}\cdot\bm{B}^{l}_{j}$. On the other
hand, lets look at the Squeeze-and-Excitation mechanism. Here we are
modulating the output of the each neuron by multiplicating it by the predicted
coefficient, which is equivalent to the multiplication of the corresponding
weight matrix column by this coefficient. Using our notions and denoting the
pre-activation (before non-linearity) vector of modulation coefficients as
$\bm{A}$ we get that in this case $\bm{W}_{h\,i,j}^{l}=\bm{A}^{l}_{i}$. While
at the first glance it looks like our model is more expressive lets not forget
about the fact that the next layer is by itself modulated with its own
Squeeze-and-Excitation, from which (omitting relu non-linearity and the fact
that it has its own sigmoid) we can get the $\bm{B}^{l}_{j}$ multiplier. Thus
it could be seen that squeeze-and-excitation modulation is roughly equivalent
to the FMM layer with the rank of 1. The same reasoning is applicable to the
AdaIN case, though, with AdaIN obviously we have the additional normalization
layer and do not have sigmoid non-linearity for the style vector which
influences the learning dynamics in its own way. We can say that squeeze-and-
excitation is the least powerful weight modulation scheme, which uses the
matrix of the rank 1 to modulate the main shared weights, on the other hand it
is cheap and simple.

Hypernetworks Vanilla hypernetwork is perhaps the most straightforward (and
the most expensive but flexible) approach to the weight modulation. It is as
simple as predicting each weight as an output of MLP. So let’s demonstrate
that any weight dynamic that can be modeled by hypernetwork could be fitted
with the FMM of high enough rank. Let’s assume that $n_{in}$ is larger than
$n_{out}$ (which is our case, but not essential for the generality of the
proof) and choose the FMM rank of $n_{in}$. In this case $\bm{A}^{l}$ and
$\bm{B}^{l}$ are matrices of the sizes $n_{in}\times n_{in}$ and $n_{in}\times
n_{out}$ respectively. Since $A$ is a square matrix we can set it to identity
constant (which is a solution easily learnt by a NN just by setting bias) and
get $\bm{W}_{h\,i,j}^{l}=\bm{B}^{l}_{i,j}$. In this case any hypernetwork
could be ”simulated” with FMM by fitting
$\bm{B}^{l}_{i,j}=\sigma^{-1}(\frac{\bm{\hat{W}}^{l}_{i,j}}{\bm{W}^{l}_{s\,i,j}})$,
where $\bm{\hat{W}}$ denotes the weight matrix predicted by the hypernetwork.
While $\sigma^{-1}$ definitely imposes some restrictions, caused by the
positiveness and the range, in our experiments adding sigmoid has not resulted
in any harm, perhaps because of the flexible calibration of the
$\bm{W}^{l}_{s}$.

We have shown that main weight modulation schemes could be seen as the
boundary particular cases of our approach. Our approach to the weight
modulation is somewhere in between of these two extremes, while reaping the
benefits of the both of them. Studying the behaviour of this transmission is
specially important for the shading light on the weight modulation at whole
and going beyond straightforward approaches.

##  Appendix F Performance on multi-class datasets

In this section, we conduct experiments on two diverse datasets to demonstrate
that our proposed architectural design improves performance in this scenario
is well. For this, we employ two datasets: LSUN-10 $256^{2}$ and
MiniImageNet-100 $128^{2}$. LSUN-10 consists on 1M images of 10 LSUN scenes,
where we take 100k images of each scene. MiniImageNet-100 consists on 100k
images of 100 ImageNet classes333https://github.com/yaoyao-liu/mini-imagenet-
tools, where each class provides 1k images. We report the results for
different models in Table 4. They demonstrate that our proposed architectural
design improves the performance for this setup as well.

Table 4: FID & IS at 300k iterations on multi-class datasets. Decoder type | LSUN-10 | MiniImageNet
---|---|---
FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$
Basic INR decoder | 216.8 | 1.0 | 271.5 | 1.03
\+ Hypernetwork-based decoder | OOM | OOM | 112.9 | 8.76
\+ Fourier embeddings | OOM | OOM | 102.8 | 9.85
\+ FMM | 23.78 | 2.48 | 84.66 | 9.32
\+ Multi-scale INR | 12.47 | 3.02 | 59.63 | 11.29
StyleGAN2 | 8.99 | 3.18 | 52.94 | 12.32
Validation set | 0.42 | 9.93 | 0.39 | 61.79