# Convolutional Neural Opacity Radiance Fields

Haimin Luo, Anpei Chen, Qixuan Zhang, Bai Pang, Minye Wu, Lan Xu, and Jingyi
Yu,  H. Luo, Q. Zhang, B. Pang, L. Xu and J. Yu are with the School of
Information Science and Technology, ShanghaiTech University, Shanghai 201210,
China. E-mail:{luohm, zhangqx1, pangbai, xulan1,
<EMAIL_ADDRESS>A. Chen and M. Wu are with the School of
Information Science and Technology, ShanghaiTech University, Shanghai 201210,
China, and the Shanghai Institute of Microsystem and Information Technology,
Chinese Academy of Sciences, Shanghai 200031, China, and also with the
University of Chinese Academy of Sciences, Beijing 100049, China.
E-mail:{chenap<EMAIL_ADDRESS>Yu is with the Shanghai
Engineering Research Center of Intelligent Vision and Imaging, School of
Information Science and Technology, ShanghaiTech University, Shanghai 201210,
China. E-mail<EMAIL_ADDRESS>
###### Abstract

Photo-realistic modeling and rendering of fuzzy objects with complex opacity
are critical for numerous immersive VR/AR applications, but it suffers from
strong view-dependent brightness, color. In this paper, we propose a novel
scheme to generate opacity radiance fields with a convolutional neural
renderer for fuzzy objects, which is the first to combine both explicit
opacity supervision and convolutional mechanism into the neural radiance field
framework so as to enable high-quality appearance and global consistent alpha
mattes generation in arbitrary novel views. More specifically, we propose an
efficient sampling strategy along with both the camera rays and image plane,
which enables efficient radiance field sampling and learning in a patch-wise
manner, as well as a novel volumetric feature integration scheme that
generates per-patch hybrid feature embeddings to reconstruct the view-
consistent fine-detailed appearance and opacity output. We further adopt a
patch-wise adversarial training scheme to preserve both high-frequency
appearance and opacity details in a self-supervised framework. We also
introduce an effective multi-view image capture system to capture high-quality
color and alpha maps for challenging fuzzy objects. Extensive experiments on
existing and our new challenging fuzzy object dataset demonstrate that our
method achieves photo-realistic, globally consistent, and fined detailed
appearance and opacity free-viewpoint rendering for various fuzzy objects.

###### Index Terms:

Computational Photography, Neural Rendering, Opacity Modelling, View
Synthesis.

##  1 Introduction

Figure 1: VR$/$AR experience with photo-realistic opacity radiance field
rendering (first column), our method is able to reconstruct fuzzy appearance
at different scales (2 - 5 columns).

The past ten years have witnessed a rapid development of 3D reconstruction
technologies for complex scenes with the popularity of commercial passive and
active image sensors, which enables numerous immersive experience and virtual
and augmented reality (VR and AR) applications and has recently attracted
substantive attention. However, the photo-realistic modeling of fuzzy objects
with complex opacity such as hair, fur and feathers remains unsolved, which
suffers from strong view-dependent brightness, color changes, leading to
difficulties in both geometry and appearance reconstruction.

For high-quality fuzzy object modeling, early solutions [1, 2, 3, 4] require
costly capture devices and systems, coded lighting, or even manually effort to
achieve high-fidelity hair strand geometry reconstruction, which is difficult
to be deployed for daily usage. To avoid the heavy reliance on precise
geometry modeling, researchers adopt image-based rendering (IBR) [5, 6, 7] to
reconstruct the appearance of furry objects by interpolating new views from
the captured ones. Specifically, to handle opacity objects, the traditional
approach [6] utilizes multi-view images and alpha mattes to compute the
angular opacity maps in novel views, which suffers from severe ghosting effect
caused by insufficient view samples and inaccurate geometry proxies. Moreover,
obtaining accurate geometry of furry objects is intractable since fur and hair
contain tens of thousands of thin fibers and their mutual occlusions are the
fundamental causes of translucency.

Only recently, the neural rendering techniques [8, 9, 10, 11, 12] bring huge
potential for photo-realistic novel view synthesis from only images input,
with various data representations such as point-clouds [13, 11], voxels [9,
14], meshes [15, 10] or implicit representation [16, 17, 8, 12]. However, the
literature on fuzzy object neural rendering remains sparse. The recent
approach [11] utilizes a neural point renderer to generate texture maps and
alpha mattes in novel views explicitly. However, extracting features from a
coarse point cloud leads to insufficient sampling and severe artifacts when
zooming in and out. Besides, researchers [12, 18, 19, 20] combine implicit
representation with volume rendering to inherently model the density of the
continuous 3D space, achieving state-of-the-art appearance rendering results,
even for opacity objects. Though these methods achieve view-consistent fuzzy
object modeling, their method still lacks fine details in both texture and
alpha. Specifically, only using a Multi-Layer Perceptron (MLP) network is
adopted to fit the continuous 3D space, leading to uncanny high-frequency
appearance and opacity details. Most recently, Positional encoding [12] and
Fourier feature mapping [21] schemes enable the MLP to handle richer texture
details, but it still fails to recover fuzzy surface which contains extremely
high-frequency variation.

In this paper, we attack the above challenges and propose a novel scheme to
generate convolutional neural opacity radiance fields for fuzzy objects, which
is the first to combine explicit opacity supervision with neural radiance
field technique (See Fig. 1 for an overview). Our novel pipeline enables high-
quality appearance and global consistent alpha mattes generation in arbitrary
novel views for more immersive VR/AR applications.

More specifically, to provide explicit opacity supervision, from the system
side we introduce an effective multi-view capture system equipped with a step
turntable and specific transparent lighting design as well as a corresponding
multi-color keying algorithm. Our novel system automatically captures both
high-quality RGB images and corresponding opacity maps for challenging fuzzy
objects in the input capture views. Based on such hybrid input, from the
algorithm side, we introduce the convolutional mechanism in the image plane
into the neural radiance field framework [19] so that enable photo-realistic
appearance and global consistent opacity generation in arbitrary novel views.
To this end, we first propose an efficient sampling strategy that utilizes the
inherent silhouette prior along the rays and encodes the spatial information
across the image plane, which enables efficient radiance field sampling and
learning in a patch-wise manner. Then, we perform a novel volumetric
integration scheme to generate a per-patch hybrid appearance and opacity
feature maps, followed by a light-weight convolutional U-Net to reconstruct
the view-consistent fine-detailed appearance and opacity output. Moreover, a
patch-wise adversarial training scheme is proposed to preserve both high-
frequency appearance and opacity details for photo-realistic rendering in a
self-supervised framework. To summarize, our main contributions include:

  * •

We present a novel convolutional neural radiance field generation scheme to
reconstruct high frequency and global consistent appearance and opacity of
fuzzy objects in novel views, achieving significant superiority to the
existing state of the art.

  * •

To enable convolutional mechanism, we propose an efficient sampling strategy,
a hybrid feature integration as well as a self-supervised adversarial training
scheme for patch-wise radiance field learning.

  * •

We introduce an effective multi-view system to capture the color and alpha
maps for challenging fuzzy objects, and our capture dataset will be made
available to stimulate further research.

Figure 2: Overview of our end-to-end ConvNeRF pipeline. Given multi-view RGBA
images, we use an SFS to infer proxy geometric for Efficient Ray Sampling. For
each sample point in the volume space, the position and direction are feeding
to an MLP based feature prediction network to represent the object at a global
level. We next concatenate nearby rays into local feature patches and decoded
them into RGB and matte with the convolutional volume renderer. An adversarial
training strategy is used on the final output to encourage fine surface
details. In the reference period, we render the entire image at once rather
than per patch rendering.

##  2 Related Work

###  2.1 Neural 3D Shape Modeling

Recent work has made a significant process on 3D object modeling and realism
free-viewpoint rendering with level sets of deep networks that implicitly map
spacial locations $xyz$ to a geometric representation (i.g., distance field
[16], occupancy Field [17, 22] etc.). The aforementioned explicit
representations require discretization (e.g., in terms of the number of
voxels, points or vertices), implicitly models shapes with a continuous
function and naturally is able to handle complicated shape topologies. The
implicit geometric modeling can be quickly learned from 3D point samples [23,
24], and the trained models can be used to reconstruct shapes from a single
image or 3D part. However, these models are limited by their requirement of
access to ground truth 3D geometry, typically obtained from synthetic 3D shape
datasets such as ShapeNet [25]. Subsequent works relax this requirement by
formulating differentiable rendering functions that allow neural implicit
shape representations to be optimized using only 2D images [8, 26].

###  2.2 Free-Viewpoint Rendering

Free-viewpoint synthesis methods are generally model input/target images as a
collection of rays and essentially aims to recover the plenoptic function [27]
from the dense samples. Earlier Image-Based Rendering (IBR) work [28] used two
planes ($uvst$) parametrization or 2PP to represent rays and render new rays
via a weighted blending of the ray samples, i.e., fusing nearby rays by
considering view angle and camera spacial distance. They are able to achieve
real-time interpolation but require much memory as they need to cache all
rays. Following work [5] bring in proxy geometric to select suitable views and
filter occluded rays by cross-projection to the image plane when ray fusion.
However, those methods are still limited by the linear blending function,
leading to severe ghosting and blurring artifact.

Most recently, seminal researches seek to implicitly represent the radiance
field and render novel views with a neural network. Deep Surface Light Fields
[29] use an MLP network to fix per-vertex radiance and learns to fill up the
missing data across angles and vertices. Deferred neural rendering [10]
presents a novel learnable neural texture to model rendering as image
translation, which uses a coarse geometry for texture projection and offers
flexible content editing. Recent works [14, 19, 30] present a learned
representation that encodes the view-dependent appearance of a 3D scene
without modeling its geometry explicitly. Another line of research extends the
free-viewpoint to animation squeezing or scene relighting by modeling and
rendering dynamic scenes through embedding spacial feature with sparse dynamic
point cloud [31], using volumetric representation to reconstruct dynamic
geometry and appearance variations jointly with only image-level supervision
[9], or modeling image formation in terms of environment lighting, object
intrinsic attributes and the light transport function [32]. A Notable
exception is NeRF [12], which implicitly models the radiance field and the
density of a volume with a neural network, then uses a direct volume rendering
function to synthesize novel views. They also demonstrate a heretofore
unprecedented level of fidelity on a range of challenging scenes. The
following work NeRF-W [19] relaxes the NeRF’s strict consistency assumptions
through modeling per-image appearance variations such as exposure, lighting,
weather, and post-processing with a learned low-dimensional latent space.
However, such pure ray-based rendering schemes are failing to recover high-
frequency surface, such as fur.

###  2.3 Image Matting

Traditional natural image matting algorithms usually rely on user-defined
trimap [33] or scribble [34] as additional input and can generally be divided
into sampling-based methods and propagation-based methods. Sampling based
methods [35, 36, 37, 38], the known foreground and background regions are
sampled as candidates to find the best pixel pair by a carefully designed
metric for estimating the alpha value of a query pixel in the unknown area. In
propagation-based methods, the alpha values in unknown regions are propagated
from the known foreground and background regions using a reformulated image
matting model according to different affinities between pixels, using various
propagation schemes [39, 40, 41, 42, 43].

Recently, learning-based methods have made an impressive process. Deep Image
Matting [44] builds a large matting dataset by alpha composition and proposes
an end-to-end deep learning framework to automatically compute alpha mattes
using RGB image and trimap as input. Following this work, different methods
have been proposed. Disentangled Image Matting [45] adapts the input trimap as
well as estimates alpha with a novel multi-task loss. HDMatting [46] estimate
the alpha matte of high resolution images patch by patch through a Cross-Patch
Contextual module guided by the given trimap. Recent methods not only estimate
the alpha matte but the foreground image [47] and the background image [48]
such that the popular perceptual loss [49, 47] or a fusion mechanism [48] can
be adopted. To achieve trimap-free matting, recent works implicitly generate
trimap using a late fusion model with a soft segmentation network [50] or
require additional background photos as input [51] for self-supervised
training.

Rather than recover alpha from a 2D image, we set out to generate free-
viewpoint alpha matte directly following 3D constraints.

##  3 Algorithm Details

In this section, we introduce the design of our convolutional neural opacity
radiance fields (ConvNeRF in short), which enables photo-realistic global-
consistent appearance and opacity rendering in novel views based on the RGBA
input from our capture system, as illustrated in Fig. 2.

Our key insight is to encode opacity information explicitly with a spatial
convolutional mechanism to enhance the neural radiance field approach NeRF
[12] for high-frequency detail modeling. Inspired by NeRF, we adopt the
similar implicit neural radiance field to represent a scene using a Multi-
Layer Perceptron (MLP), as well as the volumetric integration of predicted
density and color values along the casting rays. Please refer to NeRF [12] for
more details.

Differently, our ConvNeRF further encodes opacity explicitly with spatial
convolutional design to significantly improve the neural radiance field
reconstruction. To this end, we first propose an efficient sampling strategy
to not only utilize the inherent silhouette prior along the camera rays but
also encode the spatial information across the image plane (Sec. 3.1). Then, a
global geometric representor is adopted to map a 3D location to a high-level
radiance feature and then a novel volumetric integration scheme is adopted to
generate per-patch hybrid feature embeddings, which models the characteristics
of appearance and opacity separately for more effective radiance field
learning in a patch-wise manner (Sec. 3.2). Next, we choose a light-weight
U-Net to decode the feature patches into view-consistent appearance and
opacity output (Sec. 3.3). We further adopt a patch-wise adversarial training
scheme to preserve both high-frequency appearance and opacity details in a
self-supervised framework (Sec.3.4).

Figure 3: Feature patches construction. To enable 2D convolution, we weighted
blend the volume features of each ray as the ray feature with the ”cumproded”
density of the sample points and concatenate nearby ray features into local
patches for the following convolutional rendering.

###  3.1 Efficient Ray Sampling

To speed up the training procedure, we introduce a patch-wise sampling scheme,
which uses a coarse proxy to filter out redundant samples for more efficient
radiance field sampling and rendering.

Coarse Proxy Generation. Note that the multi-view alpha mattes encodes the
silhouette prior of the captured fuzzy object inherently, which provides a
reliable proxy to guide the sampling process in the continuous 3D space. To
this end, the input alpha mattes are binarized and dilated, and the Shape-
from-Silhouette (SfS) [52] algorithm is applied to obtain a coarse 3D proxy of
the fuzzy object.

Spatial Sampling. To further enable patch-wise spatial sampling across the
image plane, we first render two depth maps: the near-depth ( the first hit
point) and far-depth (the last hit point) for all training views by projecting
the proxy mesh to the image plane. Then, we uniformly divide the input images
and depth maps into $K\times K$ small patches without spacing and filter out
the patches outside the valid near/far-depth (i.e., pure background patches).

Ray-wise Sampling. Given the above sampled ray patches, we perform a similar
two-stage sampling scheme of the original NeRF [12] in a coarse-to-fine manner
for ray-wise radiance fields sampling to sample $N$ sampling points for each
ray of the ray patches. Differently, note that we have obtained near-far depth
priors for sampled rays, we perform the ray-wise sampling on the near-far
region only in both coarse and fine stage. Thus it requires only $1/8$ sample
points of NeRF at least and achieves much more efficient radiance field
training and rendering.

The sampled patches with tightly sampling points in valid regions are then
utilized for the following training process of the neural radiance field in a
patch-wise manner. In practice, to balance the memory requirement and sampling
effectiveness, during training $K$ is set to be 32 and we use 12 patches as a
batch with 64 samples per ray. It achieves about 4 times faster training and
about 10 times faster inference compared to the original NeRF [12] in general.

###  3.2 Patch-wise integration

Unlike NeRF predicting pixel color by integrating in a low-dimensional color
space, we instead perform a novel volumetric integration scheme to generate a
per-patch hybrid appearance and opacity feature maps to enable more effective
radiance field learning in a patch-wise manner.

In our ConvNeRF, for a given 3D location x and viewing direction $\bf d$, we
first extract its 3D features $(\textbf{f},\sigma)$ with a global geometric
representor $\bf E_{\Theta}$:

$(\textbf{f},\sigma)=\bf E_{\Theta}(\bf x,\bf d).$ (1)

Same as the original NeRF network architecture, $\bf E_{\Theta}$ is achieved
with an MLP block with optimizable weights $\bf\Theta$, we remove the last
layer of RGB color branch so that $\bf E_{\Theta}$ plays as a global feature
extractor and generates both radiance term $\bf f$ and the geometric term
$\sigma$ output.

Then for each ray of the patches, we sample $N$ points and extract their 3D
features to predict the foreground and background probability (i.e., the
alpha):

$\alpha_{i}=T_{i}(1-exp(-\sigma_{i}\delta_{i})),T_{i}=exp(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}),$
(2)

where $\delta$ is the distance between adjacent sampling points. The projected
feature for each patch pixel is obtained by an integration scheme alone the
ray:

$F_{c}=\sum_{i=1}^{N}\alpha_{i}\bf f_{i},$ (3)

As mentioned above, we decompose the free-viewpoint object rendering into
radiance and alpha components; thus, we expect to decouple those two
components when rendering. To this end, different from radiance feature
extraction which fuses the radiance term of each sampling point along a ray by
integration, we set out to encode the underlying depth prior provided by our
efficient ray sampling scheme by concatenating $\alpha$ of each sampling point
estimated by Eqn.2 together, i.e.,
$F_{d}=[\alpha_{1},\alpha_{1},\cdots,\alpha_{N}]$.

After that, we can obtain view-dependent radiance feature map
$\mathcal{F}_{c}$ and density feature map $\mathcal{F}_{d}$ for each patch for
the following convolutional rendering. Our novel patch-wise volumetric
integration scheme provides high-level appearance and opacity features that
enable high-quality RGB and alpha image synthesis.

###  3.3 Convolutional Volume Renderer

Based on the above per-patch features which encode the characteristic of
appearance and opacity, we introduce a convolutional volume renderer scheme
for view-consistent appearance and alpha matte rendering by utilizing the
spatial information, to address the issue that the original NeRF [19] fails to
recover the high-frequency details of fuzzy objects with a simple MLP.
Specifically, let $\bf G$ denote the network of our convolutional volume
renderer with parameters $\theta$, which predicts the texture image $\bf{F}$
with corresponding opacity map $\alpha$ via
$(\bf{F},\alpha)=G(\mathcal{F}_{c},\mathcal{F}_{d};\theta)$, as illustrated in
Fig. 2. Different with NeRF which renders radiance pre-multiplied by alpha,
$\bf F$ is just foreground image to avoid retaining radiance from the
background near boundary.

Note that we aim to achieve view consistent rendering, however, the local
receptive field of convolutional neural network (CNN) itself may introduce
view-inconsistency. Thus $\bf G$ is designed to be light-weight to replace the
last fully connect layer of the original NeRF network and decode the feature
maps into radiance and opacity. In this way, the global implicit
representation described in Sec. 3.2 plays a dominant role for view-
consistency.

As illustrated in Fig. 2, our volume renderer $\bf G$ consists of a radiance
branch and an opacity branch with similar U-Net architectures. Note that in
our radiance branch two downsample-upsample blocks are adopted to generate
fine-detailed texture output, while only a single downsample-upsample block is
utilized in our opacity branch. Such a sophisticated design is based on our
observation that alpha mattes are sensitive to low-level features such as
image gradients and are more suitable for a shallow network to preserve view-
consistent output.

Besides, since the input feature maps $\mathcal{F}_{c}$ and $\mathcal{F}_{d}$
suffers from incomplete integration near the boundary regions, which is
critical for fuzzy objects, we further adopt the gated convolution [53] in the
U-Net architectures for both branches, so as to enhance the denoising and
image completion capabilities of the network. For more detailed alpha matte
prediction, the texture output from the radiance branch is further
concatenated with density feature map $\mathcal{F}_{d}$ to form a hybrid input
of the opacity branch. Also, we accumulate the $\mathcal{F}_{d}$ per-pixel
first to form a coarse initial alpha matte, and the opacity branch predicts
the opacity residuals. Finally, the coarse matte and the residual one are
added to generate the final detailed alpha matte output.

To better encourage surface details (hairline, hair texture) and sharpness, we
propose to apply a GAN-based discriminator loss to the final image patch (as
shown in the right side of Fig. 2). To enable self-supervised adversarial
learning, we randomly sample patches from all the input multi-view images as
real samples of the discriminator, and the fake samples are generated using
the above volumetric render and the estimated alpha matte, which is formulated
as:

$\bf{I}_{fake}=\alpha\bf{F}+(1-\alpha)\bf{B},$ (4)

Where $\bf{F},\bf{B}$ denote the generated foreground texture and background
image, respectively We set $B=[1.0,1.0,1.0]$ as a white background in all
patches in our experiment.

###  3.4 Network Training

We train the convolutional volume renderer $\bf G$ with $3$ generation loss
and one discriminator loss. Recall that our model targets on high quality
free-viewpoint radiance rendering (RGB and its opacity) from given sparse
views RGBA images, we decomposite the prediction procedure into two layers
(the RGB and alpha layers) and final compose them together via the predicted
alpha layer. We first propose to use $L_{2}$ loss for the above two layers:

$\displaystyle\mathcal{L}_{C,\alpha}=\sum_{i=1}^{n}\|{\bf{I}}_{i}-{\tilde{\bf
I}}_{i}\|_{2}^{2}+\|{\bf{\alpha}}_{i}-{\tilde{\bf\alpha}}_{i}\|_{2}^{2}$ (5)

where $\bf\tilde{I}$ and $\tilde{\alpha}$ indicate the ground truth image and
alpha.

To encourage fine details and let outputs close to ground truth patches at the
high level, We use a VGG19 perceptual loss [49] on the output feature maps of
the $l$th layer of the VGG19 backbone, which is denoted as $\phi^{l}$, over
both the compositional image and alpha map:

$\mathcal{L}_{P}=\sum_{l\in\\{3,8\\}}\sum_{i=1}^{n}(\|\phi^{l}_{i}(\textbf{I})-\phi^{l}_{i}({\tilde{\textbf{I}}})\|_{2}^{2}+\|\phi^{l}_{i}(\alpha)-\phi^{l}_{i}({\tilde{\alpha}})\|_{2}^{2})$
(6)

Instead of computing losses on the final outputs only, we add an intermediate
loss to the MLP output to encourage multi-view consistency by considering the
regional patch-based CNN renderer has smaller perceptual fields and can not
capture global geometric, while the MLP block taking 3D location and view
direction as input and adding constraint on its output can better preserve
global consistency:

$\mathcal{L}_{N}=\sum_{i=1}^{n}(w_{\alpha}\|\alpha_{i}-\tilde{\alpha}_{i}\|_{2}^{2}+w_{i}\|\textbf{I}_{i}-\tilde{\textbf{I}}_{i}\|_{2}^{2}+\sum_{l\in\\{3,8\\}}\|\phi^{l}_{i}(\textbf{I})-\phi^{l}_{i}({\tilde{\textbf{I}}})\|_{2}^{2})$
(7)

where $w_{\alpha}=1$ if the scene is synthetic otherwise $0$ as the ground
truth alpha map of real objects is not view consistent, that is, we only
implicitly supervise the alpha with $RGB$ by considering stumpy gt alpha would
result in jitter phenomenon on the boundary region. Note that,
$w_{i}=a-b\alpha_{i}$ where $a-b=1$, this term penalties the false prediction
on empty region. We set $a=2,b=1$ in our experiment.

In summary, our optimizing objective function for generator $\mathcal{L}_{G}$
is:

$\mathcal{L}_{G}=\mathcal{L}_{C,\alpha}+\mathcal{L}_{P}+\mathcal{L}_{N}$ (8)

For the discriminator $D$, we minimize the adversarial loss [54]:

$\mathcal{L}_{D}=\|D(\textbf{I})\|_{2}^{2}+\|D(\tilde{\textbf{I}})-1\|_{2}^{2}$
(9)

Figure 4: (a) Our Capture system consists of a lit support, an array of
calibrated cameras, a precise step turntable, and a green screen. The fuzzy
object is placed on the lit support and the precise turntable is synchronized
with the camera array. (b) Samples of images captured among 6 cameras in 3
different steps. (c) The pipeline of alpha matte generation.

##  4 Capture System

Here, we describe our capture system, which produces high-quality multi-view
RGBA (RGB and Alpha) images for explicit opacity modeling of challenging fuzzy
objects. As illustrated in Fig. 4, our pipeline is equipped with an easy-to-
use capture device and stable calibration and automatic matting methods.

Device. Opacity object data could be captured through well-designed
acquisition systems proposed in NOPC [11]. However, there are some
limitations, including disability in capturing objects without support (e.g.,
hair without a head model) or capturing without the calibration box. Thus we
build a novel capture system as shown in Fig. 4(a), which consists of a
precisely controllable step turntable placed in front of the green screen and
six Nikon D750 DSLR Cameras facing towards the objects. For wig data, we use a
transparent support covered with soft light fabric to support the object and
place them on the turntable. Insert an LED light bar into the support to make
the soft light fabric lit. Before capturing, the cameras are adjusted to
overexpose the soft fabric part, which will reduce the shadows of the captured
image. When capturing images, the turntable moves forward in precise steps.
The turntable will step forward 80 steps per lap. All six cameras capture an
image per step, which will serve as the training data.

Calibration. Unlike general 3D object reconstruction, fuzzy objects are
difficult to reconstruct using structure-from-motion(SFM) techniques, which
means calibration via the reconstruction process would not work well. Previous
work[11] use an auxiliary calibration camera with a pattern box to solve this,
which takes much time to run an extra SFM pipeline for every capturing. In our
system, the calibration process only needs one time for each camera setting.
We know exactly how the camera’s external parameters are transformed from the
initial step to each step via the precise step turntable. Let $\mathbf{A}_{j}$
denote the affine transformation from the initial step to the $j$th step. The
calibration process is divided into two steps. Firstly, calibrate the
intrinsic and extrinsic parameters of six cameras at the initial step via
Zhang’s camera calibration[55], denote the $i$th camera’s intrinsic parameter
as $\mathbf{K}_{i}$ and extrinsic parameter at its $j$th step as
$\mathbf{T}_{i,j}$. Then, calculate the extrinsic parameter of each view under
the turntable’s coordinate

$\mathbf{T}_{i,j}=\mathbf{A}_{j}\mathbf{T}_{i,0}$ (10)

where $i$ is the index of cameras, and $j$ is the number of steps.

Opacity Decomposition. Image matting is an ill-posed problem due to a lack of
constraint in its formulation. To obtain alpha matte without loss of rich
details from fuzzy objects, we apply both 3D keyer and deep learning-based
matting algorithm (see Fig. 4(c)). Specifically, the keying benefits from our
specially designed capture system. In addition to the traditional green
screen, the overexposure of support reduces the shadow cast by the object,
providing a clean white screen. We key out both green and white from the image
to extract a preliminary foreground mask. A trimap is then generated from the
mask using erosion and dilation operations. We apply Context-aware Matting
[47] to predict alpha matte and foreground simultaneously, which combines the
benefits of local propagation and global contextual information and performs
visually much better on intractable data like curly hair and air bangs.

Figure 5: Object gallery. Our method can generalize well to various fuzzy
objects, including high frequency, view-dependency and translucency
appearance, such as hairstyles, clothes, toys and animals etc. Figure 6: Free
Viewpoint RGB results comparison with IBOH [6], NOPC [11] and NeRF [12] on
Cat, Girl, Wolf datasets. Our method is able to reconstruct fine details on
both geometry and appearance while keeping global view-consistency, such as
Cat’s fur texture, the pattern on Girl’s boots and the geometric details of
wolf’s hair. IBOH exhibits ghosting and aliasing, NOPC exhibits excessive
blurs and loss of geometric details and NeRF exhibits excessive noise and
blurs. See supplementary materials for more results.

##  5 Experiments

We evaluate our ConvNeRF on various furry objects. Quantitative and
qualitative evaluation in Sec. 5.1 show that our method can better preserve
high fidelity appearance details than prior work and generate globally
consistent alpha mattes in arbitrary novel views. We further perform extensive
ablation studies to validate our design choices in Sec. 5.2. We urge the
reader to view our supplementary video to better appreciate our method’s
significant improvement over baseline methods when rendering novel views.

Baselines. We adopt the conventional IBR method Image-based Opacity Hull
(IBOH) [6] and recent explicit neural rendering method Neural Opacity Point
Cloud (NOPC) [11] and implicit volume rendering method Neural Radiance Fields
(NeRF) [12] as baselines for comparisons.

Training details. We use half original image resolution for synthetic and real
objects, $200\sim 300$ images for training. We use patches of size $32\times
32$ and 64 samples (32 in coarse and fine modules) per ray for ConvNeRF
training. For the baseline models, we sample 300,000 points for each object
from the same proxy mesh with us as the dense point cloud input for the NOPC.
Also, we use 128 samples (64 in coarse and fine modules) for NeRF training. It
takes $1\sim 2$ days per object to train our model with a single NVIDIA TITAN
RTX GPU, while the NeRF takes about a week if trained on the same image number
and resolution.

###  5.1 Comparison

Figure 7: Free Viewpoint Alpha results comparison with IBOH [6], NOPC [11] and
NeRF [12] on Cat, Hairstyle 2 datasets. Our method can recover missing part
opacity such Cat’s beard from view-inconsistent alpha mattes as shown in the
first row, While IBOH fails and exhibits strong polygonal artifacts. Our
method can produce sharper alpha mattes than NOPC which suffers from strong
artifacts around the hair. NeRF fails on our challenging Hairstyle 2 dataset.
See supplementary materials for more results.

We perform quantitative and qualitative comparisons on both RGB and alpha with
the above baseline models. For qualitative evaluation, Fig. 6 shows several
our novel viewpoint RGB rendering results together with visual compassion v.s.
most recent neural scene representation methods. We can see that the IBOH
suffers from ghosting and aliasing due to the input views are in-uniform
sampled. The point cloud based NOPC can partially generate smooth texture
details with sufficient training images but still suffers from color shifting
and blur on the Girl object due to interpolation between the features
projected from the discrete spacial point cloud. The NeRF can preserve low-
frequency components but fails to recover high-frequency details due to the
limited representation ability of the MLP network. In the fur region, there is
noise caused by insufficient samples on both ray direction and the ray
resolutions. In contrast, our ConvNeRF is able to reconstruct fine texture and
geometry details and present sharply visual results with global consistency
(refer to our supplemental video).

Fig. 7 shows the visual discrepancies of free-viewpoint alpha maps rendering
results from given discrete view samples. Compared to other methods, our
ConvNeRF is able to preserve the sharpness of the hair boundary. Note that we
can easily obtain the perfect alpha map in synthesis scenes while not easy for
real data; our method is not sensitive to the quality of input alpha maps and
is able to partially recover missing part from nearby view samples (as shown
in the first row of Fig. 7).

We quantitatively evaluate our method with PSNR, LPIPS (ALEX backbone) [56]
and SSIM metric. 111We computer quantity value only on foreground region
(i.e., the object itself) by considering the background is not included in the
final render. As shown in Tab. I and Tab. II, our ConvNeRF achieves
significant improvement on both RGB and alpha results. We replace the LPIPS
metric with Sum of Absolute Distance (SAD) when evaluating the alpha result
due to the domain gap between the alpha-like image and the training set of the
network-based distance metric. Tab. III shows the average PSNR of all datasets
on semi-translucent region (i.e., $0<\alpha<1$). Our method achieves state-of-
the-art performance.

TABLE I: Quantitative comparisons of foreground images on different Objects

Method |  | IBOH | NOPC | NeRF | Ours
---|---|---|---|---|---
Wolf |  PSNR | 28.25 | 29.95 | 32.56 | 37.23
|  SSIM | 0.932 | 0.945 | 0.965 | 0.975
|  LPIPS | 0.073 | 0.058 | 0.061 | 0.022
Hair |  PSNR | 23.22 | 28.31 | 28.29 | 33.56
|  SSIM | 0.872 | 0.912 | 0.932 | 0.952
|  LPIPS | 0.104 | 0.058 | 0.070 | 0.021
Girl |  PSNR | 18.32 | 26.97 | 29.77 | 32.17
|  SSIM | 0.863 | 0.922 | 0.951 | 0.958
|  LPIPS | 0.150 | 0.099 | 0.052 | 0.035
Dog |  PSNR | 22.53 | 27.88 | 29.59 | 31.87
|  SSIM | 0.866 | 0.902 | 0.937 | 0.950
|  LPIPS | 0.127 | 0.098 | 0.053 | 0.030
Koala |  PSNR | 24.41 | 25.45 | 32.90 | 33.48
|  SSIM | 0.871 | 0.890 | 0.960 | 0.965
|  LPIPS | 0.103 | 0.117 | 0.053 | 0.023
Cat |  PSNR | 20.40 | 26.72 | 24.80 | 28.95
|  SSIM | 0.810 | 0.888 | 0.896 | 0.894
|  LPIPS | 0.198 | 0.101 | 0.187 | 0.090
Hairstyle 1 |  PSNR | 25.32 | 27.34 | 17.84 | 30.51
|  SSIM | 0.905 | 0.921 | 0.904 | 0.949
|  LPIPS | 0.067 | 0.058 | 0.135 | 0.037
Hairstyle 2 |  PSNR | 25.86 | 31.14 | 19.38 | 37.21
|  SSIM | 0.951 | 0.962 | 0.931 | 0.983
|  LPIPS | 0.104 | 0.033 | 0.131 | 0.014
Hairstyle 3 |  PSNR | 14.26 | 29.38 | 22.69 | 33.65
|  SSIM | 0.877 | 0.942 | 0.944 | 0.969
|  LPIPS | 0.130 | 0.040 | 0.073 | 0.016

TABLE II: Quantitative comparisons of alpha mattes on different Objects

Method |  | IBOH | NOPC | NeRF | Ours
---|---|---|---|---|---
Wolf |  SAD | 18.46 | 7.977 | 4.777 | 2.564
|  PSNR | 25.10 | 29.03 | 32.14 | 38.63
|  SSIM | 0.956 | 0.983 | 0.990 | 0.996
Hair |  SAD | 18.93 | 6.117 | 5.078 | 2.077
|  PSNR | 21.87 | 29.54 | 28.25 | 36.35
|  SSIM | 0.930 | 0.981 | 0.981 | 0.994
Girl |  SAD | 48.61 | 12.66 | 5.290 | 2.877
|  PSNR | 19.59 | 27.83 | 30.58 | 36.39
|  SSIM | 0.937 | 0.986 | 0.991 | 0.996
Dog |  SAD | 30.77 | 11.32 | 6.828 | 3.661
|  PSNR | 22.77 | 27.78 | 30.41 | 36.52
|  SSIM | 0.945 | 0.983 | 0.988 | 0.995
Koala |  SAD | 26.04 | 71.32 | 190.15 | 18.94
|  PSNR | 22.04 | 21.76 | 14.56 | 30.26
|  SSIM | 0.939 | 0.962 | 0.904 | 0.983
Cat |  SAD | 50.85 | 21.46 | 20.81 | 23.79
|  PSNR | 19.17 | 25.67 | 24.25 | 25.52
|  SSIM | 0.913 | 0.958 | 0.960 | 0.944
Hairstyle 1 |  SAD | 15.51 | 9.761 | 30.66 | 5.445
|  PSNR | 25.06 | 28.96 | 18.16 | 32.48
|  SSIM | 0.961 | 0.977 | 0.947 | 0.987
Hairstyle 2 |  SAD | 7.677 | 5.948 | 22.69 | 2.612
|  PSNR | 28.80 | 32.01 | 19.46 | 37.54
|  SSIM | 0.980 | 0.988 | 0.957 | 0.994
Hairstyle 3 |  SAD | 46.09 | 10.06 | 12.58 | 3.037
|  PSNR | 24.13 | 26.20 | 22.77 | 34.62
|  SSIM | 0.960 | 0.978 | 0.973 | 0.993

TABLE III: Average PSNR on semi-translucent region (U). U+ and U- represent the region after dilation and erosion.  Method | $\alpha\mathbf{F}$ | Alpha
---|---|---
U- | U | U+ | U- | U | U+
IBOH | 15.78 | 16.05 | 16.46 | 12.93 | 12.99 | 13.59
NOPC | 16.24 | 17.49 | 18.75 | 14.15 | 15.14 | 16.55
NeRF | 12.95 | 13.09 | 14.83 | 12.17 | 12.48 | 14.29
Ours | 25.23 | 26.21 | 27.06 | 20.73 | 22.45 | 24.21

###  5.2 Evaluation

To explore each part’s contribution in our pipeline, we conduct ablation
studies by choosing a set of controlled experiments, including randomized ray
sampling v.s. efficient ray sampling, Pixel-wised v.s. Convolutional renderer,
with/without GAN based discriminator loss, the range of the sampling Number
and patch size.

Randomize v.s. Efficient Ray Sampling We first compare the proposed Efficient
Ray Sampling (ERS) scheme described in Sec. 3.1, which include a baseline
model (a) (i.e., NeRF) with 64 samples per ray same as our ConvNeRF and a
model (b) with ERS. As the Wolf case shown in Fig. 8, model (a) fails to
recover surface details while ERS brings significant improvements to the
visual quality and quantitative quality shown by model (a) and (b) in Tab. IV.

Pixel-wised v.s. Convolutional Volume Renderer We train a generator model
without the discriminator, as (c) denoted in Tab. IV and Fig. 8, setting (c)
can recover better texture details on the wolf’s forehead while (a) and (b)
can only preserve low frequency information.

W/o GAN Loss We further evaluate the effect of the GAN discriminator adopted
for fine details refinement. The surface details of (c) are a little over-
smoothed while (d) present fine and sharp details on the fur texture and
hairlines, which is almost close to the ground truth. As for quantitative
metrics, (d), setting keeps high performance on foreground texture and alpha
quality, as the (c) and (d) shown in Tab. IV.

Figure 8: Visualization of the ablation study on w/o the ERS, CVR and GAN loss
modules (corresponding to the Tab IV).

Number of Sample Points. We further demonstrate the effect of different
numbers of per ray sample points and image numbers. We evaluate on
$N\in\\{16,32,48,64\\}$ and image number $S\in\\{50,100,150,200,250,300\\}$
respectively, as shown in the first and second row of Fig. 9. More sample
points can lead to better rendering quality and higher alpha prediction
accuracy such as hairlines. Note that our method is not sensitive to training
data size and can produce satisfactory results even with a few samples per
ray.

Patch Size. As we introduce the convolutional mechanism in the image plane,
the patch size plays a significant role for synthesis quality. Larger patch
size helps preserve high-order features through perceptual loss while leads to
fewer patches for training, which reduces data diversity for adversarial
training. To evaluate the effect of different patch sizes, we set the patch
size $K\in\\{8,16,32,64\\}$ and conduct quantitative(Fig. 9) and
qualitative(Fig. 10) comparisons. Notice that although the model with $K=8$
achieves similar quantitative performance with our proposed model, it exhibits
severe visual artifacts (the first row of Fig. 10).

Textured Background. As we mask out the background for real data, it may
affect the rendering quality of both NeRF and our model. Thus we render the
synthetic scene Wolf together with rich texture backgrounds and parallax to
evaluate our approach against NeRF. Tab. V indicates that both models perform
slightly better in radiance rendering, while using textured background seems
to downgrade the alpha rendering quality of both our approach and NeRF. It
follows the insight that utilizing explicit alpha supervision is more
effective than relying on the network itself to extract such alpha cues
implicitly.

TABLE IV: Quantitative evaluation of the ablation studies. Models | $\alpha\mathbf{F}$ | Alpha
---|---|---
PSNR $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SAD $\downarrow$
(a) w/o GAN, CVR, ERS | 20.27 | 0.175 | 20.18 | 23.51
(b) w/o GAN, CVR | 31.38 | 0.074 | 34.46 | 3.808
(c) w/o GAN | 37.55 | 0.026 | 37.93 | 2.642
(d) ConvNeRF | 37.23 | 0.022 | 38.63 | 2.564
Figure 9: From top to bottom: rendering quality v.s. training image number,
samples per ray, patch size Figure 10: Ablation study on the patch size of
$K=[8,16,32,64]$. Small and large sizes produce color cast and blurry
artifacts.

###  5.3 Limitation and Discussion

Our model is able to reconstruct high-frequency appearance from given multi-
view RGBA images, the combination of our patch-wise learning and the global
implicit representation described in Sec. 3 encourage view-consistency,
however, we note a bit of flicker in the around-view videos. It is mainly due
to the patch-wise adversarial loss in Eqn. 9 which increases high-frequency
details significantly with a slight trade-off of view consistency.

On the other hand, our method relies on accurately calibrated camera pose;
inaccurate camera poses for in the wild images will lead to loss of geometric
details especially for fuzzy objects. Furthermore, since our method requires
background images captured or separated by matting algorithms, it is almost
impossible to manually capture the aligned background or obtain accurate alpha
matte of images with a complex background, resulting in hard to extend it to
in the wild data.

TABLE V: Quantitative evaluation of textured background. Background |  | $\alpha\mathbf{F}$ | Alpha
---|---|---|---
| PSNR $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SAD $\downarrow$
textured | NeRF | 33.26 | 0.060 | 31.05 | 6.478
Ours | 37.95 | 0.023 | 35.51 | 4.309
white | NeRF | 32.56 | 0.061 | 32.14 | 4.777
Ours | 37.23 | 0.022 | 38.63 | 2.564

##  6 Conclusion

We have presented a novel neural rendering framework to combine both explicit
opacity modeling and convolutional mechanism into the neural radiance field,
enabling high-quality, globally consistent and free-viewpoint appearance and
opacity rendering for fuzzy objects. Our efficient sampling strategy enables
efficient radiance field sampling and learning in a patch-wise manner, while
our novel volumetric integration generates per-patch hybrid features to
reconstruct the view-consistent fine-detailed appearance and opacity output.
Our novel patch-wise adversarial training scheme further preserves the high-
frequency appearance and opacity details for photo-realistic rendering in a
self-supervised framework. Our experimental results demonstrate the
effectiveness of the proposed convolutional neural opacity radiance field for
high-quality appearance and opacity modeling. We believe that our approach is
a significant step to enable photo-realistic modeling and rendering for
challenging fuzzy objects, with many potential applications in VR/AR like
gaming, entertainment and immersive telepresence.

## Acknowledgments

This work was supported by NSFC programs (61976138, 61977047), the National
Key Research and Development Program (2018YFB2100500), STCSM
(2015F0203-000-06) and SHMEC (2019-01-07-00-01-E00003).

## References

  * [1] S. Paris, W. Chang, O. I. Kozhushnyan, W. Jarosz, W. Matusik, M. Zwicker, and F. Durand, “Hair photobooth: geometric and photometric acquisition of real hairstyles.” _ACM Trans. Graph._ , vol. 27, no. 3, p. 30, 2008.
  * [2] L. Luo, H. Li, and S. Rusinkiewicz, “Structure-aware hair capture,” _ACM Transactions on Graphics (Proceedings SIGGRAPH 2013)_ , vol. 32, no. 4, July 2013\.
  * [3] Z. Xu, H.-T. Wu, L. Wang, C. Zheng, X. Tong, and Y. Qi, “Dynamic hair capture using spacetime optimization,” _ACM Trans. Graph._ , vol. 33, no. 6, Nov. 2014. [Online]. Available: https://doi.org/10.1145/2661229.2661284
  * [4] G. Nam, C. Wu, M. H. Kim, and Y. Sheikh, “Strand-accurate multi-view hair capture,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [5] C. Buehler, M. Bosse, L. Mcmillan, S. Gortler, and M. Cohen, “Unstructured lumigraph rendering,” in _Conference on Computer Graphics and Interactive Techniques_ , 2001, pp. 425–432.
  * [6] W. Matusik, H. Pfister, A. Ngan, P. Beardsley, R. Ziegler, and L. McMillan, “Image-based 3d photography using opacity hulls,” _ACM Transactions on Graphics (TOG)_ , vol. 21, no. 3, pp. 427–437, 2002.
  * [7] N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” vol. 35, no. 6, Nov. 2016. [Online]. Available: https://doi.org/10.1145/2980179.2980251
  * [8] V. Sitzmann, M. Zollhöfer, and G. Wetzstein, “Scene representation networks: Continuous 3d-structure-aware neural scene representations,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [9] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh, “Neural volumes: Learning dynamic renderable volumes from images,” _arXiv preprint arXiv:1906.07751_ , 2019.
  * [10] J. Thies, M. Zollhöfer, and M. Nießner, “Deferred neural rendering: Image synthesis using neural textures,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 4, pp. 1–12, 2019.
  * [11] C. Wang, M. Wu, Z. Wang, L. Wang, H. Sheng, and J. Yu, “Neural opacity point cloud,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [12] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _arXiv preprint arXiv:2003.08934_ , 2020.
  * [13] K.-A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, and V. Lempitsky, “Neural point-based graphics,” _arXiv preprint arXiv:1906.08240_ , 2019.
  * [14] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, “Deepvoxels: Learning persistent 3d feature embeddings,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2437–2446.
  * [15] H. Kato, Y. Ushiku, and T. Harada, “Neural 3d mesh renderer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [16] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 165–174.
  * [17] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, “Occupancy networks: Learning 3d reconstruction in function space,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4460–4470.
  * [18] L. Liu, J. Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,” _NeurIPS_ , 2020.
  * [19] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” _arXiv preprint arXiv:2008.02268_ , 2020.
  * [20] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R.-M. Brualla, “Deformable neural radiance fields,” _arXiv preprint arXiv:2011.12948_ , 2020.
  * [21] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” 2020.
  * [22] Z. Chen and H. Zhang, “Learning implicit fields for generative shape modeling,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5939–5948.
  * [23] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger, “Convolutional occupancy networks,” _arXiv preprint arXiv:2003.04618_ , 2020\.
  * [24] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li, “Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2304–2314.
  * [25] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su _et al._ , “Shapenet: An information-rich 3d model repository,” _arXiv preprint arXiv:1512.03012_ , 2015.
  * [26] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, “Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3504–3515.
  * [27] P. Debevec, C. Bregler, M. Cohen, and L. Mcmillan, “Image-based modeling and rendering,” 1998, p. 299.
  * [28] M. Levoy, “Light field rendering,” in _Conference on Computer Graphics and Interactive Techniques_ , 1996, pp. 31–42.
  * [29] A. Chen, M. Wu, Y. Zhang, N. Li, J. Lu, S. Gao, and J. Yu, “Deep surface light fields,” _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ , vol. 1, no. 1, pp. 1–17, 2018.
  * [30] K. Zhang, G. Riegler, N. Snavely, and V. Koltun, “Nerf++: Analyzing and improving neural radiance fields,” _arXiv preprint arXiv:2010.07492_ , 2020\.
  * [31] M. Wu, Y. Wang, Q. Hu, and J. Yu, “Multi-view neural human rendering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [32] Z. Chen, A. Chen, G. Zhang, C. Wang, Y. Ji, K. N. Kutulakos, and J. Yu, “A neural rendering framework for free-viewpoint relighting,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5599–5610.
  * [33] Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski, “A bayesian approach to digital matting,” in _Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001_ , vol. 2. IEEE, 2001, pp. II–II.
  * [34] J. Wang and M. F. Cohen, “An iterative optimization approach for unified image segmentation and matting,” in _Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1_ , vol. 2. IEEE, 2005, pp. 936–943.
  * [35] E. S. Gastal and M. M. Oliveira, “Shared sampling for real-time alpha matting,” in _Computer Graphics Forum_ , vol. 29, no. 2. Wiley Online Library, 2010, pp. 575–584.
  * [36] E. Shahrian, D. Rajan, B. Price, and S. Cohen, “Improving image matting using comprehensive sampling sets,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 636–643.
  * [37] L. Karacan, A. Erdem, and E. Erdem, “Image matting with kl-divergence based sparse sampling,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 424–432.
  * [38] E. Elhamifar, G. Sapiro, and S. S. Sastry, “Dissimilarity-based sparse subset selection,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 11, pp. 2182–2197, 2015.
  * [39] J. Sun, J. Jia, C.-K. Tang, and H.-Y. Shum, “Poisson matting,” in _ACM SIGGRAPH 2004 Papers_ , 2004, pp. 315–321.
  * [40] L. Grady, T. Schiwietz, S. Aharon, and R. Westermann, “Random walks for interactive alpha-matting,” in _Proceedings of VIIP_ , vol. 2005, 2005, pp. 423–429.
  * [41] A. Levin, D. Lischinski, and Y. Weiss, “A closed-form solution to natural image matting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 30, no. 2, pp. 228–242, 2007.
  * [42] Q. Chen, D. Li, and C.-K. Tang, “Knn matting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 9, pp. 2175–2188, 2013\.
  * [43] X. Chen, D. Zou, S. Zhiying Zhou, Q. Zhao, and P. Tan, “Image matting with local and nonlocal smooth priors,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 1902–1907.
  * [44] N. Xu, B. Price, S. Cohen, and T. Huang, “Deep image matting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2970–2979.
  * [45] S. Cai, X. Zhang, H. Fan, H. Huang, J. Liu, J. Liu, J. Liu, J. Wang, and J. Sun, “Disentangled image matting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8819–8828.
  * [46] H. Yu, N. Xu, Z. Huang, Y. Zhou, and H. Shi, “High-resolution deep image matting,” _arXiv preprint arXiv:2009.06613_ , 2020.
  * [47] Q. Hou and F. Liu, “Context-aware image matting for simultaneous foreground and alpha estimation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4130–4139.
  * [48] M. Forte and F. Pitié, “$f$, $b$, alpha matting,” _arXiv preprint arXiv:2003.07711_ , 2020.
  * [49] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * [50] Y. Zhang, L. Gong, L. Fan, P. Ren, Q. Huang, H. Bao, and W. Xu, “A late fusion cnn for digital matting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7469–7478.
  * [51] S. Sengupta, V. Jayaram, B. Curless, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Background matting: The world is your green screen,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2291–2300.
  * [52] K. M. Cheung, S. Baker, and T. Kanade, “Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture,” in _2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings._ , vol. 1, 2003, pp. I–I.
  * [53] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Free-form image inpainting with gated convolution,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4471–4480.
  * [54] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [55] Z. Zhang, “Flexible camera calibration by viewing a plane from unknown orientations,” in _Proceedings of the seventh ieee international conference on computer vision_ , vol. 1. Ieee, 1999, pp. 666–673.
  * [56] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _CVPR_ , 2018\.

|  Haimin Luo received the B.S. degree from the School of Computer Science and
Technology, Huazhong University of Science and Technology, Wuhan, China, in
2019. He is currently working toward the master’s degree at ShanghaiTech
University, Shanghai, China. His research interests include computer vision,
computer graphics, and deep learning.
---|---
|  Anpei Chen received the B.S. degree from the School of Physics and
Optoelectronic Engineering, Xidian University, Shanxi, China, in 2016. He is
currently working toward the PhD degree at ShanghaiTech University, Shanghai,
China. His research interests lie at the intersection of computer graphics and
computer vision, including image synthesis$/$editing, geometric modeling, and
realistic rendering.
---|---
|  Qixuan Zhang is currently working toward the B.S. degree at ShanghaiTech
University, Shanghai, China. His research interests include computer vision,
computational photography, and deep learning.
---|---
|  Bai Pang is currently working toward the B.S. degree at ShanghaiTech
University, Shanghai, China. His research interests include computer vision,
computational photography and machine learning.
---|---
|  Minye Wu received the B.S. degree from the School of Computer Engineering
and Science, Shanghai University, Shanghai, China, in 2015. He is currently
working toward the PhD degree at ShanghaiTech University, Shanghai, China. He
is also with the Shanghai Institute of Microsystem and Information Technology,
Chinese Academy of Sciences, China, and the University of Chinese Academy of
Sciences, China. His research interests include computer vision, deep
learning, and computational photography.
---|---
|  Lan Xu received the B.E. degree from Zhejiang University in 2015 and the
Ph.D. degree from the Department of Electronic and Computer Engineering (ECE),
The Hong Kong University of Science and Technology (HKUST), in 2020. He is
currently an Assistant Professor with ShanghaiTech University. His research
interests include computer vision, computer graphics, and machine learning.
---|---
|  Jingyi Yu received B.S. from Caltech in 2000 and PhD from MIT in 2005. He
is currently the Vice Provost at the ShanghaiTech University. Before joining
ShanghaiTech, he was a full professor in the Department of Computer and
Information Sciences at University of Delaware. His research interests span a
range of topics in computer vision and computer graphics, especially on
computational photography and non-conventional optics and camera designs. He
is a recipient of the NSF CAREER Award and the AFOSR YIP Award, and has served
as an area chair of many international conferences including CVPR, ICCV, ECCV,
IJCAI and NeurIPS. He is currently a program chair of CVPR 2021 and will be a
program chair of ICCV 2025. He has been an associate editor of the IEEE
Transactions on Pattern Analysis and Machine Intelligence, the IEEE
Transactions on Image Processing, and the Elsevier Computer Vision and Image
Understanding. He is a fellow of IEEE.
---|---


and treated as control parameters. First, we create a grid (of
$[300]\times[300]$-resolution) of ordered $([\rm Ca],\,x)$-pairs over the slow
phase plane. For each pair, a phase trajectory of the model computed and the
initial transient is discarded. The trajectory is classified as either a
spiker if voltage has three or more zero crossings, and quiescent otherwise.
Each case is treated separately but in both cases, four quantities are
characterized: spike frequency, and average values $\langle V\rangle$,
$\langle x\rangle$ and $\langle[\rm Ca]\rangle$. When the system is quiescent,
the frequency is 0, the average voltage is the final voltage, which is simply
plugged into the $x$ and $[\rm Ca]$ state equations to determine $\langle
x\rangle$ and $\langle[\rm Ca]\rangle$-values.

In the case of tonic-spiking activity, we use the final three zero crossings
of the entire trajectory between, to determine the established the spike
frequency and the average voltage. The $x$\- and $[\rm Ca]$-coordinates are
calculated for every point in the sampling of the limit cycle, and
subsequently averaged. This order of operations is required due to
nonlinearities in the model dynamics. The nullclines are plotted as the zero
contours of the resulting $\langle x\rangle$ and $\langle[\rm Ca]\rangle$
arrays, while the SNIC-curve itself is evaluated as the zero contour of the
frequency array. Average voltage is only used to generate the corresponding
heat-map color-painting in the phase plane with a red hue for its oscillatory
part above the SNIC-curve, and below it with a blue hue representing quiescent
phases.

## Appendix III: Sequential phase plane analysis of network periodic orbits

In order to validate the mechanism underlying rhythmogenesis in the two
network models we visualize the movement of the nullclines over the course of
network bursting. We partitioned a network bursting trajectory into subsets
where all $S(t)$ were monotonic in time, and visualized the transformations of
the geometry as the network oscillations unfolded. During our investigation of
the excitatory-inhibitory module, we noticed that it was possible to obtain
network oscillations with frozen $x$ and $[{\rm Ca}]$-variables, suggesting
that the mechanism of rhythmogenesis was not a product of the slow phase
dynamics alone. Further investigation into the excitatory inhibitory module is
required.

We showed above that in the case of the HCO network, the onset of network
oscillations emerging from transients is coordinated by cyclicly moving
$x$-nullclines $x^{\prime}=0$ in the slow $([\rm Ca],\,x)$-phase plane. Here,
we include a supplementary Fig. 19 to further detail and validate our
analysis. Its panels A and D represent two representative snapshots of time-
varying rearrangements of the nullclines, equilibrium $x^{\prime}=0$, average
$\langle x\rangle$ and calcium $[\rm Ca]^{\prime}=0$, in a chronological
sequence over a half of each burst cycle. Since the HCO network is symmetric,
the neurons will next switch their roles to complete the full bursting cycle.
Close examination yields an interpretation by breaking a half-cycle sequence
into two stages:
Stage I. As the active pre-synaptic neuron 2 (its initial/final positions
given by blue/red dots in Panel D) drifts rightwards and approaches the SNIC-
curve (not shown) along the average $\langle x\rangle$ nullcline, its spike
frequency decreases so that the synaptic probability $S_{2}(t)$ (black line in
Panel C) drops to zero. The concurrent decay of inhibition projected from
neuron 2 onto neuron 1 (its initial/final positions given by blue/red dots in
Panel A) lets the corresponding $x_{1}$-variable rise, which in turn makes the
average $\langle x\rangle$ nullcline for neuron 2 shift to the right.
Stage II. After the $x_{1}$-variable reaches the average $langlex\rangle$
nullcline (color-coded red line in Panel A), neuron 1 starts spiking and hence
inhibiting neuron 2. The inhibition causes the average $\langle x\rangle$
nullcline for neuron 2 to shift leftwards so that the $x_{2}$ variable drop
down close to zero and that the corresponding phase trajectory of neuron 2 is
about to start traversing along the forced (red line) equilibrium nullcline
$x^{\prime}=0$ in the phase plane in Fig. 19D, and so forth.

To smooth the $V(t)$ and $s(t)$ traces of the active spiking neurons, the
voltage trajectory is partitioned to ensure that each spike belongs to a
unique subset by monitoring a threshold set around -35mV. A maximum spike time
of $1$s determines whether each subset contains a spike, or a quiescent
episode of the burst. Spike containing subsets are replaced with their voltage
averages, while quiescent voltage level are left intact. The partitioning and
averaging for each $S(t)$ follow their corresponding presynaptic voltages.
Finally, a composition of two moving averages with $0.5$s window is applied as
a filter for further smothering.

## References

## References

  * [1] A Selverston, editor.  Model Neural Networks and Behavior.  Springer, Berlin, 1985.
  * [2] E Marder and RL Calabrese.  Principles of rhythmic motor pattern generation.  Physiological Reviews, 76(3):687–717, 1996.
  * [3] PS Katz and SL Hooper.  Invertebrate central pattern generators.  In G North and R R. Greenspan, editors, Invertebrate Neurobiology. Cold Spring Harbor Laboratory Press, NY, New York, 2007.
  * [4] AI Selverston, DF Russell, JP Miller, and DG King.  The stomatogastric nervous system: structure and function of a small neural network.  Progress in neurobiology, 7:215–289, 1976.
  * [5] PSG Stein, DG Stuart, S Grillner, and AI Selverston.  Neurons, networks, and motor behavior.  MIT press, 1999.
  * [6] AI Selverston.  Model neural networks and behavior.  Springer Science & Business Media, 2013.
  * [7] E Marder.  Invertebrate neurobiology: Polymorphic neural networks.  Current Biology, 4(8):752 – 754, 1994.
  * [8] AA Prinz, V Thirumalai, and E Marder.  The functional consequences of changes in the strength and duration of synaptic inputs to oscillatory neurons.  The Journal of neuroscience, 23(3):943–954, 2003.
  * [9] AA Prinz, D Bucher, and E Marder.  Similar network activity from disparate circuit parameters.  Nature neuroscience, 7(12):1345–1352, 2004.
  * [10] JM Goaillard, AL Taylor, D Schulz, and E Marder.  Multi-dimensional measurements to understand the functional consequences of animal-to-animal variation in circuit parameters.  Nature Neuroscience, 12:1424–1430, 2009.
  * [11] E Marder.  Neuromodulation of neuronal circuits: back to the future.  Neuron, 76:1, 2012.
  * [12] WH Watson and JM Newcomb.  Modulation of swimming in the gastropod melibe leonina by nitric oxide.  J. Exp. Biol., 205(3):397, 2002.
  * [13] WH Watson, JM Newcomb, and S Thompson.  Neural correlates of swimming behavior in melibe leonina.  Biol Bull., 203(2):152, 2002.
  * [14] S Thompson and WH Watson.  Central pattern generator for swimming in melibe.  J. Exp. Biol., 208(7):1347, 2005.
  * [15] A Sakurai, JM Newcomb, JL Lillvis, and PS Katz.  Different roles for homologous interneurons in species exhibiting similar rhythmic behaviors.  Curr. Biol., 21:1036, 2011.
  * [16] JM Newcomb, A Sakurai, JL Lillvis, CA Gunaratne, and PS Katz.  Homology and homoplasy of swimming behaviors and neural circuits in the nudipleura (mollusca, gastropoda, opistho-branchia).  Proc. Natl. Acad. Sci., 109(1):10669–76, 2012.
  * [17] A Sakurai, CA Gunaratne, and PS Katz.  Two interconnected kernels of reciprocally inhibitory interneurons underlie alternating left-right swim motor pattern generation in the mollusk melibe leonina.  Journal of Neurophysiology, 112(6):317–1328, 2014.
  * [18] A Sakurai and PS Katz.  Phylogenetic and individual variation in gastropod central pattern generators.  Journal of Comparative Physiology A, 201(9):829–839, 2015.
  * [19] A Sakurai and PS Katz.  The central pattern generator underlying swimming in dendronotus iris: a simple half-center network oscillator with a twist.  Journal of neurophysiology, 116(4):1728–1742, 2016.
  * [20] A Sakurai and PS Katz.  Artificial synaptic rewiring demonstrates that distinct neural circuit configurations underlie homologous behaviors.  Current Biology, 27(12):1721–1734, 2017.
  * [21] A Sakurai and PS Katz.  Command or obey? homologous neurons differ in hierarchical position for the generation of homologous behaviors.  Journal of Neuroscience, 39(33):6460–6471, 2019.
  * [22] A Sakurai and PS Katz.  Bursting emerges from the complementary roles of neurons in a four-cell network.  Journal of Neurophysiology, 127(4):1054–1066, 2022.
  * [23] RE Plant and M Kim.  On the mechanism underlying bursting in the aplysia abdominal ganglion r15 cell.  Math. Biosci., 26:357–375, 1975.
  * [24] RE Plant and M Kim.  Mathematical description of a bursting pacemaker neuron by a modification of the hodgkin-huxley equations.  Biophysics Journal, 16:227–244, 1976.
  * [25] RE Plant.  Bifurcation and resonance in a model for bursting nerve cells.  J. Math. Biol, 11:15–32, 1981.
  * [26] AR Rittenhouse and CH Price.  Peripheral axons of the parabolic burster neuron r15.  Brain Research, 333:330–335, 1985.
  * [27] ES Levitan and IB Levitan.  Serotonin acting via cyclic amp enhances both the hyperpolarizing and depolarizing phases of bursting pacemaker activity in the aplysia neuron R15.  J Neuroscience, 8:1152–1161, 1988.
  * [28] J Rinzel.  Bursting oscillations in an excitable membrane model.  Lecture Notes in Mathematics, 1151:304–316, 1985.
  * [29] J Rinzel.  A formal classification of bursting mechanisms in excitable systems.  1987\.
  * [30] GB Ermentrout and N Kopell.  Parabolic bursting in an excitable system coupled with a slow oscillation.  SIAM J. Applied Mathematics, 46:233–253, 1986.
  * [31] SM Baer, J Rinzel, and H Carrillo.  Analys of an autonomous phase model for neuronal parabolic bursting.  Journal of Mathematical Biology, 33:309–333, 1995.
  * [32] J Rinzel and YS Lee.  On different mechanisms for membrane potential bursting, volume 66.  In: Othmer, H.G. (ed.) Nonlinear Oscillations in Biology and Chemistry: Lecture Notes in Biomathematics, Springer, Berlin, 1986.
  * [33] J Rinzel and YS Lee.  Dissection of a model for neuronal parabolic bursting.  Journal of Mathematical Biology, 25(6):653–675, 1987.
  * [34] CC Canavier, JW Clark, and JH Byrne.  Simulation of the bursting activity of neuron R15 in aplysia: Role of ionic currents, calcium balance, and modulatory transmitters.  J Neurophysiology, 66(6):2107–2124., 1991.
  * [35] R Bertran.  A computational study of the effects of serotonin on a molluscan burster neuron.  Biol. Cybern., 69:257–267, 1993.
  * [36] CC Canavier, DA Baxter, JW Clark, and JH Byrne.  Nonlinear dynamics in a model neuron provide a novel mechanism for transient synaptic inputs to produce long-term alterations of postsynaptic activity.  J Neurophysiol, 69(6):2252–2257, Jun 1993.
  * [37] RJ Butera, J W Jr. Clark, CC Canavier, D A Baxter, and J H Byrne.  Analysis of the effects of modulatory agents on a modeled bursting neuron: Dynamic interactions between voltage and calcium dependent systems.  J Comput. Neuroscience, 2(1):19–44, 1995.
  * [38] R Butera.  Multirhythmic bursting.  Chaos, 8(1):274–284, 1998.
  * [39] FH Sieling and R Butera.  Aplysia R15 neuron.  Scholarpedia, 6(10):4181, 2011.
  * [40] D Alaçam and AL Shilnikov.  Making a swim central pattern generator out of latent parabolic bursters.  International Journal of Bifurcation and Chaos, 25(07):1540003, 2015\.
  * [41] AL Shilnikov and G Cymbalyuk.  Homoclinic saddle-node orbit bifurcations en a route between tonic spiking and bursting in neuron models, invited review.  Regular & Chaotic Dynamics, 3(9):281–297, 2004.
  * [42] AL Shilnikov.  Complete dynamical analysis of an interneuron model.  J. Nonlinear Dynamics, 68(3):305–328, 2012.
  * [43] LP Shilnikov, AL Shilnikov, DV Turaev, and LO Chua.  Methods of qualitative theory in nonlinear dynamics, volume 1 and 2.  World Scientific, Singapore, 1998, 2001.
  * [44] H Ju, AB Neiman, and AL Shilnikov.  Bottom-up approach to torus bifurcation in neuron models.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 28(10):106317, 2018.
  * [45] AA Andronov and EA Leontovich.  Some cases of dependence of limit cycles on a parameter.  Uchenye zapiski Gorkovskogo Universiteta, 6:3–24, 1937.
  * [46] AA Andronov, EA Leontovich, et al.  To the theory of changing of qualitative structure of trajectories on the plane.  In Dokl. Akad. Nauk, volume 21, pages 427–430, 1938.
  * [47] AA Andronov, EA Leontovich, II Gordon, and AG Mayer.  Qualitative theory of dynamical systems of the second order.  Moscow: Mir, 1966.
  * [48] AA Andronov, EA Leontovich, II Gordon, and AG Mayer.  Bifurcations theory for dynamical systems on the plane.  Moscow: Nauka, 1967.
  * [49] LP Shilnikov.  Some instances of generation of periodic motions in n-space.  Doklady Akademii Nauk, 143(2):289–292, 1962.
  * [50] LP Shilnikov.  Some cases of generation of period motions from singular trajectories.  Matematicheskii Sbornik, 103(4):443–466, 1963.
  * [51] AL Shilnikov, LP Shilnikov, and DV Turaev.  On some mathematical topics in classical synchronization: a tutorial.  J Bifurcations and Chaos, 14(7):2143–2160, 2004.
  * [52] VS Afraimovich, SV Gonchenko, LM Lerman, AL Shilnikov, and DV Turaev.  Scientific heritage of LP Shilnikov.  Regular and Chaotic Dynamics, 19(4):435–460, 2014.
  * [53] VS Afraimovich, LA Belyakov, SV Gonchenko, LM Lerman, AD Morozov, D Turaev, and AL Shilnikov.  Selected scientific works of L.P. Shilnikov.  Nizhny Novgorod: UNN, 2017.
  * [54] AL Shilnikov and G Cymbalyuk.  Homoclinic bifurcations of periodic orbits en a route from tonic spiking to bursting in neuron models.  Regular and Chaotic dynamics, 9(3):281–297, 2004.
  * [55] AL Shilnikov, LP Shilnikov, and DV Turaev.  Blue sky catastrophe in singularly perturbed systems.  Moscow Math. Journal, 5(1):205–211, 2005.
  * [56] AL Shilnikov and ML Kolomiets.  Methods of the qualitative theory for the Hindmarsh-Rose model: a case study. a tutorial.  J Bifurcations and Chaos, 18(7):1–32, 2008.
  * [57] XJ Wang and J Rinzel.  Alternating and synchronous rhythms in reciprocally inhibitory model neurons.  J. Neural Comp., 4:84–97, 1992.
  * [58] N Kopell and D Somers.  Rapid synchronization through fast threshold modulation.  Biol. Cybernetics, 68:5, 1993.
  * [59] IV Belykh and AL Shilnikov.  When weak inhibition synchronizes strongly desynchronizing networks of bursting neurons.  Phys Rev Lett, 101(7):078102, 2008.
  * [60] AL Shilnikov, R Gordon, and IV Belykh.  Polyrhythmic synchronization in bursting networking motifs.  Chaos, 18(3):037120, Sep 2008.
  * [61] S Jalil, IV Belykh, and AL Shilnikov.  Fast reciprocal inhibition can synchronize bursting neurons.  Physical Review E, 81(4, 2), APR 2010.
  * [62] J Wojcik, R Clewley, and AL Shilnikov.  Order parameter for bursting polyrhythms in multifunctional central pattern generators.  Phys Rev E, 83:056209–6, 2011.
  * [63] J Wojcik, R Clewley, J Schwabedal, and AL Shilnikov.  Key bifurcations of bursting polyrhythms in 3-cell central pattern generators.  PLoS ONE, 9(4), 2014.
  * [64] JTC Schwabedal, DE Knapper, and AL Shilnikov.  Qualitative and quantitative stability analysis of penta-rhythmic circuits.  Nonlinearity, 29(12):3647, 2016.
  * [65] A Kelley and AL Shilnikov.  2 $\theta$-burster for rhythm-generating circuits.  Frontiers in Applied Mathematics and Statistics, 6:588904, 2020\.
  * [66] J Collens, K Pusuluri, A Kelley, D Knapper, T Xing, S Basodi, Deni̇z Alacam, and AL Shilnikov.  Dynamics and bifurcations in multistable 3-cell neural networks.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(7):072101, 2020.
  * [67] K Pusuluri, S Basodi, and AL Shilnikov.  Computational exposition of multistable rhythms in 4-cell neural circuits.  Communications in Nonlinear Science and Numerical Simulation, 83:105139, 2020.
  * [68] S Jalil, D Allen, J Youker, and AL Shilnikov.  Toward robust phase-locking in melibe swim central pattern generator models.  Chaos, 23(4):046105, 2013.
  * [69] JTC Schwabedal, AB Neiman, and AL Shilnikov.  Robust design of polyrhythmic neural circuits.  Phys. Rev. E, 90:022715, 2014.
  * [70] M Lodi, AL Shilnikov, and M Storace.  Design principles for central pattern generators with preset rhythms.  IEEE Transactions on Neural Networks and Learning Systems, 31(9):3658–3669, 2019.
  * [71] M Beveridge.  Sea slug lateral swimming style.  Natural History, May 2009.
  * [72] W Rall.  Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic input.  J. Neurophysiology, 30(50):1138–1168, 1967.
  * [73] A Destexhe, ZF Mainen, and TJ Sejnowski.  Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism.  Journal of computational neuroscience, 1(3):195–230, 1994.
  * [74] S Jalil, IV Belykh, and AL Shilnikov.  Spikes matter for phase-locked bursting in inhibitory neurons.  Physical Review E, 85(3, 2), MAR 22 2012.
  * [75] V Baruzzi, M Lodi, M Storace, and AL Shilnikov.  Towards more biologically plausible central-pattern-generator models.  Physical Review E, 104(6):064405, 2021.
  * [76] V Baruzzi, M Lodi, M Storace, and AL Shilnikov.  Generalized half-center oscillators with short-term synaptic plasticity.  Physical Review E, 102(3):032406, 2020.
  * [77] A Destexhe.  Kinetic models of synaptic transmission.  Methods In Neuronal Modeling, pages 1–26, 1998.
  * [78] A Destexhe, D Contreras, TJ Sejnowski, and M Steriade.  A Model of Spindle Rhythmicity in the Isolated Thalamic Reticular Nucleus.  Journal of Neurophysiology, 72(2):803–818, 1994.
  * [79] D Golomb, XJ Wang, and J. Rinzel.  Propagation of spindle waves in a thalamic slice model.  J. Neural Comp., 75(2):750–769, 1996.
  * [80] A. Hill, J. Lu, MA Masino, OH Olsen, and RL. Calabrese.  A model of a segmental oscillator in the leech heartbeat neuronal network.  Journal of computational neuroscience, 10(3):281–302, 2001.
  * [81] T Graham-Brown.  The intrinsic factors in the act of progression in the mammal.  Lond B Biol Society, 84(572):308–319, 1911.
  * [82] DH Perkel and B Mulloney.  Mechanism of postinhibitory rebound in molluscan neurons.  Science, 185(4146):181–183, 1974.
  * [83] F Skinner, N Kopell, and E Marder.  Mechanisms for oscillation and frequency control in networks of mutually inhibitory relaxation oscillators.  Comput. Neurosci., 1:69, 1994.
  * [84] FK Skinner, L Zhang, JL Perez Velazquez, and PL Carlen.  Bursting in inhibitory interneuronal networks: A role for gap-junctional coupling.  J. Neurophysiol, 81:1274, 1999.
  * [85] N Kopell and G Ermentrout.  Mechanisms of phase-locking and frequency control in pairs of coupled neural oscillators.  Handbook on Dynamical Systems, 2:3–54, 2002.
  * [86] JD Angstadt, JL Grassmann, KM Theriault, and SM Levasseur.  Mechanisms of post-inhibitory rebound and its modulation by serotonin in excitatory swim motor neurons of the medicinal leech.  Journal of Comparative Physiology A-Neuroethology, Sensory, Neural and Behavioral Physiology, 191(8):715–732, 2005.
  * [87] V Matveev, A Bose, and F Nadim.  Capturing the bursting dynamics of a two-cell inhibitory network using a one-dimensional map.  Journal of Computational Neuroscience, 23(2):169–187, OCT 2007.
  * [88] A Szucs, R Huera, MI Rabinovich, and AL Silverston.  Robust microcircuit synchronization by inhibitory connections.  Neuron, 78(61):439–453, 2009.
  * [89] S Daun, JE Rubin, and IA Rybak.  Control of oscillation periods and phase durations in half-center central pattern generators: a comparative mechanistic analysis.  Journal of Computational Neuroscience, 27(1):3–36, 2009.
  * [90] R Nagornov, G Osipov, M Komarov, A Pikovsky, and AL Shilnikov.  Mixed-mode synchronization between two inhibitory neurons with post-inhibitory rebound.  Communications in Nonlinear Science and Numerical Simulation, 36:175–191, 2016.
  * [91] S Angrist.  Perpetual motion machines.  Scientific American, 218(1):115–122, 1968.
  * [92] AL Shilnikov and G Cymbalyuk.  Transition between tonic spiking and bursting in a neuron model via the blue-sky catastrophe.  Phys Rev Lett, 94(4):048101, Feb 2005.
  * [93] AL Shilnikov.  Complete dynamical analysis of a neuron model.  Nonlinear Dynamics, 68(3, SI):305–328, may 2012.