XX XX 8 Month Publication Title

FEATURE ARTICLE: Garment modelling, Sewing pattern design,
Surface flattening

# PerfectTailor: Scale-Preserving 2D Pattern Adjustment Driven by 3D Garment
Editing

Anran Qi and Takeo Igarashi  The University of Tokyo, Tokyo, 113-0033, Japan

(2024)

###### Abstract

We address the problem of modifying a given well-designed 2D sewing pattern to
accommodate garment edits in the 3D space. Existing methods usually adjust the
sewing pattern by applying uniform flattening to the 3D garment. The problems
are twofold: first, it ignores local scaling of the 2D sewing pattern such as
shrinking ribs of cuffs; second, it does not respect the implicit design rules
and conventions of the industry, such as the use of straight edges for
simplicity and precision in sewing. To address those problems, we present a
pattern adjustment method that considers the non-uniform local scaling of the
2D sewing pattern by utilizing the intrinsic scale matrix. In addition, we
preserve the original boundary shape by an as-similar-as-possible geometric
constraint when desirable. We build a prototype with a set of commonly used
alteration operations and showcase the capability of our method via a number
of alteration examples throughout the paper.

††journal: Publication Name

Current garment alternation design is mostly centered around 2D sewing pattern
space which involves numerous pattern editing operations to achieve the
envisioned alterations of 3D garments. In practice, achieving the correct
pattern adjustment not only necessitates specialized expertise in garment
design but is also time-consuming. This is because designers need to justify
both the envisioned 3D geometric changes of the garment and the embedded
intrinsic design, such as smocking, elastic threading design, and _etc_.

Figure 1: (a) Two 3D garments with hard yellow indicating the corresponding
part to (b) the original panel designs. Panels flattened by (c) Sheffer et
al.11 (d) Igarashi et al.4 and (e) our method. (c) and (d) generate smaller
panels than the original panel (top row) and irregular boundaries (bottom
row). In both cases, our method can produce the same panel shapes as that of
the original panels.  Figure 2: A user alteration example. (a) A 3D garment
model from the front and back view (top row) and its corresponding sewing
pattern (bottom row). The matching colour indicates the garment and sewing
pattern correspondence. It is worth mentioning that the bottom of the sleeve
(hard yellow) fits tighter than the upper part (soft yellow) due to its
elastic string design, while their corresponding pattern has a similar width.
(b) Our function sets allow the user to alter the garment directly in 3D space
based on their preference and our pattern adjustment method updates the
pattern accordingly. Top row: the 3D garment edits by the user; Bottom row:
the pattern adjustment results (Dash lines illustrate the original panel
shape). For Step 1: scale operation, we show the pattern adjustment results of
Igarashi et al.4 which is a naive geometric surface parametrization technique
and ours. Igarashi et al.4 generates a much smaller panel due to directly
flattening the surface. Thus unlike us, it is not able to preserve the
original embedded design. Our system incorporates the left-right symmetry in
garment design literature, _i.e.,_ the user only needs to edit on a single
side of the garment and our system automatically mirrors those edits across
the other side. (c) The garment is simulated by the pattern from Igarashi et
al.4 (top row) and ours (bottom row). The smaller panel produced by Igarashi
et al.4 causes the tearing of the sleeve highlighted in the black frame box.

To speed up the design process and reduce the required expertise, researchers
have proposed many powerful methods1,2 to edit the garment directly in 3D
space, and then automatically adjust the 2D pattern accordingly. Those methods
usually assume the pre-existence of the sewing patterns. This matches the
practice — designers often start with an existing pattern to create either
real or virtual outfits. As a well-designed sewing pattern has already
undergone design and development processes, it can significantly streamline
the design process for designers, saving time and resources in comparison to
developing a new pattern from scratch. However, the existing methods use
uniform flattening (geometric surface parametrization) to update the sewing
pattern after 3D editing. This is suboptimal for the following two reasons.
Firstly, it is grounded in the hypothesis that the local sizes of both 2D and
3D triangles in the sewing pattern and 3D garment design remain mostly
constant. For certain secondary textile designs like smocking and elastic
thread design, the 3D triangle on the garment shrinks or expands due to
embedded stitching force during the simulation and draping, which makes the
local sizes of 2D and 3D triangles non-uniform. Thus, uniformly flattening the
3D garment surface leads to misestimating the sewing panel size (see Figure 1
top row). Secondly, this technique usually produces panels with irregular
boundaries. This neglects the implicit rules and conventions in the industry’s
design practices, such as straight edges, symmetric shapes and _etc._ (see
Figure 1 bottom row).

Unlike previous work, we propose a pattern adjustment method specifically for
synchronizing a well-designed 2D pattern according to the user’s edits in 3D
space. Our method considers the non-uniform local scaling of the 2D sewing
pattern and respects the implicit rules and conventions of the industry.
Specifically, our method memorizes the local intrinsic scale matrix between
the 2D pattern and the 3D drape of the initial design and uses it when
updating the 2D pattern to compensate for the user’s edit in 3D space. In
addition, we propose an as-similar-as-possible constraint to preserve the
original panel boundary shape when the user modifies the entire panel. With
our adjustment method, we develop a set of commonly-used editing operations to
support garment alteration, such as scaling a specific part for fitting,
extending and shortening the along the boundary, and cutting to achieve the
desired shape (see Figure 2) and demonstrate alteration results on a number of
garments, showing its usability and generalizability.

Figure 3: Computation flow of our flattening method.  (1) (2) and  (3)
indicate the calculation of Equation 1, Equation 2 and Equation 3,
respectively.

## RELATED WORK

### Garment Design

Various computational garment design techniques offer tools to automate the
adjustment of underlying 2D patterns. These methods contribute to accelerating
the design workflow and minimizing the need for extensive expertise in the
field. Sensitive Couture9 proposes a bidirectional interactive garment edit
method by leveraging the fast simulation technique. It begins with well-
designed sewing patterns and supports a subset of garment edits in 3D but is
restricted to the dragging of vertices and edges.

Pietroni et al.5 automatically generates the 2D sewing pattern from an input
3D shape by first creating the panel patch layout and then flattening the
patch considering the anisotropic material property of the woven fabric. Wolff
et al.10 optimizes the 3D rest shape of the garment to maximise the fit and
comfort under a range of poses and body shapes. The corresponding sewing
pattern is generated by Sharp et al.6 which directly optimizes the distortion
in the cutting and flattening process. These methods applies uniform
flattening, thus cannot handle non-uniform local scaling of the sewing pattern
and geometric constraints on boundary shapes.

Bartle et al.1 proposes a fixed-point iteration method to optimize the pattern
by minimizing the distortion between physical simulation results and the
target draped garment so that it will generate the target garment geometry
after simulation in the context of direct garment editing. The optimization
accounts for pattern deformation caused by the physical forces during draping.
However, physical simulation and optimization take time, requiring certain
effort (e.g., parameter tuning) to make it work. Our method bypasses physical
simulation by taking a purely geometric approach, which is faster and more
stable. Combining our method with theirs can be an interesting future work.

### Surface Flattening

Surface flattening (i.e., parametrization) methods take a surface with disk
topology and aim to optimize its 2D mapping based on the defined distortion
measurement. This is a fundamental and well-studied problem in computer
graphics. We refer the reader to the survey8 for a more complete background.
Here we review a few works that are commonly used in the garment design
literature for completeness. Those works can be roughly classified into two
categories: geometry-based methods4,11,12,13 and physics-based methods.14,15

Geometry-based methods formulate the surface flattening as a distortion
minimization problem based on either vertex, edge, or face. Sheffer et al.11
defines an angle preservation metric and a set of constraints on the angles to
ensure the validity of the flat mesh. Then formulate it as a constrained
minimization problem. As-rigid-as-possible methods use a local-global
optimization approach to optimize an isometric distortion measurement defined
based on a set of edges12 or triangles.4,13

Physics-based methods use the elastic energy model to drive the deformation of
each 2D triangle. McCartney et al.14 proposes to use a relatively simple
elastic model to calculate strain energy to measure the movement of vertices
in each 2D triangle. Later, Wang et al.15 converts the energy into force and
utilizes it within the Lagrange equation to calculate the movement of each 2D
vertex.

Different from the aforementioned works which flatten the 3D surface from
scratch, we focus on flattening the surface with an initial condition to keep
the original design intention.

## METHOD

Our key observation is the uniform flattening of the garment surface will
dissipate the embedded initial design in the pattern. To this end, we propose
to memorize the local scale matrix between the 2D pattern and the 3D drape of
the initial design and use it when updating the 2D pattern to accommodate user
editing on the 3D drape. Specifically, we seek a pattern adjustment method
that transforms the original 2D pattern $X$ with 2D triangles $S$ to the new
2D pattern $X^{edit}$ with 2D triangles $T^{edit}$, which will follow the
user’s edit from the original 3D garment $x$ with 3D triangles $s$ to the
edited 3D garment $x^{edit}$ with 3D triangle $s^{edit}$, whilst keeping the
originally embedded design (see Figure 3).

Motivated by Bartle et al.,1 we aim at modelling the initially embedded local
scaling as an _intrinsic scale matrix_ and keep the matrix during the updating
process. Since this matrix is invariant to rigid transformations, we can
consider first projecting the 3D triangle $s_{i}$ of the 3D garment onto the
2D $x\mbox{--}y$ plane, denoted as $t_{i}$. Then rotate the $S_{i}$ to align
the $t_{i}$, denoted as $T_{i}$, so that a designated edge corresponded in
$t_{i}$ and $T_{i}$ is aligned with the $x$-axis (see Figure 3). Now we can
formulate each triangle as

$\begin{pmatrix}\lvert u\rvert&\lvert v\rvert sin(\theta)\\\ 0&\lvert v\rvert
cos(\theta)\end{pmatrix}$ (1)

where $u$ and $v$ are edge vectors and $\theta$ is the angle between them. We
model the intrinsic scale matrix as a 2D transformation $M$:

$\begin{split}M_{i}=t_{i}T_{i}^{-1}\end{split}$ (2)

At each time, the user edits the 3D garment, we update the 3D garment geometry
based on well-established geometric rules and get the edited 3D triangle
$s_{i}^{edit}$ with its 2D projection $t_{i}^{edit}$.

Recap that our goal is to keep the intrinsic scale matrix during the updating
process. To do this, we seek to find the optimal 2D triangles $T^{edit}$ that
when multiplied with this matrix will produce the target triangles $t^{edit}$:

$\begin{split}t_{i}^{edit}=M_{i}T_{i}^{edit}\end{split}$ (3)

Thus, we update the $T_{i}^{edit}$ as

$\begin{split}T_{i}^{edit}=M_{i}^{-1}t_{i}^{edit}=T_{i}t_{i}^{-1}t_{i}^{edit}\end{split}$
(4)

Stitching After the updating stage, the new triangle has the desired property.
However, each triangle is processed independently, so it is unlikely that they
will not form a continuous 2D mesh pattern. We, therefore, need to stitch all
the triangles together to form a valid pattern $S^{edit}$. This is a well-
studied mesh parametrization problem that can be solved using geometry-based
surface flattening methods. We use the as-rigid-as-possible surface flattening
technique4 to get our final results.

After extensive deliberations with professional garment designers, we decided
to preserve the original pattern shape, maintaining the integrity of the
initial pattern when the user edit will affect the entire sewing panel
equally. Therefore, we propose

an as-similar-as-possible constraint to preserve the discrete tangent of
boundary vertices $v_{i}^{tan}$ expressed as
$\frac{v^{x}_{(i+1)}-v^{x}_{(i-1)}}{v^{y}_{(i+1)}-v^{y}_{(i-1)}}$ in the
stitching process (inset). We define final the quadratic error function as

$\begin{split}&\arg\min\limits_{v^{\prime}\in V}\sum_{(i,j)\in
E}\|(v^{\prime}_{j}-v^{\prime}_{i})-(v_{j}-v_{i})\|^{2}\\\ &+w_{1}\sum_{i\in
C}\|(v^{\prime}_{i}-C_{i})\|^{2}+w_{2}\sum_{i\in
B}\|(v^{tan^{\prime}}_{i}-B_{i}^{tan})\|^{2}\end{split}$ (5)

where $v_{i}$ and $v_{i}^{{}^{\prime}}$ are the vertex coordinate of the
triangle in the original pattern ($T_{i}$) and the new targeted pattern
($T_{i}^{edit}$), respectively. $E$ is a set of edges, $C$ is a set of fixed
vertices, $C_{i}$ is the fixed vertex coordinate, $B$ is the set of boundary
vertices and $B_{i}$ is the the boundary vertex coordinate. We fix the
endpoints of an edge at the center of the original pattern and use
$w_{1}=w_{2}=1000$ for the examples in this paper.

## SYSTEM OVERVIEW and IMPLEMENTATION

### User Interface

The system has two windows: the 3D window and the 2D pattern window for
displaying the 3D garment and 2D sewing pattern, respectively (see Figure 4).
The user can edit the 3D garment using editing operations provided by the
system in the 3D window. In the 2D pattern window, the user can freely move
the panel to adjust the pattern’s layout by clicking. The top shows the
system’s editing functions. Since interacting with these editing functions is
straightforward and hence not described here. Please see the accompanying
video for details. In the following section, we detail the supported
operations and illustrate them with multiple examples.

Figure 4: A screen snapshot of the system. The left (beige) is the 3D window
displaying the 3D garment on a human body. Red dots indicate key feature
points on the surface of the human body, e.g., front neck point, and busty
points. The right is the 2D pattern window showing the sewing pattern. The top
shows the functions of the system. Figure 5: Scale the internal sleeve part by
editing the seam line. (a) The red stitches visualize the seam line of the
lantern sleeve. (b) The user edits the seam line to scale the internal sleeve
(soft yellow) perpendicular to the human body whilst fixing the bottom part
around the wrist. (c) The scaled internal sleeve (hard yellow). (d) The
original sleeve panel. (e) The updated panel by Igarashi et al.4 (f, g) The
updated panel by our method without/with the as-similar-as-possible
constraint, respectively. Compared with (g), Igarashi et al.4 (e) has a
slightly larger pattern with the severely shrunk lower boundary. This is
because it directly flattens the deformed 3D triangles, which are deformed due
to stitching force, gravity, and _etc_ in the simulation. (f) has the right
size but its curved boundary damages the original design. With our as-similar-
as-possible constraint, (g) has the right size whilst maintaining the original
panel’s boundary design.

### Editing Operations

Starting with the input 3D garment and the corresponding pattern, we implement
a set of simple and commonly used alteration operations that allow the user to
explore the complex redesign space:

Scale: The user can select a part on the 3D garment and scale either along
(see Figure 6 top row) or perpendicular (see Figure 6 bottom row) to the human
skeleton direction by dragging. Furthermore, our system also supports the
scaling of the internal garment part leveraging the seam line. In the garment
literature, a seam line refers to the line or path created by joining two or
more pieces of fabric together using stitches (see Figure 5 (a)). Our system
utilizes the seam line to allow the user to scale the internal garment part
either along (see Figure 5) or perpendicular (see Figure 7) to the human body.
In such instances, after thorough discussions with the designers, we opt to
update the whole sewing panel using our proposed flattening method with the
as-similar-as-possible constraint to preserve the original panel’s boundary
shape. The rationale behind this is the user edit will affect the entire
sewing panel equally, thus it is optimal to maintain the original panel’s
boundary design as closely as possible.

Figure 6: The user scales the bottom part of the garment perpendicular to the
human skeleton making it looser (top row), along with the human skeleton
making it longer (bottom row). (a) The original garment. Soft yellow indicates
the panels affected by the user edits. (b) The updated garment geometry. Dark
yellow indicates the parts selected by the user, being customised. (c) The
original panels. (d) The updated panels (Dash lines illustrate the original
panel shape). Figure 7: Scale the internal sleeve part along the body by
editing the seam line. Top row: The user moves the seam line upwards making
the internal sleeve shorter. Bottom row: The user moves the seam line upwards
whilst fixing the lower boundary of the bottom sleeve. This leads to a shorter
internal sleeve but a longer bottom sleeve. The red circle in (b) highlights
the difference. (a) The original garment. Soft yellow indicates the panels
affected by the user edits. (b) The updated garment geometry. Dark yellow
indicates the parts selected by the user and being customised. (c) The
original panels. (d) The updated panels.

Cut: We allow the user to sketch on the 3D garment to cut the garment into the
desired shape. By default the user sketches from one viewpoint, and we will
automatically form a closed loop as the cutting line on the 3D garment to cut
both sides. We also enable the user to only cut one side if the user
specifies. Then we re-triangulate the meshes affected by the new cutting line
and update the sewing pattern by directly transferring the barycentric
coordinate of the new vertex into the local coordinate system of the
corresponding 2D triangle. For the detailed algorithm, we refer the reader to
the Teddy system.3

Shorten: The user can drag the boundary to the desired position to shorten the
garment (see Figure 2 (b) Step 3 and Figure 8). In detail, we compute the iso-
line on the garment surface mesh where the distance to the boundary is the
user-specified shortened distance. We take the computed iso-line as the
cutting line on the 3D garment and utilize the Cut function to shorten the
garment.

Figure 8: Shorten example. The user iteratively shortens the garment four
times to explore various collar designs. Red curves on (a) and (b) indicate
the cutting lines on the 3D garment and sewing pattern, respectively.

Extend: The user can also extend the garment by dragging the boundary to the
desired position (see Figure 2 (b) Step 2 and Figure 9). Following the
observation from Brouet et al.2, people tend to preserve the slope and tangent
plane orientations across the garment surface when transferring the garment.
We follow the same principle by appending the triangle faces which share the
same surface norm with the connected triangle faces. Body-garment collisions
sometimes occur when extending. We resolve it by pushing the vertex towards
the norm direction of the nearest triangle on the body surface. Finally, we
update the sewing pattern in the same way as that of the Cut function.

Figure 9: The user extends the V-shape neck collar.

F5 Tighten/Loosen: We also allow the user to tighten/loosen the garment by
directly over-sketching the silhouette of the 3D garment. The system deforms
the garment to meet the silhouette and updates the pattern with our proposed
flattening method. For the detailed 3D deformation algorithm, we refer the
reader to the Nealen et al.16

Those editing operations allow the user to alter garments and showcase the
capabilities of our proposed pattern adjustment method, but more operations
could be added to enhance the redesign capacity such as adding folds and
darts.

## CONCLUSION

We present a pattern adjustment method that aims to preserve the embedded
design for garment alteration. Besides, we develop a set of editing operations
to support alteration and showcase the capability of our adjustment method and
functions via several examples throughout the paper. The editing operations
introduced in this paper are not exhaustive, the the core algorithm, "scale-
preserving flattening" is general and can be applied to other 3D modelling
operations.

## ACKNOWLEDGMENT

This work is supported by SHIMA SEIKI MFG., LTD. This work is also partially
supported by JST CREST (JPMJCR17A1).

## References

  * [1] A. Bartle, A. Sheffer, V. G. Kim, D. M. Kaufman, N. Vining, and F. Berthouzoz, “ Physics-driven pattern adjustment for direct 3D garment editing,” ACM Transactions on Graphics (TOG)., vol. 35, no. 4, pp. 50–1, 2016. (journal)
  * [2] R. Brouet, A. Sheffer, L. Boissieux, and M. P. Cani, “ Design preserving garment transfer,” ACM Transactions on Graphics (TOG)., vol. 35, no. 4, pp. 1–11, 2012. (journal)
  * [3] T. Igarashi, S. Matsuoka, and H. Tanaka, “Teddy: a sketching interface for 3D freeform design,” ACM Transactions on Graphics (TOG)., pp. 409–-416, 1999. (journal)
  * [4] T. Igarashi, T. Moscovich, and J. F. Hughes, “As-rigid-as-possible shape manipulation,” ACM Transactions on Graphics (TOG)., vol. 24, no. 3, pp. 1134–1141, 2005. (journal)
  * [5] N. Pietroni, C. Dumery, R. Falque, M. Liu, T. Vidal-Calleja, and O. Sorkine-Hornung, “Computational pattern making from 3D garment models,” ACM Transactions on Graphics (TOG)., vol. 41, no. 4, pp. 1–14, 2022. (journal)
  * [6] N. Sharp, and K. Crane, “Variational surface cutting,” ACM Transactions on Graphics (TOG)., vol. 37, no. 4, pp. 1–13, 2018. (journal)
  * [7] A. Sheffer, B. Lévy, M. Mogilnitsky, and A. Bogomyakov, “ABF++: fast and robust angle based flattening,” ACM Transactions on Graphics (TOG)., vol. 24, no. 2, pp. 311–330, 2005. (journal)
  * [8] A. Sheffer, E. Praun, and K. Rose, “Mesh parameterization methods and their applications,” Foundations and Trends® in Computer Graphics and Vision., vol. 24, no. 2, pp. 105–171, 2007. (journal)
  * [9] N. Umetani, D. M. Kaufman, T. Igarashi, and E. Grinspun, “Sensitive couture for interactive garment modeling and editing,” ACM Transactions on Graphics (TOG)., vol. 30, no. 4, pp. 1–12, 2011. (journal)
  * [10] K. Wolff, P. Herholz, V. Ziegler, F. Link, N. Brügel, and O, Sorkine-Hornung, “Designing Personalized Garments with Body Movement,” Computer Graphics Forum., vol. 42, no. 1, pp. 180–194, 2023. (journal)
  * [11] A. Sheffer, B. Lévy, M. Mogilnitsky, and A.Bogomyakov, “ABF++: fast and robust angle based flattening,” ACM Transactions on Graphics (TOG)., vol. 24, no. 2, pp. 311–330, 2005. (journal)
  * [12] O. Sorkine, and M.Alexa, “As-rigid-as-possible surface modeling,” Symposium on Geometry processing. Vol. 4, pp. 109–116, 2007. (journal)
  * [13] L. Liu, L. Zhang, Y. Xu, C. Gotsman, and S.J. Gortler, “A local/global approach to mesh parameterization,” Computer graphics forum., Vol. 27, No. 5, pp. 1495–1504, 2008. (journal)
  * [14] J. McCartney, B.K. Hinds, and B.L. Seow, “The flattening of triangulated surfaces incorporating darts and gussets,” Computer-Aided Design., Vol. 31, No. 4, pp. 249–260, 1999. (journal)
  * [15] C.C. Wang, S.S. Smith, and M.M. Yuen, “Surface flattening based on energy model,” Computer-Aided Design., Vol. 34, No. 11, pp. 823–833, 2002. (journal)
  * [16] A. Nealen, O. Sorkine, M. Alexa, and D. Cohen-Or, “A sketch-based interface for detail-preserving mesh editing,” ACM Transactions on Graphics., pp. 1142–1147, 2005. (journal)

Anran Qi is a Project Assistant Professor at the Graduate School of
Information Science and Technology at The University of Tokyo. Her research
focuses on garment design and human sketch understanding from graphics,
machine learning, and human-computer interaction perspectives. She received a
Ph.D from the Centre for Vision, Speech and Signal Processing (CVSSP),
University of Surrey in 2021. Contact her at<EMAIL_ADDRESS>Takeo
Igarashi is a Professor of Computer Science Department at The University of
Tokyo. His research interest is in user interfaces and interactive computer
graphics in general. He received a Ph.D. from the Department of Information
Engineering at The University of Tokyo in 2000. He received the SIGGRAPH
Significant New Researcher Award, JSPS Award, CHI Academy Award, and the Asia
Graphics 2020 Outstanding Technical Contributions Award. Contact him at
<EMAIL_ADDRESS>


# Ev-NeRF: Event Based Neural Radiance Field

Inwoo Hwang1, Junho Kim1, and Young Min Kim1,2,
1Department of Electrical and Computer Engineering, Seoul National University
2Interdisciplinary Program in Artificial Intelligence and INMC, Seoul National
University
Young Min Kim is the corresponding author.

###### Abstract

We present Ev-NeRF, a Neural Radiance Field derived from event data. While
event cameras can measure subtle brightness changes in high frame rates, the
measurements in low lighting or extreme motion suffer from significant domain
discrepancy with complex noise. As a result, the performance of event-based
vision tasks does not transfer to challenging environments, where the event
cameras are expected to thrive over normal cameras. We find that the multi-
view consistency of NeRF provides a powerful self-supervision signal for
eliminating spurious measurements and extracting the consistent underlying
structure despite highly noisy input. Instead of posed images of the original
NeRF, the input to Ev-NeRF is the event measurements accompanied by the
movements of the sensors. Using the loss function that reflects the
measurement model of the sensor, Ev-NeRF creates an integrated neural volume
that summarizes the unstructured and sparse data points captured for about 2-4
seconds. The generated neural volume can also produce intensity images from
novel views with reasonable depth estimates, which can serve as a high-quality
input to various vision-based tasks. Our results show that Ev-NeRF achieves
competitive performance for intensity image reconstruction under extreme noise
and high-dynamic-range imaging.

##  1 Introduction

Event cameras are neuromorphic sensors, where individual pixels detect changes
of brightness that exceed a threshold. The output of event cameras is a
sequence of asynchronous events composed of the polarity, pixel location, and
the time stamp, occurring only at a sparse set of locations where the
brightness change is detected. They have many advantages over conventional
cameras such as high temporal resolution, low energy consumption, and high
dynamic range [15]. However, the measurements of the same object change
significantly under different motion or lighting conditions causing domain
discrepancy in real-world deployment [24, 12, 64]. While event cameras are
expected to prosper under extreme environmental conditions, the performance of
event-based vision tasks often deteriorates due to the significant domain
shift with severe noise.

Figure 1: (a) Ev-NeRF operates with event data obtained from a moving event
camera. (b) Ev-NeRF learns the implicit volume with the raw event output of
the sensor and serves as a solution for various event-based applications, such
as high dynamic range imaging, noise reduction, depth estimation, intensity
image reconstruction, and novel-view intensity image reconstruction.

The output of event streams is very different from an ordinary image, which is
a two-dimensional array with dense color values. Many existing approaches
using event data compile them into a more structured form for denoising [10,
11, 30, 14, 2, 13], or directly perform downstream tasks such as motion
estimation [34, 29, 37, 53] or pose estimation [35, 5]. Nonetheless, training
data is often limited and the performance of event-based vision is often
inferior to the performance of the same tasks with conventional images [24].
The complex noise characteristics and domain discrepancy further complicate
developing practical algorithms for event cameras.

Inspired by the recent success of Neural Radiance Fields (NeRF) [33], we
propose Ev-NeRF, a neural radiance field built directly from raw event data,
as shown in Figure 1(a). Ev-NeRF builds a 3D volumetric representation that
can concurrently explain events associated with the camera movement. Given the
5D input of location and viewing direction, NeRF outputs the volume density
and emitted color, which can be aggregated to synthesize an image from an
arbitrary viewpoint by the volume rendering. While NeRF is trained to minimize
the color discrepancy between the synthesized image and the ground truth
image, Ev-NeRF is trained with a new loss function that incorporates the
sensor movement and the resulting events triggered by the difference of
brightness.

Ev-NeRF properly handles the complex noise in event cameras without ground
truth supervision, and at the same time, enjoys the technical advantages of
the sensor over conventional cameras. The volumetric aggregation in the
formulation effectively reduces the prevalent noise in event measurements
[15], as the complex spatial and temporal noises lack multi-view consistency.
Further, the associated intensity values in Ev-NeRF are in a high dynamic
range (HDR), as the provided measurements of event cameras are sensitive to
extreme lighting beyond the dynamic range of the conventional camera.

Interestingly, the created volumetric representation is a solution to many of
the vision problems tackled in previous works using event data, as shown in
Figure 1(b). While the event data only contain the relative changes in the
brightness instead of the absolute term, the trained volume can synthesize the
intensity image for ordinary computer vision, which is one of frequently
tackled problems in the community [9, 22, 3, 45, 44, 43, 38, 54, 63, 47, 58,
6, 57]. Further, the reconstructed density volume can represent the
approximate 3D structure of the scene. This is inherent from the original NeRF
formulation enforcing the multi-view consistency, and the quality of 3D
reconstruction is superior to the 3D structure built from previous approaches
[23, 42, 66].

Our contributions can be summarized as follows:

  * •

We suggest Ev-NeRF, which combines the popular NeRF formulation with the raw
event output of a neuromorphic camera for the first time.

  * •

Ev-NeRF is highly robust to event noise and builds a coherent 3D structure
that can provide high-quality observations.

  * •

The created neural volume serves as solutions for various event-based
applications, namely intensity image reconstruction, novel-view image
synthesis, 3D reconstruction, and HDR imaging.

  * •

Ev-NeRF demonstrates performance comparable to many of existing event-vision
algorithms that are dedicated to a specific task in the experimental result.

Given the strong experimental result, we expect Ev-NeRF to expand the possible
application area of event-based vision that fully leverages the potential of
the sensor.

##  2 Related works

In this section, we review the key tasks in event-based vision, along with
existing work on neural implicit 3D representations.

#### Processing Event Data

Although event cameras can acquire visual information in challenging
conditions such as low-lighting or extreme motion, a significant domain gap
occurs due to the large amount of noise which further leads to performance
degradation [24, 50, 40, 12, 25]. Wu et al. [64] first demonstrated that
event-based vision can deteriorate due to increased noise levels, although the
assessments were mainly conducted in synthetic events. Kim et al. [24] further
introduced a large-scale dataset enabling systematic assessment of object
recognition tasks, and demonstrated that large camera motion or illumination
change leads to greater amounts of noise that ultimately deteriorates
performance. Existing approaches denoise the raw data to cope with such
adversaries [64, 59, 60], or suggest stacking events to overcome domain gaps
under extreme lighting condition [51]. On the other hand, Ev-NeRF can
compensate for the spurious noises by enforcing multi-view consistency for the
scene geometry.

Instead of handling complex data characteristics from the raw data, many
approaches aggregate the sequential measurements into an ordinary image or 3D
geometry. Early attempts for intensity image reconstruction are inspired from
statistical methods [9, 22, 3]. Several subsequent approaches suggest various
network architecture designs to improve the image quality or computational
cost [47, 58, 6, 57, 63]. Because the sensor has a high dynamic range, the
intensity image restoration can be explicitly designed for HDR images [67, 58]
or by applying domain adaptation to day-light condition [52]. For estimating
3D geometry, recent event-based SLAM methods utilize classical techniques [23,
22, 16, 21, 66, 17], minimizing the energy function formulated over the image-
like event representations. On the other hand, for event-based depth
estimation both classical method [19, 61] and learning-based approaches [20,
56] coexist.

However, for any of the aforementioned tasks, it is challenging to obtain a
large-scale dataset with the ground-truth label or formulate the correct
measurement model that represents the wide range of possible sensor
characteristics. [44, 45] suggested generating the training data using
simulator [43]. [54] examined the statistical aspect to reduce the gap. [38]
proposed a self-supervised learning framework with the aid of optical flow and
does not require ground truth, but their reconstructed images are
characterized by several artifacts. On the contrary, Ev-NeRF works without the
ground truth or synthetic data and shows stable results comparable to the
state-of-the-art in intensity image reconstruction or depth estimation.

#### Neural Implicit 3D Representation

Neural Implicit 3D Representation is gaining popularity due to its strong
advantage of memory requirements, no restrictions on spatial resolution, and
representation capability. Several works [39, 31, 8] showed the advantage of
neural implicit representations with 3D supervision. NeRF (Neural Radiance
Fields) [33] proposes an implicit representation of 3D coordinates and viewing
direction which can synthesize images with volume rendering techniques. The
resulting neural volume contains information about 3D volume density and
emitted radiance for rendering images. Motivated by the photo-realistic
quality of the produced images, a large number of subsequent works spurred to
overcome the limitation of the original NeRF including: enabling fast
convergence and rendering [36, 55, 1]; handling the input images with unknown
or noisy camera poses [62, 28]; recovering hdr scenes with noisy raw images
[32]; or processing dynamic scenes [41, 27]. Our method learns the NeRF volume
with event data. By enforcing the multi-view consistency of collected
measurements, Ev-NeRF produces a high-quality image or depth in a novel view
and effectively removes spurious noises of an event camera. While there also
exists a concurrent work learning implicit volume with events [46], Ev-NeRF
extensively reveals the practical capability.

Figure 2: Overview of our method. According to the measurement model of the
sensor, the events $\mathcal{E}_{j}$ accumulated during a short time interval
$[T_{j},T_{j+1})$ should reflect the difference in brightness. Using the
implicit volume, we render intensity frames from the view points of two
adjacent event camera poses, $\mathcal{T}_{j}$ and $\mathcal{T}_{j+1}$. Event
rendering loss is the discrepancy between the accumulated event $B_{j}(r)$ and
difference in the intensity of adjacent rendered frames $\Delta L_{j}(r)$.

##  3 Background

For the completeness of discussion, we include the event generation model of
the sensor followed by the mathematical formulations of the neural radiance
fields (NeRF), which serves as the two main components for deriving Ev-NeRF.

#### Event Generation Model

Instead of recording the absolute color values of the image pixels, an event
camera records asynchronous changes of the brightness as a sequence of events
$E_{k}=\left(u_{k},v_{k},t_{k},p_{k}\right)$, indicating that the brightness
change at the pixel coordinate $\left(u_{k},v_{k}\right)$ reaches a specific
threshold $B$ at time $t_{k}$,

$|L\left(u_{k},v_{k},t_{k}\right)-L(u_{k},v_{k},t_{k}-\delta
t)|\geq\left|B\right|,$ (1)

where $L=\log\left(I\right)$ is the logarithm of brightness $I$ and $\delta t$
is time that has passed since the last event. $p_{k}\in\\{+,-\\}$ is the
polarity denoting whether the brightness change is positive or negative. It is
known that the threshold for triggering positive events is different from the
one for the negative events [15], which we denote $B^{+}$ and $B^{-}$,
respectively. If we accumulate the events occurring for a given period of time
$\Delta t$, the brightness change in a specific pixel can be approximated by
[15]

$\Delta L\left(u,v,\Delta t\right)=\sum_{\begin{subarray}{c}t_{k}\in\Delta
t,\\\ (u_{k},v_{k})=(u,v)\end{subarray}}p_{k}\left|B^{p_{k}}\right|.$ (2)

The threshold $B^{p_{i}}$ can be different under various physical conditions,
which further challenges the event-based vision, in addition to complex noise
characteristics of the sensor.

#### Neural Radiance Fields

Ev-NeRF takes inspiration from NeRF [33] which is trained to accumulate the
volumetric information with 2D supervision. The supervision signal for NeRF is
the total squared error between the rendered and true pixel colors. Basically
the neural network $F_{\theta}(\cdot)$ receives the input of the 3D coordinate
$\mathbf{x}_{i}\in\mathbb{R}^{3}$ and the ray direction
$\mathbf{d}_{i}\in\mathbb{S}^{2}$ and outputs the density
$\sigma_{i}\in\mathbb{R}$ and emitted radiance
$\textbf{c}_{i}\in\mathbb{R}^{3}$

$F_{\theta}:(\gamma_{x}(\mathbf{x}_{i}),\gamma_{d}(\mathbf{d}_{i}))\rightarrow(\sigma_{i},\mathbf{c}_{i}).$
(3)

Here $\gamma\left(\cdot\right)$ is sinusoidal positional encoding function
which successfully captures the high-frequency information along the spatial
direction. With positional encoding and coarse-to-fine sampling techniques, a
neural network is trained to synthesize high-quality novel view images.

Following the classical volume rendering technique, each pixel is rendered by
sampling $N$ points $\mathbf{x}_{1},\ldots,\mathbf{x}_{N}$ of the volume
density along the ray $r(\mathbf{x}_{0},\mathbf{d})$. $\mathbf{x}_{0}$ is the
initial point of the ray located at the focal point of the camera using the
pinhole camera model. The final rendered color of the pixel is aggregated
along the ray as

$\hat{C}\left(r\right)=\sum^{N}_{i=1}A_{i}\alpha_{i}\textbf{c}_{i}.$ (4)

$A_{i}=\exp\left(-\sum^{i-1}_{l=1}\sigma_{l}\delta_{l}\right)$ denotes the
accumulated transmittance along the ray, and
$\alpha_{i}=1-\exp\left(-\sigma_{i}\delta_{i}\right)$ denotes the alpha value,
where $\delta_{i}=\left\|\mathbf{x}_{i+1}-\mathbf{x}_{i}\right\|$ is the
distance between adjacent samples. Additionally, the depth along the ray
direction can be approximated with a similar formulation:

$\hat{D}\left(r\right)=\sum^{N}_{i=1}A_{i}\alpha_{i}s_{i},$ (5)

where $s_{i}$ denotes the distance between $\mathbf{x}_{0}$ and
$\textbf{x}_{i}$.

##  4 Method

Ev-NeRF creates neural implicit representation $F_{\theta}$ of a static scene
as NeRF. Since an event is triggered when the brightness changes, we use a
slice of the event sequence
$\mathcal{E}_{j}=\\{E_{k}=(u_{k},v_{k},p_{k},t_{k})|T_{j}\leq
t_{k}<T_{j+1}\\}$ during a small duration of time $[{T}_{j},{T}_{j+1})$. The
motion of the camera is provided by the starting and ending poses of the
period, $\mathcal{T}_{j}$ and $\mathcal{T}_{j+1}$. The neural network for Ev-
NeRF regresses for one dimensional emitted luminance value
$y_{i}\in\mathbb{R}$ instead of RGB color values,

$F_{\theta}:\left(\gamma_{x}\left(\mathbf{x}_{i}\right),\gamma_{d}\left(\mathbf{d}_{i}\right)\right)\rightarrow\left(\sigma_{i},y_{i}\right).$
(6)

This is a natural choice considering that the event only records the
brightness change in a single channel.

After the neural network is trained, we can render the intensity image from an
arbitrary viewpoint adapting Equation 4. To elaborate, if we define the camera
ray $r$ that passes through the pixel location $(u,v)$ from a camera pose
$\mathcal{T}$, we can sample $N$ points along the ray and apply volume
rendering technique to find the intensity of the pixel

$\hat{I}(\mathcal{T}(r))=\sum_{i=1}^{N}A_{i}\alpha_{i}y_{i}.$ (7)

The depth measurement can also be approximated using Equation 5.

When we combine the formulation with the event generation model, we also
jointly optimize for the unknown thresholds, $B^{+}_{j}$ and $B^{-}_{j}$, in
addition to the implicit neural volume $F_{\theta}(\cdot)$. We assume that the
threshold is a function of time and polarity, but is spatially the same for
all pixels. More specifically, we assume that the threshold is constant in
each time interval $[{T}_{j},{T}_{j+1})$, but changes when the time interval
changes.

The total loss used to train Ev-NeRF is given by

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{event}}+\lambda\mathcal{L}_{\text{thres}}.$
(8)

$\mathcal{L}_{\text{event}}$ is the event rendering loss which replaces the
image rendering loss in conventional NeRF. Here we combine the event
generation model in Section 3 with the volume rendering formulation of the
original NeRF. $\mathcal{L}_{\text{thres}}$ is the threshold bound loss,
designed to avoid degenerate cases.

#### Event Rendering Loss

Our loss for training $F_{\theta}$ compares the recorded events and the
difference in the rendered intensity, as shown in Figure 2. Let us denote the
intensity values of the pixel ray $r$ at time ${T}_{j}$ and ${T}_{j+1}$ as
$\hat{I}(\mathcal{T}_{j}(r))$ and $\hat{I}(\mathcal{T}_{j+1}(r))$,
respectively, where the intensity images are obtained using Equation 7. Then
we can calculate $\Delta L_{j}$ at the pixel ray $r$ as

$\Delta
L_{j}(r)=\log\hat{I}\left(\mathcal{T}_{j+1}(r)\right)-\log\hat{I}\left(\mathcal{T}_{j}(r)\right).$
(9)

Using the event generation model in Equation 2, $\Delta L_{j}(r)$ in Equation
9 should be measured by the accumulated sum of the events
$B_{j}(r)=\sum_{E_{k}\in\mathcal{E}_{j}(r)}p_{k}B_{j}^{p_{k}}$ within the time
interval $[{T}_{j},{T}_{j+1})$.

Our event-rendering loss $\mathcal{L}_{\text{event}}$ is the total sum of
discrepancy for all time intervals $j$ and rays $r$ available in the batch,

$\mathcal{L}_{\text{event}}=\sum_{j}\sum_{r}f_{B_{j}^{+},B_{j}^{-}}^{2}(\Delta
L_{j}(r)-B_{j}(r)),$ (10)

where the function $f$ penalizes the discrepancy above the sensor threshold by
incorporating a dead zone $\left[B^{-},B^{+}\right]$ as described in [3]:

$f_{B^{+},B^{-}}(x)=\begin{cases}x-B^{+},&\mbox{if }x>B^{+},\\\ 0,&\mbox{if
}B^{-}\leq x\leq B^{+},\\\ -x+B^{-},&\mbox{if }x<B^{-}.\end{cases}$ (11)

Therefore we only focus on where the measurements provide enough evidence for
the brightness changes.

#### Threshold Bound Loss

While the joint optimization over the unknown threshold values
$B_{j}^{+},B_{j}^{-}$ improves the performance of Ev-NeRF, the additional
parameters further challenge the optimization process which already is highly
under-constrained with the unknown brightness values $I$. Without additional
constraints, we empirically observed that the network often converges to the
trivial solution with $\Delta I=0$ and the threshold value $0$. The threshold
bound loss is a simple prior to keep the threshold values within the
reasonable bound:

$\mathcal{L}_{\text{thres}}=r\left(B_{0}^{+}-B_{j}^{+}\right)+r\left(B_{j}^{-}-B_{0}^{-}\right),$
(12)

where $r\left(\cdot\right)$ is a ReLU function. In our experiments, we set
$B_{0}^{+}=0.3$ and $B_{0}^{-}=-0.3$, based on our prior knowledge about
threshold scale [15].

Figure 3: Qualitative comparison on intensity image reconstruction.

##  5 Results

Once we train Ev-NeRF, from highly corrupted event sequence data, the coherent
implicit volumetric representation of NeRF is learned. The integrated
representation can generate images or depth estimates in a novel view, which
are compared against previous works or designed baseline. We evaluate Ev-NeRF
on both synthetic and real dataset.

#### Implementation Detail

The input to Ev-NeRF is the stream of event data obtained from an event sensor
moving around a static scene. The stream data is accompanied by a sequence of
the sensor’s intermediate positions which are time-stamped. The sensor
positions can be acquired from an additional sensor or structure from motion
(SfM) using intensity images or provided from simulator.

For real world data, we calculate the poses by running SfM [48, 49] with the
intensity frames, which are provided in the datasets [35, 54, 66] recorded at
about 24 Hz. Except for this process, the intensity frames are not available
to Ev-NeRF during training and are used only for evaluation. We use about 50
to 100 consecutive event slices $\mathcal{E}_{j}$ to train a neural volume,
where the length of each time slice $[{T}_{j},{T}_{j+1})$ is chosen to be the
frame rate of intensity frames, about 1/24s. The duration of the total event
sequence used for Ev-NeRF is roughly 2-4 seconds. For each event slice
$\mathcal{E}_{j}$ corresponding to $[{T}_{j},{T}_{j+1})$, we add random events
equivalent to 5% of the number of events that occurred at time slice
$[{T}_{j},{T}_{j+1})$ during training. We find that the additional random
noise slightly improves the quality of neural representation in ambiguous
regions, which is further described in the supplementary material.

###  5.1 Robust Intensity Image Reconstruction

Ev-NeRF creates the NeRF volume that aggregates multiple observations, which
is robust against a variety of perturbation such as the amount of noise or
extreme light conditions. We compare the quality of the synthesized image
against previous works on intensity image reconstruction under noisy or low-
light condition. We use three available real-world datasets that are widely
used for event-based image reconstruction, namely IJRR [35], HQF [54], and
Stereo DAVIS dataset [66] for quantitative comparison. For the stereo DAVIS
dataset, we only use the measurements from a single event camera. Unless
otherwise noted, we use four sub sequences (dynamic_6dof, office_spiral,
office_zigzag, hdr_boxes) from the IJRR dataset, three sequences
(reflective_materials, high_texture_plants, still_life) from HQF dataset and
two sequences (monitor, reader) from Stereo DAVIS dataset.

#### Intensity Image Reconstruction

Figure 3 shows exemplar reconstructed images, and different approaches exhibit
different kinds of artifacts. We compare the performance of Ev-NeRF against
three baselines: E2VID [44], E2VID+ [54] and ssl-E2VID [38]. E2VID [44] is
trained in a supervised fashion with a large amount of synthetic data. E2VID+
[54] adjusts the synthetic training data to better fit the distribution of the
real data. ssl-E2VID [38] tries to overcome the domain gap and suggests a
self-supervised approach achieving results comparable to E2VID+ [54] without
ground truth data. Unlike Ev-NeRF, which is trained for each scene, previous
works are trained in a supervised fashion with a synthetic dataset composed of
a pair of ground truth images and event measurements. A quantitative
comparison of image reconstruction on real-world scenes is in the
supplementary material. More importantly, Ev-NeRF only enforces multi-view
consistency without further domain-dependent training, whereas previous
approaches are trained in a fixed resolution dedicated to a specific
measurement condition. We further validate intensity image reconstruction
results on the CED Dataset [7] with a different resolution (346$\times$240) in
the supplementary.

#### Noise Resistant Image Reconstruction

Figure 4: Effect of noise on various image reconstruction methods. Even in
serious noise conditions, Ev-NeRF can robustly reconstruct images whose
quality is comparable to that of other approaches in normal conditions. Figure
5:  Qualitative comparison on image reconstruction given input with various
noise levels. Ev-NeRF shows little performance degradation even with extremely
noisy inputs.

The performance gap becomes prominent when the measurements exhibit severe
noises, introducing domain shift. We use the event camera simulator v2e [18]
to synthetically add realistic sensor noise due to photon fluctuations or
invalid threshold values. The simulator allows us to control the amount of
noise, which we indicate with the ratio of the number of noisy events added to
the number of existing events. Figure 4 evaluates the SSIM of the
reconstructed intensity image compared against the ground truth intensity
image with the office_zigzag scene in the IJRR [35]. The effectiveness of Ev-
NeRF is prominent for noisy events, where it compensates the complex noise
despite over 70% of noise and achieves comparable quality as the state-of-the-
art method. In contrast, other methods rapidly deteriorate under noisy data,
as they suffer from the domain shift caused by severe noise, which is not
observed during the training neural network under the supervised set-up.

Figure 5 shows the visual comparison of reconstructed images of NeRF for
inputs with different noise levels against E2VID+ [54], which is the state-of-
the-art method for intensity image reconstruction. While E2VID+ [54] is
vulnerable even to small noise, the intrinsic multi-view consistency of Ev-
NeRF results in stable performance regardless of noise level and alleviates
the effect of noise. Additional results are provided in the supplementary
material.

Figure 6: Qualitative comparison on intensity image reconstruction and depth
estimation of the NeRF volume trained with (a) event data and (b) intensity
images captured in low-light conditions.

Task | E2VID+ [54] \+ NeRF | Ev-NeRF | E2VID+ [54] \+ NeRF Noise 0.3 | Ev-NeRF Noise 0.3
---|---|---|---|---
| Abs Rel $\downarrow$ | RMSE $\downarrow$ | Sq Rel $\downarrow$ | Abs Rel $\downarrow$ | RMSE $\downarrow$ | Sq Rel $\downarrow$ | Abs Rel $\downarrow$ | RMSE $\downarrow$ | Sq Rel $\downarrow$ | Abs Rel $\downarrow$ | RMSE $\downarrow$ | Sq Rel $\downarrow$
Depth | 0.082 | 0.092 | 0.101 | 0.034 | 0.052 | 0.053 | 0.099 | 0.112 | 0.133 | 0.038 | 0.056 | 0.058
| MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
Intensity | 0.02 | 0.89 | 0.15 | 0.01 | 0.92 | 0.07 | 0.03 | 0.83 | 0.19 | 0.01 | 0.90 | 0.08
Novel View | 0.03 | 0.85 | 0.17 | 0.01 | 0.91 | 0.07 | 0.04 | 0.75 | 0.25 | 0.01 | 0.89 | 0.09

Table 1:  Quantitative results for depth estimation, intensity image
reconstrunction, and novel view synthesis on synthetic datasets. Ev-NeRF
consistently outperforms the designed baseline even in noisy conditions.

#### HDR Image Reconstruction

In addition to noise reduction, under extreme light conditions, the
reconstructed images from Ev-NeRF naturally contain high dynamic range
information without further processing. Figure 6(a) shows the qualitative
results on HDR imaging. Compared to the intensity images concurrently captured
in low-light set-up ((b), left), which are unknown to the algorithm, the
sensor measurements detect the subtle details within the scene ((a), left),
which are compiled to produce the HDR intensity images ((a), middle). This is
because Ev-NeRF generates the intensity values to reflect the fine-grained
changes in illumination without saturation and is agnostic to any prior
absolute values of the intensity that might be clipped to a smaller range.
While we use intensity images to find the poses, the neural volume trained
with event data contains variations and details that could not be captured
with low-quality intensity images, especially in challenging lighting.

In Figure 6(b), we also present the rendering using the neural volume of
ordinary NeRF for comparison, which is trained with the intensity images of
the same sequence obtained from a DAVIS camera [4]. The intensity-based NeRF
is trained with the MSE loss between the rendered and measured intensity,
following [33]. The reconstructed images and depths are superior when using
Ev-NeRF, therefore fully exploiting the dynamic range of the sensor.

Under extreme lighting, not only do intensity frames suffer from spurious
noise, but also the event data is mixed with severe noise, and the performance
of vision-based approaches severely deteriorates [24, 25]. The multi-view
consistency in Ev-NeRF compensates for the uncharacteristic noises and
reliably reconstructs novel views and depth information beyond the level
obtained from vanilla NeRF using intensity images.

###  5.2 Scene Structure Estimation

#### Evaluation with Synthetic Data

Figure 7:  Qualitative results for intensity image, dense depth by baseline
(E2VID+ [54] \+ NeRF) and Ev-NeRF.

The NeRF volume aggregates multiple measurements into a coherent structure
such that we can generate images from arbitrary viewpoint. Also, the density
values of NeRF volume can provide estimates for 3D structure as in Equation 5.
Since the real-world datasets for event-based vision lack dense depth
annotations, the synthetic data can provide quantitative evaluation on the
created NeRF volume. Also, the 3D points found with existing event-based
algorithms are limited to sparse reconstruction that can generate events and
no prior work is able to reconstruct novel view images. Therefore, we design
baseline for quantitatively comparison.

For the synthetic data, we capture a high frame rate video from typical
trajectories for NeRF with Blender, and the event simulator [18] generates
events with realistic noise from the video. For the baseline, we first
reconstruct intensity images from events using E2VID+ [54] and then use them
to learn vanilla NeRF [33]. The proposed method serves as a powerful baseline
that can reconstruct dense depth and an intensity image from a novel view.
Figure 7 shows an object in Blender, generated events, and the pairs of an
intensity image and the depth generated from the designed baseline and Ev-
NeRF.

Table 1 compares the depth estimation results using three metrics: average
relative error (Abs Rel), squared relative difference (Sq Rel), root mean
squared error (RMSE). Also the reconstructed intensity images in the given or
novel views against the ground truth intensity images are compared using three
metrics: mean squared error (MSE), structural similarity (SSIM) and perceptual
similarity (LPIPS) [65]. In baseline, images reconstructed through E2VID+ [54]
lack multi-view consistency and result in an artifact, which reduces the
accuracy of 3D structures. Ev-NeRF directly integrates events and Table 1
indicates that it is superior to the proposed baseline for depth
reconstruction. Also, the high-quality 3D structure aids in more accurate
reconstruction of intensity images or novel-view synthesis, especially in the
presence of noise.

#### Evaluation with Real Data

Figure 8: Qualitative comparison on 3D structure estimation. Ground-truth
intensity frame, semi-dense depth by [66], dense depth, and intensity image by
Ev-NeRF in order. Figure 9: Qualitative results on depth estimation on a real
dataset. The intensity images (top) are not observed by the algorithm.

There is no prior work that is bound to the exact set-up as Ev-NeRF, and no
real-world dataset with ground-truth is available to quantitatively evaluate
the dense 3D structure. As an alternative, we compare the acquired geometry
against the 3D information extracted from an event-based SLAM approach [66].
Event-based SLAM utilizes measurements from additional hardware and compute
the camera pose and the geometric structure. Specifically, Zhou et al. [66]
use a pair of temporally synchronized event cameras. On the other hand, Ev-
NeRF uses a single event camera with known poses. Even though the detailed
setup is different, both estimate 3D structure from the measurements of a
moving sensor observing a static scene, whose results are provided in Figure
8. Zhou et al. [66] reconstruct depth only in the area where the events have
occurred, and therefore recover semi-dense 3D structure (second column in
Figure 8). In contrast, Ev-NeRF reconstructs implicit 3d volume and can
produce denser depth with intensity images (third and last columns in Figure
8).

We can also find qualitative results from the dataset used in Sec. 5.1. Figure
6 contains the depth estimation from low lighting conditions. Additionally,
Figure 9 shows that Ev-NeRF creates reasonable depth estimates for challenging
real-world scenes with reflective materials, fine geometry details, or HDR
measurements where existing approaches might fail.

| MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
Scene | Given | Novel | Given | Novel | Given | Novel
office_zigzag | 0.03 | 0.04 | 0.42 | 0.41 | 0.27 | 0.28
office_spiral | 0.03 | 0.03 | 0.41 | 0.40 | 0.27 | 0.27
boxes | 0.04 | 0.04 | 0.48 | 0.46 | 0.31 | 0.33
dynamic_6dof | 0.19 | 0.20 | 0.26 | 0.26 | 0.41 | 0.43
reflective_materials | 0.05 | 0.06 | 0.40 | 0.40 | 0.35 | 0.38
high_texture_plants | 0.03 | 0.03 | 0.44 | 0.42 | 0.34 | 0.36
still_life | 0.03 | 0.04 | 0.53 | 0.52 | 0.18 | 0.18

Table 2: Quantitative results of novel view reconstruction. Compared to given
views provided during training, only a small performance gap is observed to
reconstruct images of novel views.

We also evaluate the performance of novel view synthesis. We divide each sub-
sequence in the dataset into a training and test set, and synthesize the
images at the viewpoints included in the test set with Ev-NeRF trained with
the training set. Table 2 compares the reconstructed images against ground
truth intensity images. The reported metrics for novel-view images are
compatible with the results for the given view. Thus, the ability of NeRF is
nicely transferred to Ev-NeRF.

##  6 Conclusions and Future Work

We present Ev-NeRF, which learns the implicit volume of the neural radiance
field from the raw stream of events generated by a neuromorphic camera. The
inherent multi-view consistency creates a representation remarkably robust to
noisy inputs, which is a critical challenge for using a neuromorphic sensor,
yet exploits the subtle brightness changes detected from the sensor. Further,
the created NeRF volume can generate intensity images or estimate depth, whose
quality is comparable to many existing supervised methods exclusively designed
to solve a specific task. To the best of our knowledge, Ev-NeRF is the first
attempt to incorporate the NeRF formulation with raw event data and can
advance with the abundant subsequent works that overcome the limitations of
NeRF, such as handling dynamic scenes to better incorporate the fast temporal
resolution of the sensor [41, 27], reducing training and rendering time [36,
55, 1], and alleviating the requirements of known camera poses [62, 28].

#### Acknowledgements

This research was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government(MSIT) (No. 2020R1C1C1008195), Samsung
Electronics Co., Ltd, and Creative-Pioneering Researchers Program through
Seoul National University. Inwoo Hwang is grateful for financial support from
Hyundai Motor Chung Mong-Koo Foundation.

## References

  * [1] Alex Yu and Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks, 2021.
  * [2] R Baldwin, Mohammed Almatrafi, Vijayan Asari, and Keigo Hirakawa.  Event probability mask (epm) and event denoising convolutional neural network (edncnn) for neuromorphic cameras.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [3] Patrick Bardow, Andrew J. Davison, and Stefan Leutenegger.  Simultaneous optical flow and intensity estimation from an event camera.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 884–892, 2016.
  * [4] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck.  A 240 × 180 130 db 3 µs latency global shutter spatiotemporal vision sensor.  IEEE Journal of Solid-State Circuits, 49(10):2333–2341, 2014.
  * [5] Samuel Bryner, Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza.  Event-based, direct camera tracking from a photometric 3D map using nonlinear optimization.  In IEEE Int. Conf. Robot. Autom. (ICRA), 2019.
  * [6] Pablo Rodrigo Gantier Cadena, Yeqiang Qian, Chunxiang Wang, and Ming Yang.  Spade-e2vid: Spatially-adaptive denormalization for event-based video reconstruction.  IEEE Transactions on Image Processing, 30:2488–2500, 2021.
  * [7] Timo Stoffregen Cedric Scheerlinck, Henri Rebecq and Davide Scaramuzza Nick Barnes, Robert Mahon.  Ced: Color event camera dataset.  In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
  * [8] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [9] Matthew Cook, Luca Gugelmann, Florian Jug, Christoph Krautz, and Angelika Steger.  Interacting maps for fast visual interpretation.  In The 2011 International Joint Conference on Neural Networks, pages 770–776, 2011.
  * [10] Daniel Czech and Garrick Orchard.  Evaluating noise filtering for event-based asynchronous change detection image sensors.  In 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pages 19–24, 2016.
  * [11] Tobi Delbruck.  Frame-free dynamic digital vision.  In Proceedings of Intl. Symposium on Secure-Life Electronics, Advanced Electronics for Quality Life and Society, pages 6–7, 2008.
  * [12] Yongjian Deng, Youfu Li, and Hao Chen.  Amae: Adaptive motion-agnostic encoder for event-based object classification.  IEEE Robotics and Automation Letters, 5(3):4596–4603, 2020.
  * [13] Peiqi Duan, Zihao W. Wang, Xinyu Zhou, Yi Ma, and Boxin Shi.  Eventzoom: Learning to denoise and super resolve neuromorphic events.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12824–12833, June 2021.
  * [14] Yang Feng, Hengyi Lv, Hailong Liu, Yisa Zhang, Yuyao Xiao, and Chengshan Han.  Event density based denoising method for dynamic vision sensor.  Applied Sciences, 10:2024, 2020.
  * [15] Guillermo Gallego, Tobi Delbruck, Garrick Michael Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jorg Conradt, Kostas Daniilidis, and Davide Scaramuzza.  Event-based vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
  * [16] Cheng Gu, Erik Learned-Miller, Daniel Sheldon, Guillermo Gallego, and Pia Bideau.  The spatio-temporal poisson point process: A simple model for the alignment of event camera data.  In International Conference on Computer Vision (ICCV), 2021.
  * [17] Javier Hidalgo-Carrió, Guillermo Gallego, and Davide Scaramuzza.  Event-aided direct sparse odometry.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
  * [18] Y Hu, S C Liu, and T Delbruck.  v2e: From video frames to realistic DVS event camera streams.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2021.
  * [19] Xueyan Huang, Yueyi Zhang, and Zhiwei Xiong.  High-speed structured light based 3d scanning using an event camera.  Opt. Express, 29(22):35864–35876, Oct 2021.
  * [20] Daniel Gehrig Javier Hidalgo-Carrio and Davide Scaramuzza.  Learning monocular dense depth from events.  IEEE International Conference on 3D Vision.(3DV), 2020.
  * [21] Jianhao Jiao, Huaiyang Huang, Liang Li, Zhijian He, Yilong Zhu, and Ming Liu.  Comparing representations in tracking for event camera-based SLAM.  CoRR, abs/2104.09887, 2021.
  * [22] Hanme Kim, Ankur Handa, Ryad Benosman, Sio-Hoi Ieng, and Andrew Davison.  Simultaneous mosaicing and tracking with an event camera.  In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
  * [23] Hanme Kim, Stefan Leutenegger, and Andrew J. Davison.  Real-time 3d reconstruction and 6-dof tracking with an event camera.  In ECCV, 2016.
  * [24] Junho Kim, Jaehyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim.  N-imagenet: Towards robust, fine-grained object recognition with event cameras.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2146–2156, October 2021.
  * [25] Junho Kim, Inwoo Hwang, and Young Min Kim.  Ev-tta: Test-time adaptation for event-based object recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
  * [26] Diederik P. Kingma and Jimmy Lei Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [27] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [28] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.  Barf: Bundle-adjusting neural radiance fields.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [29] Daqi Liu, Álvaro Parra, and Tat-Jun Chin.  Globally optimal contrast maximisation for event-based motion estimation.  In European Conference on Computer Vision (ECCV), 2020.
  * [30] Hongjie Liu, Christian Brandli, Chenghan Li, Shih-Chii Liu, and Tobi Delbruck.  Design of a spatiotemporal correlation filter for event-based sensors.  In 2015 IEEE International Symposium on Circuits and Systems (ISCAS), pages 722–725, 2015.
  * [31] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [32] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, and Jonathan T. Barron.  NeRF in the dark: High dynamic range view synthesis from noisy raw images.  arXiv, 2021.
  * [33] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [34] Anton Mitrokhin, Zhiyuan Hua, Cornelia Fermuller, and Yiannis Aloimonos.  Learning visual motion segmentation using event surfaces.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [35] Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam.  The International Journal of Robotics Research, 36(2):142–149, Feb 2017.
  * [36] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  arXiv:2201.05989, Jan. 2022.
  * [37] Urbano Miguel Nunes and Yiannis Demiris.  Entropy minimisation framework for event-based vision model estimation.  In European Conference on Computer Vision (ECCV), 2020.
  * [38] Federico Paredes-Vallés and Guido C. H. E. de Croon.  Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy.  CVPR, 2021.
  * [39] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [40] Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi.  Learning to detect objects with a 1 megapixel event camera.  arXiv preprint arXiv:2009.13436, 2020.
  * [41] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  arXiv preprint arXiv:2011.13961, 2020.
  * [42] Henri Rebecq, Guillermo Gallego, Elias Mueggler, and Davide Scaramuzza.  EMVS: Event-based multi-view stereo—3D reconstruction with an event camera in real-time.  Int. J. Comput. Vis., 126:1394–1414, Dec. 2018.
  * [43] Henri Rebecq, Daniel Gehrig, and Davide Scaramuzza.  ESIM: an open event camera simulator.  Conf. on Robotics Learning (CoRL), Oct. 2018.
  * [44] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza.  Events-to-video: Bringing modern computer vision to event cameras.  IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2019.
  * [45] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza.  High speed and high dynamic range video with an event camera.  IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI), 2019.
  * [46] Viktor Rudnev, Mohamed Elgharib, Christian Theobalt, and Vladislav Golyanik.  Eventnerf: Neural radiance fields from a single colour event camera, 2022\.
  * [47] Cedric Scheerlinck, Henri Rebecq, Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza.  Fast image reconstruction with an event camera.  In IEEE Winter Conf. Appl. Comput. Vis. (WACV), pages 156–163, 2020.
  * [48] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [49] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision (ECCV), 2016.
  * [50] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman.  HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification.  arXiv preprint arXiv:2018.00186, June 2018.
  * [51] I. S.MohammadMostafavi, Lin Wang, and Kuk-Jin Yoon.  Learning to reconstruct hdr images from events, with applications to depth and flow prediction.  International Journal of Computer Vision, pages 1–21, 2021.
  * [52] Yu Zhang Song Zhang, Zhe Jiang, Dongqing Zou, Jimmy Ren, and Bin Zhou.  Learning to see in the dark with events.  In ECCV, 2020.
  * [53] Timo Stoffregen, Guillermo Gallego, Tom Drummond, Lindsay Kleeman, and Davide Scaramuzza.  Event-based motion segmentation by motion compensation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [54] T. Stoffregen, C. Scheerlinck, D. Scaramuzza, T. Drummond, N. Barnes, L. Kleeman, and R. Mahoney.  Reducing the sim-to-real gap for event cameras.  In ECCV, 2020.
  * [55] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction, 2021.
  * [56] Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch.  Learning an event sequence embedding for dense event-based deep stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [57] Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, and Wen Yang.  Event enhanced high-quality image recovery.  In European Conference on Computer Vision. Springer, 2020.
  * [58] Lin Wang, S. Mohammad Mostafavi I. , Yo-Sung Ho, and Kuk-Jin Yoon.  Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [59] Y. Wang, B. Du, Y. Shen, K. Wu, G. Zhao, J. Sun, and H. Wen.  Ev-gait: Event-based robust gait recognition using dynamic vision sensors.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6351–6360, 2019.
  * [60] Zihao Wang, Peiqi Duan, Oliver Cossairt, Aggelos Katsaggelos, Tiejun Huang, and Boxin Shi.  Joint filtering of intensity images and neuromorphic events for high-resolution noise-robust imaging.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [61] Ziwei Wang, Liyuan Pan, Yonhon Ng, Zheyu Zhuang, and Robert Mahony.  Stereo hybrid event-frame (shef) cameras for 3d perception.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
  * [62] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  NeRF$--$: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [63] Wenming Weng, Yueyi Zhang, and Zhiwei Xiong.  Event-based video reconstruction using transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2563–2572, October 2021.
  * [64] J. Wu, C. Ma, X. Yu, and G. Shi.  Denoising of event-based sensors with spatial-temporal correlation.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4437–4441, 2020.
  * [65] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [66] Yi Zhou, Guillermo Gallego, Henri Rebecq, Laurent Kneip, Hongdong li, and Davide Scaramuzza.  Semi-dense 3d reconstruction with a stereo event camera.  In European Conference on Computer Vision (ECCV), 2018.
  * [67] Yunhao Zou, Yinqiang Zheng, Tsuyoshi Takatani, and Ying Fu.  Learning to reconstruct high speed and high dynamic range videos from events.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2024–2033, June 2021.

##  Appendix A Additional Implementation Details

Figure 10: Network Architecture of Ev-NeRF. The network consists of fully-
connected layers. The numbers in the colored blocks indicate the dimension of
the corresponding layers.

We implement Ev-NeRF using PyTorch using the NeRF formulation. For positional
encoding, we use 10 frequencies for $\mathbf{x}$ and 4 for $\mathbf{d}$. The
weight $\lambda$ of $\mathcal{L}_{\text{thres}}$ in Equation 8 is set large to
1000 to avoid thresholds from continuing to decrease. For all experiments, we
use the Adam optimizer [26] with a learning rate of $5\times 10^{-4}$. Ev-NeRF
takes about an hour in RTX 3090 GPU to train per scene and 1.5 seconds to
render a single image.

Similar to NeRF, the neural network takes the 3D coordinate and the ray
direction as input and outputs the volume density and emitted radiance.
Sinusoidal positional encoding, $\gamma\left(\cdot\right)$, is applied to
input variables. Figure 10 shows the detailed architecture of our network. The
architecture mostly follows that of NeRF [33], however it predicts the emitted
luminance value instead of color values. Following NeRF [33], we add zero-
mean, unit-variance Gaussian random noise to the density $\sigma$ for slightly
improved performance.

##  Appendix B Dataset Description

For the real-world data, we use a sub-sequence of the event sequences from
IJRR [35], HQF [54] and Stereo DAVIS [66] for training. The data includes
intensity images at regular time intervals (about 24 Hz) and asynchronous
event data. These datasets are generated with a DAVIS240C [4] event camera and
both intensity images and events have a resolution of 240$\times$180\. In our
setup, we assume camera poses are given, which are calculated from running SfM
[48, 49] with the intensity frames. Except for this process, the intensity
frames are not available during training and are used only for evaluation. We
use sub-sequences with a length of 50 to 100 intensity frames for training and
the exact frame index corresponding to the original dataset [35, 54] is
described in Table 3. For comparison with event-based SLAM, we use two
sequences (simulation_3planes, reader) from [66].

For the synthetic data, we examine the extracted scene structure using models
widely used for NeRF [33], namely lego, hotdog, mic, drums, and chair. Figure
11 shows examples of models and generated events.

Dataset | Scene | start frame index | end frame index
---|---|---|---
IJRR | office_zigzag | 0 | 100
office_spiral | 0 | 100
boxes | 230 | 330
dynamic_6dof | 30 | 130
hdr_boxes | 70 | 120
HQF | reflective_materials | 60 | 150
high_texture_plants | 930 | 1000
still_life | 300 | 350
Stereo DAVIS | monitor | 0 | 100
reader | 0 | 100

Table 3:  The start and end frame index for intensity frames used for our
experiments. We use sub-sequences of original datasets, namely IJRR [35], HQF
[54] and Stereo DAVIS [66]. Figure 11: Visualization of generated events for
each scene.

##  Appendix C Description on Convergence

Figure 12: Intermediate results over the training time for each scene.

| MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
Scene | w/o joint | w/o noise inj. | full | w/o joint | w/o noise inj. | full | w/o joint | w/o noise inj. | full
office_zigzag | 0.03 | 0.04 | 0.03 | 0.40 | 0.41 | 0.42 | 0.29 | 0.27 | 0.27
office_spiral | 0.04 | 0.04 | 0.03 | 0.41 | 0.41 | 0.41 | 0.29 | 0.28 | 0.27
boxes | 0.04 | 0.06 | 0.04 | 0.47 | 0.45 | 0.48 | 0.32 | 0.33 | 0.31
dynamic_6dof | 0.21 | 0.24 | 0.19 | 0.25 | 0.24 | 0.26 | 0.42 | 0.43 | 0.41
reflective_materials | 0.06 | 0.07 | 0.05 | 0.39 | 0.38 | 0.40 | 0.35 | 0.36 | 0.35
high_texture_plants | 0.04 | 0.03 | 0.03 | 0.43 | 0.43 | 0.44 | 0.34 | 0.35 | 0.34
still_life | 0.04 | 0.05 | 0.03 | 0.52 | 0.51 | 0.53 | 0.21 | 0.19 | 0.18
monitor | 0.05 | 0.08 | 0.03 | 0.30 | 0.26 | 0.32 | 0.38 | 0.39 | 0.37
reader | 0.11 | 0.12 | 0.09 | 0.44 | 0.43 | 0.45 | 0.38 | 0.36 | 0.35

Table 4: Ablation study on the effect of joint training of the sensor
threshold values and noise injection. The columns display results without
joint training, without noise injection, and the full model, respectively. Our
full training method shows the optimal reconstructed image quality.

While events provide only changes of brightness, we empirically found it
converges to reliable absolute brightness as the learning progresses. The
supplementary video shows how Ev-NeRF converges on absolute brightness as
training progresses. Figure 12 contains a few representative images. Ev-NeRF
obtains a rough 3D structure in about 10 min, and the subsequent timesteps
focus on capturing further details.

##  Appendix D Ablation Studies

#### Joint Training

As shown in Equation (8), we propose joint training that concurrently
optimizes $\Delta I$ and the threshold values $B_{j}^{+},B_{j}^{-}$ of all
timestamps with $\lambda=1000$. We verify the advantage of our joint training
on intensity reconstruction in Table 4. We compare the proposed joint training
against the ablated version with pre-fixed threshold values $\pm 0.3$ and
$\lambda=0$. Results show that the joint training scheme is beneficial to the
reconstructed image quality.

Figure 13: Visual analysis on noise injection, which helps solve ambiguity in
places such as walls where events do not occur.

#### Noise Injection

We verify that the additional random noise slightly improves the quality of
Ev-NeRF. When we compile the training data of events occurred during time
slice $[T_{j},T_{j+1})$, we add random events whose amount is 5% of the number
of events that occurred at time. Table 4 numerically compares the quality of
reconstructed images with random noise injection against the original event
slice $\mathcal{E}_{j}$ and verifies that the noise injection is beneficial to
the image quality. Figure 13 visualizes the effect of noise injection. The
additional noise helps the neural volume to resolve ambiguity in the areas
where events do not occur, such as the solid-colored wall behind the monitor.

##  Appendix E Additional Results

Here we show more examples of quantitative and qualitative results. The
results are presented in various scenes with three baselines: E2VID [44],
E2VID+ [54] and ssl-E2VID [38], which are designed to reconstruct intensity
images.

###  E.1 Intensity Image Reconstruction

#### Additional Quantitative Results

| MSE $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
Scene | E2VID | E2VID+ | ssl-E2VID | Ours | E2VID | E2VID+ | ssl-E2VID | Ours | E2VID | E2VID+ | ssl-E2VID | Ours
office_zigzag | 0.07 | 0.05 | 0.08 | 0.03 | 0.38 | 0.39 | 0.34 | 0.42 | 0.34 | 0.25 | 0.40 | 0.27
office_spiral | 0.06 | 0.05 | 0.07 | 0.03 | 0.38 | 0.39 | 0.37 | 0.41 | 0.35 | 0.27 | 0.39 | 0.27
boxes | 0.06 | 0.03 | 0.08 | 0.04 | 0.47 | 0.60 | 0.45 | 0.48 | 0.31 | 0.20 | 0.37 | 0.31
dynamic_6dof | 0.12 | 0.06 | 0.15 | 0.19 | 0.29 | 0.34 | 0.28 | 0.26 | 0.40 | 0.33 | 0.54 | 0.41
reflective_materials | 0.07 | 0.05 | 0.08 | 0.05 | 0.39 | 0.45 | 0.30 | 0.40 | 0.31 | 0.24 | 0.38 | 0.35
high_texture_plants | 0.04 | 0.02 | 0.05 | 0.03 | 0.42 | 0.55 | 0.42 | 0.44 | 0.21 | 0.12 | 0.23 | 0.34
still_life | 0.05 | 0.02 | 0.08 | 0.03 | 0.50 | 0.61 | 0.40 | 0.53 | 0.23 | 0.13 | 0.29 | 0.18
monitor | 0.04 | 0.04 | 0.09 | 0.03 | 0.31 | 0.36 | 0.33 | 0.32 | 0.39 | 0.20 | 0.34 | 0.37
reader | 0.07 | 0.04 | 0.09 | 0.09 | 0.42 | 0.43 | 0.38 | 0.45 | 0.31 | 0.25 | 0.40 | 0.35

Table 5: Quantitative comparison of image reconstruction on scenes from the
IJRR [35], HQF [54] and Stereo DAVIS [66] dataset. The results with the best
performance are in bold. We additionally underline the runner-up metric.

Table 5 displays the quantitative comparison against baseline methods with
real-world datasets, namely IJRR [35], HQF [54], and Stereo DAVIS dataset
[66]. We compared using three metrics: mean squared error (MSE), structural
similarity (SSIM) and perceptual similarity (LPIPS) [65]. For real-world data,
Ev-NeRF outperforms ssl-E2VID and is on par with E2VID and E2VID+ without
observing the ground truth intensity frames.

However, we used sub-sequences of real datasets and event camera trajectory
from sub-sequences is not optimal for NeRF, which is typically trained with
cameras located on the hemisphere around the object from roughly constant
distances. This causes Ev-NeRF to show lower performance compared to E2VID+
for some sequences. As mentioned in Table 1 from the main paper, with typical
trajectories for NeRF, the quality of intensity images of Ev-NeRF is then
consistently superior to the baseline, especially in the presence of noise.

#### Additional Qualitative Results for Intensity Image and Novel View
Synthesis

Figure 15 contains the results of intensity image reconstruction for all of
the sequences we use dwith a qualitative comparison with various baselines and
Figure 16 shows image reconstruction results observed from various camera
poses. Also, the supplementary video contains the event sequences used for
training, paired with corresponding reconstructed intensity images and depth
results. Figure 17 shows novel view synthesis observed from the viewpoints
that are not available in the input dataset. Ev-NeRF maintains comparable
performance for all scenes. Additional results on novel view synthesis are
shown in the supplementary video.

#### Intensity Image Reconstruction on Different Sensor Resolution

We further validate the domain-invariance of Ev-NeRF with intensity image
reconstruction results on the Color Event Camera Dataset (CED) [7]. The data
is composed of three channels of color events in a different resolutions
(346$\times$240). All of the datasets presented in other sections are
processed in the resolution of 240$\times$180 bound to the sensor resolution.
Ev-NeRF does not assume any fixed resolution of the scene and can be applied
in multiple color channels without fine-tuning. We extend our approach to
three color channels following the method suggested in [7]; we first
reconstruct the intensity images via red, green, and blue channels
respectively, and upsample the individual channels of the image to the
original resolution to produce a single color image. Figure 14 shows an
exemplar color DAVIS frame and reconstructed color image for simple_jenga
scene from CED [7]. Ev-NeRF can find the scene structure within reasonable
ranges. More importantly, there is no additional training to account for the
domain shift in sensor characteristics or resolution.

###  E.2 Noise Resistant Image Reconstruction

Ev-NeRF is extremely robust under noise and maintains the quality of the
reconstruction under data of various noise levels. In addition to the visual
example shown in Figure 4 of the main text, we show results given data
corrupted with different levels of noise in Figure 18. Figure 19 contains more
results of image reconstruction with severe noise of ratio 0.9. Ev-NeRF
exhibits little degradation in performance and therefore can be useful in an
extreme environments subject to unknown noise characteristics.

Figure 14: Color image reconstruction through Ev-NeRF.  Figure 15: Qualitative
comparison on intensity image reconstruction under various scenes. Figure 16:
Qualitative intensity image reconstruction results at various time steps.
Figure 17: Qualitative results for novel view image reconstruction. Figure 18:
Qualitative comparison under different noise levels. Figure 19: Noise
reduction of events over various scenes with extreme noise ratio of 0.9.