11institutetext: Dept. of Informatics, Univ. of Zurich and
Dept. of Neuroinformatics, Univ. of Zurich and ETH Zurich 11email:
<EMAIL_ADDRESS>
# Dense Continuous-Time Optical Flow from Events and Frames

Mathias Gehrig  Manasi Muglikar  Davide Scaramuzza

###### Abstract

We present a method for estimating dense continuous-time optical flow.
Traditional dense optical flow methods compute the pixel displacement between
two images. Due to missing information, these approaches cannot recover the
pixel trajectories in the blind time between two images. In this work, we show
that it is possible to compute per-pixel, continuous-time optical flow by
additionally using events from an event camera. Events provide temporally
fine-grained information about movement in image space due to their
asynchronous nature and microsecond response time. We leverage these benefits
to predict pixel trajectories densely in continuous-time via parameterized
Bézier curves. To achieve this, we introduce multiple innovations to build a
neural network with strong inductive biases for this task: First, we build
multiple sequential correlation volumes in time using event data. Second, we
use Bézier curves to index these correlation volumes at multiple timestamps
along the trajectory. Third, we use the retrieved correlation to update the
Bézier curve representations iteratively. Our method can optionally include
image pairs to boost performance further. The proposed approach outperforms
existing image-based and event-based methods by 11.5 % lower EPE on DSEC-Flow.
Finally, we introduce a novel synthetic dataset MultiFlow for pixel trajectory
regression on which our method is currently the only successful approach.

###### Keywords:

Event Cameras, Optical Flow, Non-linear Flow

|  |  |
---|---|---|---
(a) First Frame |  (b) Second Frame |  (c) Events |  (d) Prediction
Figure 1: Our method, trained only on simulated events, predicts dense pixel
trajectories in continuous-time on real sequences.

##  1 Introduction

Optical flow estimation is a fundamental low-level vision task that informs
about motion in pixel space. It has numerous practical applications in
computational photography and videography, video compression, inverse
graphics, object tracking, and robotics.

The majority of previous work addresses this problem by finding dense
correspondences between two images. However, applications that use optical
flow often rely on linear motion models. For example, temporal image synthesis
tasks in dynamic scenes, such as frame interpolation or dynamic view synthesis
[32, 25], tend to rely on such assumptions. If the frame-rate is very high,
such a linear motion model can be reasonable, but it breaks down if long-range
correspondences are sought [45, 46] or if the camera device is not capable of
capturing high fps video due to storage or power consumption constraints [42].
On top of this, estimating pixel motion in the wild must deal with occlusions
and objects leaving the camera field of view. Modern optical flow methods [21]
attempt to address partially occluded objects but still completely fail if an
independent object is only visible in one of the two frames (see Figure 6).

To address all of these challenges at once we use event cameras. In contrast
to frame-based sensors, which capture images at regular intervals, event
cameras register per-pixel brightness changes asynchronously and at very high
temporal resolution (microseconds). Additionally, they are robust to motion
blur and have a very high dynamic range [10]. For a survey on event cameras,
we refer to Gallego et al. [10]. All these advantageous features make them
very attractive for optical flow estimation. This has inspired prior work to
propose methods to predict pixel displacements similar to two-frame optical
flow [53, 14]. However, event cameras provide visual cues in continuous-time,
which means that predicting discrete pixel displacements ignores much of the
device’s potential. With our method, we propose to close this gap with the
following methodological innovations:

  * •

Instead of predicting pixel displacements, we generalize this concept and
estimate control points of Bézier curves for each pixel. As a result, our
method can regress the trajectory of each pixel at arbitrary times. It also
enables the second innovation:

  * •

We use multiple correlation volumes in time to search for pixel
correspondences. We then use the estimated Bézier curves to retrieve
correlation features of all correlation volumes simultaneously. This enables
the incorporation of motion prior and facilitates the task of finding accurate
pixel trajectories.

  * •

The use of image data is optional. We show that our approach can also work
purely with event data and enable dense, pixel-wise trajectories. Furthermore,
we show that a combination of image and event data outperforms single-modality
approaches on both real and synthetic datasets.

The full evaluation of our method is currently not possible with existing
public datasets. To the best of our knowledge, there is no dataset with moving
objects and accurate ground truth for pixel trajectories from a single
reference view. Furthermore, we also require both event and image data which
is by itself a rare combination. Therefore, we propose a new dataset inspired
by the FlyingChairs dataset [8] that fulfills the aforementioned
specifications. The overall dataset consists of approximately 10000 training
and 2000 test sequences with moving objects undergoing similarity
transformations in continuous-time. We use this dataset to quantitatively
evaluate our method and to show promising sim2real transfer performance.
Finally, we provide quantitative experiments on DSEC-Flow [14], a real-world
dataset featuring a multi-camera setup consisting of event and global shutter
cameras with ground truth pixel displacement maps.

##  2 Related Work

###  2.1 Optical Flow

For conciseness, we focus mostly on neural-network-based (NN-based) methods.

####  2.1.1 Image-based

The vast majority of neural network-based optical flow method considers the
task of estimating dense pixel displacements from a pair of frames [41, 44,
40, 16, 22, 43, 9, 2, 49, 18, 50, 51, 8]. A common component of many highly
successful methods are explicit correlation volumes that guide the matching
process. This inductive bias enables high performance and data efficiency [41]
as well as strong cross-dataset generalization [49].

Multi-Frame optical flow estimation has been mostly explored in the self-
supervised learning setting [15, 19, 28] or optimization-based literature [20,
37, 11]. Some NN-based approaches use an additional pair of frames to
initialize the optical flow prediction [41] or use warped flow as input [36]
to a second stage. From these multi-frame approaches, Slow Flow [20] is
conceptually most related. Slow Flow estimates optical flow $\mathbf{F}_{1\to
N}$ given a sequence of $N$ images at high frame rate (1000 fps) with dense
tracking using an optimization-based approach. However, it requires a high-
speed camera and complex occlusion handling that incurs a trade-off between
drift and accurate prediction at motion boundaries.

Figure 2: Overview of the framework. For concise presentation, we illustrate
the purely event-based approach. We create multiple correlation voxel grids in
a sequence to compute correlation volumes. Bézier curves $\mathbf{B}$,
parameterized by control points $\mathcal{P}$, represent the continuous-time
pixel trajectories. Along these trajectories, we index correlation volumes at
their associated timestamps. An update operator uses the output of this lookup
operation to update the Bézier curves. Finally, this loop is repeated in an
iterative fashion.

####  2.1.2 Event-based

Event-based optical flow algorithms can be categorized into four categories:
(i) Asynchronous methods [5, 12] using the Lucas-Kanade algorithm [27]. (ii)
Plane fitting-based methods that exploit the local plane-like shape of spatio-
temporal event streams [6, 30]. (iii) Variational optimization-based
approaches [4, 33] that incorporate image data[33] or simultaneously estimate
image intensity [4]. (iv) Learning-based approaches, most of which are trained
via self-supervision [53, 54, 24, 34, 47]. Supervision is provided either by
images[53, 54, 24] or events[34, 47]. Our approach is related to E-RAFT [14],
which adapts the RAFT [41] framework to event data to leverage correlation
features from cost volumes to estimate dense pixel correspondences for large
displacements. In contrast to E-RAFT or RAFT, we predict pixel trajectories
using multiple views while we also show the advantages of combining events and
frames.

###  2.2 Continuous Tracking

Continuous-time trajectory estimation of camera poses has been proposed in the
context of rolling shutter compensation [23] as well as visual-intertial
odometry for event cameras [31]. Instead, we are interested in regressing
pixel-trajectories, which is more closely related to high-speed feature
tracking for event cameras [12, 38, 1, 52]. Our approach is related to the
work of Seok et al. [38] who use quadratic Bézier curves to sparsely track
features by maximizing the variance of the image of warped events on local
patches. In contrast to our work, this method can only sparsely track features
where events are present, cannot incorporate learned priors from data, and
does not offer the possibility to include images.

###  2.3 Datasets for Optical Flow

Existing datasets can be categorized as image-based optical flow datasets and
event-based optical flow datasets.

####  2.3.1 Image-based

The seminal work of FlowNet [8] proposed a large synthetic dataset called
”FlyingChairs” to train their CNN. FlyingThings3D[29] introduced a synthetic
stereo-video dataset with scene flow ground truth. AutoFlow[39] proposes to
render large-scale training data with rich data augmentation in 2D for optical
flow estimation. These datasets, however, do not provide event data and ground
truth pixel trajectories beyond two views.

####  2.3.2 Event-based

MVSEC [53] contains 5 outdoor driving sequences and 4 indoor sequences.
However, optical flow ground truth from MVSEC suffers from inaccuracies in
calibration [14] and only features very small displacements. DSEC-Flow, a more
recent dataset, addresses these shortcomings by providing accurate but sparse
optical flow groundtruth. The main downside of both MVSEC and DSEC-Flow is
that they both do not include optical flow ground truth for dynamic objects.

In contrast to the aforementioned datasets, our synthetic dataset ”MultiFlow”
features accurate and dense ground truth for pixel trajectories as well as
image and event data. The concept of pixel trajectories is a generalization of
pixel displacement and can be used to supervise and evaluate multi-view
approaches.

##  3 Method

The presented approach uses components from RAFT [41]. We aim at keeping this
section self-contained but skip details that are explained in the
aforementioned publication. Figure 2 provides an overview of the methodology.

###  3.1 Problem Definition

Our method is tasked with estimating a function

$\displaystyle\mathbf{B}:\mathcal{T}\times\mathbb{N}_{0}\times\mathbb{N}_{0}$
$\displaystyle\to\mathbb{R}^{2}$ (1) $\displaystyle(\tau,x,y)$
$\displaystyle\mapsto\mathbf{B}(\tau,x,y)$ (2)

that describes the per-pixel trajectories in time on the image plane.
$\mathcal{T}=\mathbb{R}\cap[0,1]$ describes the domain consisting of the
normalized time
$\tau(t)=\nicefrac{{(t-t_{\text{r}})}}{{(t_{\text{t}}-t_{\text{r}})}}$ , where
$\tau=0$ corresponds to the reference time $t_{\text{r}}$ from which the pixel
trajectory starts and $\tau=1$ corresponds to the target time $t_{\text{t}}$
when the pixel trajectory ends. This formulation can be used to find the pixel
displacement at any time between $t_{\text{r}}$ and $t_{\text{t}}$.

###  3.2 Input Data Preparation

The proposed approach uses features extracted from event data and optionally a
pair of images to further boost performance.

####  3.2.1 Multi-View Event Representations

Event cameras have a fundamentally different working principle than frame-
based cameras. Instead of acquiring frames, event cameras receive asynchronous
events with high temporal resolution. An event
$e_{k}(t)=(x_{k},y_{k},t,p_{k})$ is a tuple containing information about a
pixel $(x_{k},y_{k})$ for which a positive or negative brightness change
$p_{k}$ was registered at time $t$. Note that the time $t$ typically has
microsecond resolution, which provides precise temporal information about
motion in the scene.

Figure 3: Voxel Grid Generation. Events are interpolated into a voxel grid
representation before being divided into sub-voxel grids that are either used
as input to the context encoder or the correlation encoder.

The first step of the feature extraction pipeline is the construction of a
discrete spatio-temporal representation from a sequence of events. For our
experiments, we choose the voxel grid representation by Zhu et al. [54] due to
its simplicity and possibility to extract features in a sliding window. An
overview of this process is provided by Figure 3.

Given the task of estimating continuous pixel trajectories from
$t_{\text{r}}$, the reference time, to the target time $t_{\text{t}}$, Our
method first computes a _base_ voxel grid, visualized in red in Figure 3,
consisting of $M+N-1$ discrete bins along the time dimension via interpolation
of event data [54]. The first $M$ bins are computed from events prior to the
reference time $t_{\text{r}}$ and are required to extract the correlation
features later in the pipeline. The last $N$ bins are computed mostly from
events during the time window $[t_{\text{t}},t_{\text{r}}]$ when the pixel
trajectories are estimated. These $N$ bins are required to extract the context
features.The bin at the reference timestamp $t_{\text{r}}$ will be reused by
both correlation and context features such that the base voxel grid consists
of $M+N-1$ bins.

From this base voxel grid, we first extract the context voxel grid
$V^{\text{CT}}$, visualized in yellow in Figure 3. The next step is the
extraction of up to $N$ correlation voxel grids $V^{\text{CR}}_{i}$,
visualized in green in Figure 3, in a sliding-window fashion from the base
voxel grid. Each of these correlation voxel grids contain information about
the contrast differences in the scene at slightly different times. They will
later be used to guide the search for pixel trajectories via lookup
operations. We refer to the correlation voxel grids also as _views_ because
they each represent a distinct timestamp. If we choose to extract at least two
correlation voxel grids apart from the reference timestamp $t_{\text{r}}$, we
label the method as _multi-view_ approach. Find further algorithmic details in
the appendix.

####  3.2.2 Optional Frame-based Input

As we show in the experimental results, the performance of the proposed method
improves if image data is used in addition to events only. To do so, we
acquire a reference frame $I_{\text{r}}$ at $t_{\text{r}}$ and a target frame
$I_{\text{t}}$ at $t_{\text{t}}$. These frames contain richer information
about texture at the boundary timestamps of the regressed pixel trajectories
and thus simplify the correspondence search.

###  3.3 Feature Extraction

Features are extracted from input images and voxel grids with a convolutional
network. The basic architecture of the encoders follows prior work [41, 14].
The encoders extract $D=256$ dimensional features from their input at $1/8$th
of the original resolution using residual blocks with striding for
downsampling the feature maps. From now on, we use $H^{\prime}=H/8$ and
$W^{\prime}=W/8$ to refer to the downsampled resolution.

The context feature encoder $f_{\text{CT}}:\mathbb{R}^{(N+3)\times H\times
W}\mapsto\mathbb{R}^{D\times H^{\prime}\times W^{\prime}}$ concatenates the
voxel grid $V^{\text{CT}}$ with the frame $I_{\text{r}}$, if available, to
extract combined features. The reference frame $I_{\text{r}}$ informs the
network about the absolute intensity of the reference pixels while the context
voxel grid $V^{\text{CT}}$ provides rich information about motion during the
time duration of the pixel trajectories. However, contrast information is not
explicitly considered yet, which will be provided by the correlation feature
encoders.

The correlation encoder $f_{\text{CR}}^{V}:\mathbb{R}^{M\times H\times
W}\mapsto\mathbb{R}^{D\times H^{\prime}\times W^{\prime}}$ for event
representations computes features from the $N$ correlation voxel grids in
parallel, by sharing the weights. The result of this operation are $N$ feature
maps, each assigned to the timestamp associated with the last bin of each
correlation voxel grid.

If in addition, a pair of frames is available, we extract image features with
an additional encoder $f_{\text{CR}}^{I}:\mathbb{R}^{3\times H\times
W}\mapsto\mathbb{R}^{D\times H^{\prime}\times W^{\prime}}$.

###  3.4 Multi-View Correlation Volumes

We compute correlation volumes to guide correspondence search in space and
time by associating subsequent views with the reference view. This association
is visualized in Figure 3.5 and 2. We use the features created from voxel grid
$V_{0}^{CR}$ and optionally $I_{\text{r}}$ to create the feature maps for the
reference view. For each subsequent view $i$, we compute features from voxel
grid $V_{i}^{CR}$ to compute the correlation volume $C_{i}^{1}$ as in
definition (3). This step is repeated until view $N-1$ is reached. For the
final/target view, we optionally compute a correlation volume $C_{I}^{1}$ from
the boundary image features as in definition (4). A more detailed description
is available in the supplementary material.

$\displaystyle\mathbf{C}_{n}^{l}\left(f_{\text{CR}}^{V}(V_{0}^{\text{CR}}),f_{\text{CR}}^{V}(V_{n}^{\text{CR}})\right)$
$\displaystyle\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times\nicefrac{{H^{\prime}}}{{2^{l}}}\times\nicefrac{{W^{\prime}}}{{2^{l}}}}$
(3)
$\displaystyle\mathbf{C}_{I}^{l}\left(f_{\text{CR}}^{I}(I_{\text{r}}),f_{\text{CR}}^{I}(I_{\text{t}})\right)$
$\displaystyle\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times\nicefrac{{H^{\prime}}}{{2^{l}}}\times\nicefrac{{W^{\prime}}}{{2^{l}}}}$
(4)

###  3.5 Iterative Multi-View Flow Updates

This section describes the update scheme that uses correlation volumes
introduced in the previous section 3.4.

Figure 4: Correlation-volume pairs with indicated lookup direction. The upper
row shows lookups from voxel grid features and the lower row shows lookups
from image features. Figure 5: Illustrative example of a Bézier prediction and
$N_{k}=4$ groundtruth flow maps from which the loss function (9) is computed.

####  3.5.1 Continuous-Time Flow

We seek a representation of the function $\mathbf{B}(\tau,x,y)$, as introduced
in equation (1), that generalizes the conventional displacement prediction of
two-frame optical flow methods. To achieve this, we choose to use Bézier
curves defined as

$\mathbf{B}(\tau,x,y)=\sum_{i=0}^{n}\begin{pmatrix}n\\\
i\end{pmatrix}(1-\tau)^{n-i}\tau^{i}\mathbf{P}_{i}(x,y).\;$ (5)

$\mathbf{B}(\tau,x,y)$ describes the displacement of pixel $(x,y)$ for the
normalized time $\tau\in[0,1]$. As an example, the Bézier curves of degree
$n=1$ simply represent a linear trajectories in space and time that is fully
described by $\mathbf{P}_{1}$. The task of our method, however, is to regress
the parameter set

$\mathcal{P}=\left\\{\mathbf{P}_{1},\ldots,\mathbf{P}_{n}\right\\}$ (6)

$\mathbf{P}_{0}=\mathbf{0}$ does not have to be estimated because it defines
the starting point of the trajectory that coincides with the reference pixels.
The advantages of working with Bézier curves is that they are fast to evaluate
and can be concisely summarized by a set of parameters $\mathcal{P}$.

The degree of the Bézier curves is a fixed parameter. We found that it is best
to set it equal to the number of supervision points along the trajectory for
an accurate trajectory prediction (details in supplementary material).

##### Iterative Bézier Updates

The Bézier curves are initialized with $\mathbf{B}=\mathbf{0}$ such that
$\mathcal{P}=\\{\mathbf{0},\ldots,\mathbf{0}\\}$. Each iteration adds an
update to the parameter set $\mathcal{P}$:
$\mathbf{P}_{i}^{k+1}=\mathbf{P}_{i}^{k}+\Delta\mathbf{P}_{i},\;\forall
i\in\\{1,\ldots,n\\}$

####  3.5.2 Multi-View Correlation Lookup

Similar to the original RAFT implementation, we use lookup operations to
extract features from the correlation volumes. In contrast, however, we
extract features from multiple correlation volumes, each associated to a
unique normalized timestamp. Given the current estimate of the Bézier control
points $\mathcal{P}$, we map each pixel $\mathbf{x}=(x,y)$ at time $\tau=0$ to
the estimated corresponding pixel location
$\mathbf{x}^{\prime}(\tau)=(x^{\prime}(\tau),y^{\prime}(\tau))$ at time
$\tau$:

$\mathbf{x}^{\prime}(\tau)=\mathbf{x}+\mathbf{B}(\tau,\mathbf{x})$ (7)

Similar to RAFT, the lookup is performed in a local neighborhood $\mathcal{N}$
around the corresponding pixel location $\mathbf{x}^{\prime}(\tau)$:

$\mathcal{N}(\mathbf{x}^{\prime}(\tau))=\left\\{\mathbf{x}^{\prime}(\tau)+\mathbf{dx}\,|\,\mathbf{dx}\in\mathbb{Z}^{2},||\mathbf{dx}||_{\infty}\leq
r\right\\}$ (8)

Lookups are performed on all available correlation volumes with bilinear
sampling. We use a constant lookup radius of $r=4$, as in the original RAFT
implementation, to increase the effective lookup radius. Finally, the values
from the union of lookup operations is concatenated into a single feature map.

####  3.5.3 Upsampling of Bézier Curves

The Bézier control points are estimated at $\nicefrac{{1}}{{8}}$-th of the
original resolution. Since $\mathbf{B}(\cdot)$ is linear in the control points
$\mathcal{P}$, we can upsample the Bézier curves to the full resolution using
convex upsampling [41]. As a result, the Bézier curves at the full resolution
will be a learned convex combination of $3\times 3$ grids of Bézier curves at
the lower resolution.

###  3.6 Supervision

We supervise the model with $N_{k}$ ground truth flow maps along the
trajectory, visualized in Figure 3.5.

$\mathcal{L}=\frac{1}{N_{k}}\sum_{i=1}^{N_{i}}\gamma^{N_{i}-i}\sum_{k=1}^{N_{k}}||\mathbf{f}_{gt}(\tau_{k})-\mathbf{B}_{i}(\tau_{k})||_{1}$
(9)

where $\mathbf{B}_{i}$ is the Bézier curve at iteration $i$,
$\tau_{k}\in[0,1]$ are the evaluation timestamps of the Bézier curves, and
$\gamma=0.8$. For a single displacement map, such as in two-frame optical
flow, the corresponding parameters are $N_{k}=1$ and $\tau_{1}=1$.

##  4 MultiFlow Dataset

To the best of our knowledge, there are no publicly available datasets with
ground truth for pixel trajectories, which is more general than pixel
displacements, in combination with events and images. Therefore, we create a
new synthetic dataset to evaluate the proposed approach.

###  4.1 Data Generation Procedure

We generate 9895 training and 1985 test sequences in simulation for MultiFlow.
The background images are sampled from the Flickr30K dataset [48]. The
foreground objects are extracted from 650 PNG images, 450 for training and 200
for testing, by masking out transparent regions using the alpha channels.

The duration of each sequence is defined to be one second. We randomly sample
3 to 4 data points over similarity transformations in 2D for each foreground
and background object. Interpolation with piecewise cubic spline polynomials
provide the continous-time transformation of all objects throughout the
duration of each sequence. The generated similarity transformation
trajectories are used to both render images and compute ground truth pixel
trajectories. To generate events, we use ESIM [35] on frames rendered at 1000
frames per second on the full sequence. We refer to the supplementary material
for dataset examples and a more detailed description.

|  |  |
---|---|---|---
|  |  |
|  |  |
(a) First Frame |  (b) Second Frame |  (c) Ours |  (d) RAFT+GMA [21]
Figure 6: Predictions of our method (c) and the strongest baseline, RAFT+GMA
[21] (d), on the MultiFlow dataset. Predictions are shown in blue and the
ground truth trajectory is visualized in red. The background is a colorization
of the ground truth flow to highlight moving objects. Events are not shown for
conciseness but are also used by our method. Best viewed in PDF form.

##  5 Experiments

This section has four distinct purposes. First, to show quantitative real-
world performance on DSEC-Flow on traditional two-view metrics. Second, to
show quantitative performance on the _main task_ on MultiFlow: predicting
continuous-time pixel trajectories. Third, to show qualitative performance for
sim2real transfer using models trained on our MultiFlow dataset. Finally, an
ablation study highlights the importance of the proposed components.

##### Implementation Details

Our models are implemented in Pytorch and trained from scratch with random
weights on each dataset. We use AdamW [26] with gradient clipping in the range
of $[-\mathbf{1},\mathbf{1}]$ and a OneCycle learning rate with 200k
iterations and a batch size of 3. On both datasets, we perform random
horizontal and vertical flipping as well as random cropping of the input data.
For all experiments, our method uses 5 views to compute the cost volumes.

Input EPE AE 1PE 2PE 3PE EV-FlowNet [53] E 2.32 - 55.4 29.8 18.6 E-RAFT [14] E
0.79 2.85 12.74 4.74 2.68 RAFT [41] I 0.78 2.44 12.40 4.60 2.61 RAFT + GMA
[21] I 0.94 2.66 12.98 5.08 2.96 Ours E 0.75 2.68 11.90 4.41 2.44 Ours E+I
0.69 2.42 9.70 3.42 1.88

Table 1: Results on DSEC-Flow. Our method outperforms prior work even when
only using events. Adding image data further improves performance.

###  5.1 DSEC-Flow

DSEC-Flow [14, 13] is a driving dataset with stereo event and global shutter
cameras. The purpose of this experiment is to show the quantitative
performance of our approach on real-world data. However, we can only test on
traditional two-view optical flow metrics because no pixel trajectory ground
truth is available on DSEC-Flow. In other words, for displacement prediction,
the quality of the intermediate continuous trajectory is irrelevant. Instead,
only the final displacement prediction is evaluated using end-point-error
(EPE) and angular error (AE) metrics [3]. We also report the $X$-point error,
a metric that reports the percentage of pixels with EPE higher than $X$
pixels.

For our experiments that combine image and event data, we warp images to the
event camera. Due to the small baseline between both cameras, we can warp the
image to a plane at an infinite depth and re-project it into the event camera
coordinate frame. The disparity is negligible.

We use $M=N=5$ bins for both the context and correlation voxel grid (see
Section 3.2.1). We also experimented with a higher number of bins but did not
observe significant improvements. Our model is supervised with the 2-view
version of our loss proposed in equation (9). This loss function is used
because DSEC-Flow only provides ground truth for pixel displacements. Finally,
we choose a degree of 2 for the Bézier curves mostly to account for non-linear
motion due to change in depth.

The performance of EV-FlowNet [53] and E-RAFT is taken from Gehrig et al.
[14]. We additionally train the frame-based RAFT model [41] and GMA [21], an
addition over RAFT, to also compare against purely frame-based approaches.

####  5.1.1 Quantitative Evaluation

Table 1 summarizes our results on the DSEC-Flow test set. Our approach, using
both event data and frames, achieves 0.69 EPE which is 11.5% lower than RAFT
[41] and 8 % lower than our own method using only event data. Furthermore, our
purely event-based approach achieves an EPE of 0.75 which is 5% lower than the
EPE of 0.79 that E-RAFT achieves. Overall the performance of E-RAFT [14] is
comparable to RAFT while EV-FlowNet [53] is not competitive. In our
experiments, the GMA version of RAFT does not outperform the RAFT baseline on
this dataset.

These results indicate that the proposed method, even though designed to work
for regressing pixel trajectories, is competitive with two-view approaches on
the task of pixel displacement prediction. Note that we have not used any
additional ground truth information compared to the baselines.

Trajectory Metrics 2-View Metrics Input TEPE TAE EPE AE E-RAFT [14] E (6.70)
(18.44) 7.56 6.19 RAFT [41] I (6.89) (19.31) 7.42 6.71 RAFT + GMA [21] I
(5.14) (16.35) 1.22 1.30 Ours E 1.48 3.64 2.69 3.86 Ours E+I 1.08 2.77 1.88
2.70

Table 2: Results on MultiFlow. TEPE and TAE are the trajectory version of EPE
and AE. Metrics in brackets are computed using a linear motion model because
these method are not able to estimate trajectories.

###  5.2 MultiFlow

The experiments on MultiFlow assess the pixel trajectory regression
capabilities. To achieve this, we introduce an extension of EPE and AE to
trajectories.

TEPE
$\displaystyle=\frac{1}{N_{k}}\sum_{k}^{N_{k}}\text{EPE}(\mathbf{f}_{pred}(t_{k}),\mathbf{f}_{gt}(t_{k}))$
(10)

We define TAE in an analogous way.

We train three previously published baselines for an extensive comparison.
First, RAFT [41] and RAFT+GMA [21] for a comparison against frame-based
approaches. Second, E-RAFT [14] for a comparison against a recent event-based
approach. We focus on these architectures because they are related to our
approach and achieve state-of-the-art performance on public benchmarks [7,
13]. We train our method on the loss defined by equation (9) while the two-
view approaches are trained on the two-view version of the loss [41]. We
supervise our methods with $10$ flow maps along the trajectory using loss (9)
and set the Bézier curve degree to 10 according to section 3.5.1.

|  |  |
---|---|---|---
|  |  |
|  |  |
(a) First Frame |  (b) Second Frame |  (c) Ours |  (d) RAFT+GMA [21]
Figure 7: Predictions of our method (c) and RAFT+GMA [21] (d), on the HS-ERGB
[42] dataset. Predictions are shown in blue. The background is a colorization
flow to highlight moving objects. Events are not shown for conciseness.

####  5.2.1 Quantitative Evaluation

Table 2 summarizes the quantitative results. The baseline models are only able
to predict pixel displacements. Therefore, for a fairer comparison, we use a
linear motion model to compare against our methods on the trajectory metrics
TEPE and TAE. Our methods achieve up to 4.8 times lower trajectory end-point
errors compared to linear motion model baselines. This discrepancy occurs
mostly because our models are successfully predicting the pixel trajectories
while the linear motion model is insufficient for the complexity of pixel
motion in this dataset. As expected, combining events and frames further
improves performance by a large margin on all metrics for our approaches. Both
RAFT [41] and E-RAFT [14] underperform compared to our approach on the 2-view
metrics for which they were designed. A notable exception is the RAFT+GMA [21]
which performs best on the 2-view metrics among all methods. However, RAFT+GMA
is not designed for trajectory prediction and therefore falls short in TEPE
and TAE compared to our approach.

####  5.2.2 Qualitative Analysis

Figure 6 illustrates the predictions of our model, using events and frames, in
comparison to RAFT+GMA [21]. Our approaches successfully predict the
continuous trajectory, even if the objects leave the field of view. This is
not the case for the two-view method RAFT+GMA, as seen in the first row of
Figure 6. Evidently, it fails at predicting the motion of objects that are
missing in the second view because it is impossible to establish a match. The
second row shows that our method can accurately predict pixel trajectories
while RAFT+GMA does well in predicting the final pixel displacement.

###  5.3 Qualitative Sim2Real Results

To qualitatively assess the real-world capabilities of our method, we use our
purely event-based model, trained only on the simulation dataset MultiFlow,
and show predictions on the HS-ERGB dataset [42] in Figure 7. Although our
method is only trained on MultiFlow, it can correctly predict non-linear
motion of moving objects while the frame-based baseline [21] fails due to
large motion and ambiguities in the input frames.

Trajectory Metrics 2-View Metrics Input TEPE TAE EPE AE w/o Bézier E+I 6.26
18.16 6.46 5.23 w/o multi-view E+I 1.56 3.73 2.29 3.13 Reference E+I 1.08 2.77
1.88 2.70 w/o Bézier E 6.55 18.39 6.98 5.87 w/o multi-view E 2.24 5.14 3.76
5.03 Reference E 1.48 3.64 2.69 3.86

Table 3: Ablation experiments on MultiFlow. Final model settings are
underlined.

###  5.4 Ablation Study

We study the contributions of the two main components of our approach on the
MultiFlow dataset, summarized in Table 3. The two reference models are the
full models as described in the methodology. First, we remove the Bézier curve
regression and note a drastic drop in performance on all metrics. Removing the
Bézier curves is equivalent to assuming linear motion. Linear motion is not
representative for this dataset and hampers the correlation lookup. Second, we
remove the multi-view lookup scheme and note that both TEPE and EPE increase
between 22% and 51% with respect to the event-based and hybrid reference
model. We conclude that both components are important for achieving the best
results. Ablation studies on the loss function (9) and Bézier curve degree as
well as runtime and parameter details can be found in the appendix.

##  6 Conclusion

We have introduced a method for estimating continuous-time pixel trajectories
from events and frames. The proposed approach regresses Bézier curves and uses
the pixel trajectories to extract correlation features. Our experimental
results demonstrate that the proposed method can accurately predict continuous
pixel trajectories while at the same time outperforming strong baselines not
only in simulation but also on real data. Finally, our sim2real results
suggest that the new MultiFlow dataset can also be used to pretrain pixel
trajectory regression models for downstream applications.

##  7 Acknowledgment

This work was supported by Huawei Zurich Research Center; by the National
Centre of Competence in Research (NCCR) Robotics (grant agreement No.
51NF40-185543) through the Swiss National Science Foundation (SNSF), and the
European Research Council (ERC) under grant agreement No. 864042
(AGILEFLIGHT).

## Appendix

##  8 Additional Ablation Studies and Model Details

This section provides additional ablation studies and runtime metrics for the
interested reader.

###  8.1 Degree of the Bézier Curves

The goal of this experiment is to answer the question: Which Bézier curve
degree is appropriate for my model?

To answer this question, we train 4 different models with Bézier curve degrees
1, 5, 10 and 15 using frames and events as input to the model. The supervision
on pixel trajectories at training time is at 20 Hz, that we supervise the
model on the pixel trajectory every 50 milliseconds.

|  TEPE |  TAE
---|---|---
Bézier Degree | 20 Hz | 100 Hz | 20 Hz | 100 Hz
1 | 6.26 | 6.00 | 18.16 | 18.68
5 | 1.18 | 1.10 | 3.06 | 3.09
10 | 1.08 | 1.01 | 2.77 | 2.99
15 | 1.04 | 1.02 | 2.70 | 3.46
Table 4: Trajectory errors of our method with a varying number of Bézier curve
degree evaluated with either 20 Hz (every 50 ms) or j100 Hz (every 10ms)
ground truth along the trajectories. All variants were trained with trajectory
supervision at 20 Hz. Although a Bézier degree of 15 improves the 20 Hz
performance, the error metrics increase when the evaluation takes place at 100
Hz. Best performance in bold.

Intuitively, one would think that a higher Bézier degree always improves the
performance. Table 4 indeed shows that a higher Bézier degree improves
performance, when the models are evaluated at 20 Hz. The models were also
trained with a supervision signal at 20 Hz, that is with supervision at
regular time intervals of 50 milliseconds. Interestingly, an evaluation of
these models at a higher frequency of 100 Hz reveals a decrease in performance
for the model with a Bézier degree of 15. The reason for that is that we
observe a test-time overfitting of the predicted trajectory at the timestamps
on which the model was supervised. In our case, we supervised the model at
regular intervals of 50 milliseconds. An evaluation on the same timestamps
does not reveal this phenomenon, which is why an evaluation at 100 Hz is
required.

Hence, the answer to the original question is: For accurate pixel trajectory
predictions, the Bézier degree should be chosen equal to the number of
supervision points along the pixel trajectories.

The degree can be set higher than the number of supervision points to improve
performance on the timestamps of interest. However, the pixel trajectories may
not be as accurate anymore. An example for this is our DSEC-Flow experiments
where we have only 1 supervision point (2-view) but found that a Bézier degree
of 2 improves the performance on the 2-view metric. This is a reasonable
approach as long as the downstream application does not require accurate pixel
trajectories.

###  8.2 Loss Function

This experiment examines the influence of the trajectory loss on trajectory
and 2-view metrics. We train 2 additional models with the 2-view loss. One
using only events and a second one using both frames and events. Table 5
summarizes the results.

For both input variations, the error metrics substantially decrease when the
model is trained with the trajectory loss. This reduction is evident for both
the trajectory metrics and also the 2-view metrics. This indicates, that the
trajectory loss function is a better choice for the proposed method than the
2-view loss.

Finally, the results in Table 5 indicate that we could further reduce the
errors of the DSEC-Flow experiment with the appropriate ground truth.

Trajectory Metrics 2-View Metrics Input TEPE TAE EPE AE 2-View Loss E+I 16.18
19.37 2.96 4.19 Trajectory Loss E+I 1.08 2.77 1.88 2.70 2-View Loss E 16.73
21.63 5.80 8.71 Trajectory Loss E 1.48 3.64 2.69 3.86

Table 5: The trajectory loss refers to the loss function of equation (10) of
the main paper, with $N_{k}>1$. The 2-View loss refers to the same loss
function with $N_{k}=1$. Training our method with the trajectory loss leads to
drastically lower errors even for the 2-View metrics.

###  8.3 Number of Correlation Lookups

Table 6 shows that increasing the number of correlation lookups improves
performance consistently on all metrics. However, substantially increasing the
number of correlation volumes incurs higher memory consumption and increases
computational demand. This result suggests a trade-off between performance and
compute and memory requirements.

Note that the number of correlation volumes is the number of correlation
lookups (in time) + 1 because the correlation volume at the reference time has
to be accounted for.

|  Trajectory Metrics |  2-View Metrics
---|---|---
# Correlation Lookups | TEPE | TAE | EPE | AE
1 | 1.56 | 3.73 | 2.29 | 3.13
3 | 1.13 | 2.87 | 1.96 | 2.75
5 |  1.08 |  2.77 |  1.88 |  2.70
Table 6: Increasing the number of correlation volumes/lookups in time improves
performance. The underlined row refers to the reference model used in the
experiments.

###  8.4 Runtime Metrics

Table 7 shows inference time in milliseconds, parameter count in millions of
parameters and memory consumption at inference time in Gigabytes on samples
from the MultiFlow dataset. Overall, our proposed method has higher demands on
the presented metrics compared to RAFT [41], which is expected because it is a
generalization of the RAFT architecture.

| Input | Inference time [ms] | Params | Memory [GB]
---|---|---|---|---
RAFT [41] | I | 61 | 5.3 M | 1.20
Ours | E | 76 | 5.6 M | 1.65
Ours | E+I | 97 | 5.9 M | 1.65
Table 7: Comparison of inference time, parameter count and memory consumption
on MultiFlow. These numbers have been obtained on a Titan RTX GPU with an
implementation using Pytorch version 1.10.2

##  9 Implementation Details

This section supplements the main paper with more detailed descriptions of
selected parts of the methodology.

###  9.1 Input Data Preparation

####  9.1.1 Multi-View Event Representations

Event cameras have a fundamentally different working principle than frame-
based cameras. Instead of acquiring frames, event cameras receive asynchronous
events with high temporal resolution. An event
$e_{i}=(x_{i},y_{i},t_{i},p_{i})$ is a tuple containing information about a
pixel $(x_{i},y_{i})$ for which a positive or negative brightness change
$p_{i}$ was registered at time $t_{i}$. Time $t_{i}$ typically has microsecond
resolution, which provides precise temporal information about motion in the
scene.

The first step of the feature extraction pipeline is the construction of a
discrete spatio-temporal representation from a sequence of events. For our
experiments, we choose the voxel grid representation by Zhu et al. [54] due to
its simplicity and possibility to extract features in a sliding window. An
overview of this process is provided by Figure 3.

Given the task of estimating continuous pixel trajectories from
$t_{\text{r}}$, the reference time, to the target time $t_{\text{t}}$, we
first compute the temporal discretization $\Delta
t=\frac{t_{\text{t}}-t_{\text{r}}}{N-1}$. $N$ is the number of bins that we
later use to compute context features from event data. Having computed $\Delta
t$, we choose $M$ to be the number of bins that we will use to compute
correlation features from event data. The bin with index $M-1$ corresponds to
the reference timestamp $t_{\text{r}}$ and will be reused by both correlation
and context features such that the _base_ voxel grid consists of $M+N-1$ bins.
This base voxel grid is visualized in red in Figure 3, and we compute it as
follows:

$\displaystyle\begin{aligned}
t_{i}^{*}&=(M+N-1)(t_{i}-t_{0})/(t_{\text{t}}-t_{0})\\\
V(x,y,t)&=\sum_{e_{i}|t_{i}\in[t_{0},t_{\text{t}}]}p_{i}k_{b}(x-x_{i})k_{b}(y-y_{i})k_{b}(t-t_{i}^{*})\\\
k_{b}(a)&=\max(0,1-|a|)\end{aligned}$ (11)

where $t_{0}=t_{\text{r}}-\Delta t\cdot(M-1)$ and $k_{b}$ is the bilinear
interpolation kernel [17].

Figure 3 shows how we extract sub-voxel-grids from the overall voxel grid with
$M+N-1$ bins. To provide contextual information to the network, we extract a
context voxel grid $V^{\text{CT}}\in\mathbb{R}^{N\times H\times W}$ which is
visualized in yellow in Figure 3. It consists of $N$ bins starting with the
bin at index $M-1$, associated with $t_{\text{r}}$, to the target time
$t_{\text{t}}$ associated with the final bin at index $N+M-2$. For the
correlation encoder, we extract a maximum of $N$ voxel grids
$V^{\text{CR}}_{i}\in\mathbb{R}^{M\times H\times W},\;i\in{0,1,...,N-1}$,
visualized in green in Figure 3, consisting of $M$ bins each in a sliding
window fashion.

Note that $N$ is the maximum number of correlation voxel grids but typically
only a subset is used for a better trade-off between compute and performance.

###  9.2 Multi-View Correlation Volumes

We compute correlation volumes to guide correspondence search in space and
time by associating subsequent views with the reference view. This association
is visualized in Figure 3.5. We use the features computed from voxel grid
$V_{0}^{CR}$ and optionally $I_{\text{r}}$ to create the feature maps for the
reference view. For each subsequent view $i$, we compute features from voxel
grid $V_{i}^{CR}$ to compute the correlation volume $C_{i}^{1}$ as in
definition (12). This step is repeated until view $N-1$ is reached. For the
final/target view, we optionally compute a correlation volume $C_{I}^{1}$ from
the boundary image features as in definition (13).

$\displaystyle\mathbf{C}_{n}^{l}\left(f_{\text{CR}}^{V}(V_{0}^{\text{CR}}),f_{\text{CR}}^{V}(V_{n}^{\text{CR}})\right)$
$\displaystyle\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times\nicefrac{{H^{\prime}}}{{2^{l}}}\times\nicefrac{{W^{\prime}}}{{2^{l}}}}$
(12)
$\displaystyle\mathbf{C}_{I}^{l}\left(f_{\text{CR}}^{I}(I_{\text{r}}),f_{\text{CR}}^{I}(I_{\text{t}})\right)$
$\displaystyle\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times\nicefrac{{H^{\prime}}}{{2^{l}}}\times\nicefrac{{W^{\prime}}}{{2^{l}}}}$
(13) $\displaystyle C_{ijkl}(a,b)$ $\displaystyle=\sum_{h}a_{ijh}\cdot
b_{klh}$ (14)

Overall, this process results in a set of correlation volumes expressed in
definitions (15), (16) and (17).

$\displaystyle\mathcal{C}_{V}$
$\displaystyle=\left\\{\\{\mathbf{C}_{1}^{1},\ldots,\mathbf{C}_{1}^{L_{1}}\\},\ldots,\\{\mathbf{C}_{N-1}^{1},\ldots,\mathbf{C}_{N-1}^{L_{N-1}}\\}\right\\}$
(15) $\displaystyle\mathcal{C}_{I}$
$\displaystyle=\left\\{\mathbf{C}_{I}^{1},\ldots,\mathbf{C}_{I}^{L_{I}}\right\\}$
(16) $\displaystyle\mathcal{C}$
$\displaystyle=\mathcal{C}_{V}\cup\mathcal{C}_{I}$ (17)

The set of final correlation volumes also incorporates $L_{i}-1$ pooled, lower
resolution correlation volumes. We found that it is sufficient to use the
following setting in our experiments:

  * •

$L_{I}=4$ and $L_{N-1}=4$: Correlation lookup for the final target view
requires a large receptive field, otherwise the trajectories might not
converge to the end-point.

  * •

$L_{i}=1,\;i=\\{1,...,N-2\\}$: These intermediate correlation volumes are
mostly used to guide the pixel trajectories locally to accurate
correspondences.

##  10 MultiFlow Dataset Generation

This section provides a more formal and detailed overview of the data
generation process that was used to create the MultiFlow dataset.

The duration of each sequence in the dataset is fixed to one second. We sample
either 3 or 4 (random choice of 50% probability each) control points in the
space of pixel translation, scale, and rotation of the foreground (FG) and
background (BG) objects in the image plane. The time assigned to control
points is sampled from the range $[0,1]$ while ensuring strictly increasing
values to maintain the temporal order.

The control points are sampled according to the following discrete-time
process with parameters shown in Table 8:

$\displaystyle
X_{k+1}=\begin{cases}\hat{\gamma}\cdot\hat{X}_{\text{k+1}}^{\text{Det}}+(\hat{\gamma}-1)\cdot\hat{X}_{k+1}^{\text{Stoch}}&(\hat{\alpha}=0)\land(\hat{\beta}=0)\\\
X_{k}&(\hat{\alpha}=1)\lor(\hat{\beta}=1)\end{cases}$ (18)

where

$\displaystyle\hat{X}_{\text{k+1}}^{\text{Det}}=X_{k}+\frac{t_{k+1}-t_{k}}{t_{k}-t_{k-1}}(X_{k}-X_{k-1})$
(19)

refers to the deterministic part of the process that uses a constant velocity
model. It is inspired by the law of conservation of momentum from classical
mechanics. $t_{k}$ is the time associated with control point $k$. To introduce
more variability in the data generation process, we introduce a stochastic
process:

$\displaystyle\text{Translation \& Rotation: }\hat{X}_{k+1}^{\text{Stoch}}$
$\displaystyle=X_{k}+\Delta X$ $\displaystyle\Delta
X\sim\text{Uni}(-\theta,\theta)$ (20) $\displaystyle\text{Scale:
}\hat{X}_{k+1}^{\text{Stoch}}$ $\displaystyle=X_{k}\cdot\Delta X$
$\displaystyle\Delta X\sim(1+\delta_{0})^{(2\delta_{1}-1)},$ (21)
$\displaystyle\delta_{0}\sim\text{Uni}(0,\theta),\;\delta_{1}\sim\text{Bern}(0.5)$

_Uni_ stands for the continuous uniform distribution and _Bern_ refers to the
Bernoulli distribution.

$\hat{\alpha}\sim\text{Bern}(\alpha)$

models the probability that the similarity transformation (translation,
rotation _and_ scale) of the object remains constant during the whole
sequence.

$\hat{\beta}\sim\text{Bern}(\beta)$

is sampled separately for each component (translation, rotation _or_ scale) of
the similarity transformation and models the probability that the component
remains constant during the sequence.

$\hat{\gamma}\sim\text{Uni}(0,\gamma)$

is sampled separately for each component (translation, rotation _or_ scale) of
the similarity transformation and models the degree of the constant velocity
component. For $\hat{\gamma}=1$, the new control point is computed
deterministically with a constant velocity model according to equation (19).
$\hat{\gamma}=0$ corresponds to a purely stochastic update according to
equations (20) or (21). $\hat{\gamma}\in(0,1)$ results in a convex combination
of both as shown in equation (18).

Component | $\alpha$ | $\beta$ | $\gamma$ | $\theta$
---|---|---|---|---
Translation BG | 0.1 | 0 | 0.8 | 30
Rotation BG | 0.7 | 0.6 | 10
Scale BG | 0.4 | 0.3 | 0.15
Translation FG | 0 | 0 | 0.9 | 120
Rotation FG | 0.3 | 0.6 | 30
Scale FG | 0.3 | 0.3 | 0.30
Table 8: Parameters of the distribution of the transformation coefficients for
foreground objects (FG) and background (BG). The values of $\theta$ are shown
in _pixels_ for translation, in _degree_ for rotation and in _dimensionless
quantity_ for scale. The values of $\alpha$ affect the overall similarity
transformation and therefore belong to translation, rotation and scale
simultaneously.

We compute continuous-time trajectories over similarity transformations via
cubic spline interpolation on the $K\in\\{3,4\\}$ acquired control points
${X_{0},\ldots,X_{K-1}}$. It would be possible to sample a higher number of
control points, but we found that 4 control points are sufficient to capture
interesting non-linear motion that we could also observe in real-world
applications of our algorithm (see Figure 7 of the main paper).

##### Pixel Trajectory Ground Truth

The ground truth for the pixel trajectories is computed with respect to a
reference time at $0.4$ seconds. The pixel trajectory is expressed as pixel
displacements at different target timestamps with respect to the original
pixel location at the reference time. Overall, the dataset consists of pixel
trajectory ground truth at intervals of 10 milliseconds up to time $0.9$
seconds. The first $0.4$ and last $0.1$ seconds can be used as additional
context. In our experiments, we use events until timestamp $0.4$ seconds to
build the first correlation voxel grid (index 0 to $M-1$ in Figure 3). Less
context leads to less expressive correlation features which lead to a trade-
off between dataset size and context information.

##  11 Experimental Details on MultiFlow

We generate a voxel grid with 65 bins using events from $t=0.1$ to $t=0.9$.
Our experiments with substantially lower number of bins (e.g. 9 as in the
DSEC-Flow experiments) led to suboptimal performance. We observed that the
reason for this is the overwriting of polarities in the voxel grid. Increasing
the temporal resolution of the voxel grid alleviates this problem. The 65 bins
of the voxel grid are then divided into a context voxel grid with $N=41$ bins
and 6 correlation voxel grids with $M=25$ bins each (see Figure 3). The
correlation voxel grids are extracted in regular intervals of 8 such that the
25th bin corresponds to the reference time and the 65th bin corresponds to the
final timestamp of the trajectory. With these 6 correlation voxel grids we can
compute 5 lookup tables in time to guide the pixel trajectory search of the
network. In principle, we could also have up to 25 correlation voxel grids
with this setup, but that would incur an unreasonable amount of memory and
computational overhead with minimal gain in performance, as indicated in Table
6.

Qualitative predictions of our model, using events and frames, on the test set
of MultiFlow are shown in Figure 8.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
(a) First Frame |  (b) Second Frame |  (c) 2-View Prediction |  (d) Trajectory Prediction
Figure 8: Predictions of our model (E+I) on our MultiFlow test set. (a) shows
the frame at the reference time, (b) the frame at the final timestamp of the
trajectory, (c) shows the predicted pixel displacement between reference and
final timestamp, and (d) shows the full prediction of the pixel trajectories.
The predicted trajectory is shown in blue and the ground truth in red. The
background of (d) is the colorization of the 2-view ground truth.

##  12 Geometric Motivation: Non-linear Continuous Flow is Omnipresent

Continuous-time motion of pixels in image space are in general non-linear in
either space and/or time. To illustrate this, consider a simple scenario of a
camera moving along the principle axis with velocity $v$ in a static scene
with a single 3D point $P$ with depth $D$ at $t=0$. In the camera frame, $P$
is moving towards the camera plane:

$P(t)=(X,Y,Z(t)),\;Z(t)=D-v\cdot t$ (22)

Assuming a focal length $f$ and principle point $(x_{0},y_{0})$, the point
trajectory $p(t)=(x(t),y(t))$ on the image plane is

$p(t)=\begin{bmatrix}f\cdot\frac{X}{D-v\cdot t}+x_{0}\\\
f\cdot\frac{Y}{D-v\cdot t}+y_{0}\\\ \end{bmatrix}$ (23)

Hence, the projected trajectory $p(t)$ appears linear in the image plane
$(x,y)$ but non-linear in time $t$. Therefore, even in this simple scenario,
we require a trajectory that departs from the linearity assumption.

Indeed, scenes with purely linear (in space and time) pixel trajectories are
rare special cases. For example, a static camera observing a rigid dynamic
object undergoing pure translation with constant velocity parallel to the
principle plane of the camera in an otherwise static scene.

This motivates us to model pixel motion with Bézier curves that can adapt to
non-linear motion in space and time. In principle, it is also possible to
regress discrete trajectories and fit a spline through them. However, this
approach would complicate the implementation significantly, especially if we
require control points in non-uniform time intervals.

## References

  * [1] Alzugaray López, I., Chli, M.: Haste: Multi-hypothesis asynchronous speeded-up tracking of events. In: British Mach. Vis. Conf. (BMVC) (2020)
  * [2] Bailer, C., Varanasi, K., Stricker, D.: Cnn-based patch matching for optical flow with thresholded hinge embedding loss. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (July 2017)
  * [3] Baker, S., Lewis, D.S.J., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2011)
  * [4] Bardow, P., Davison, A.J., Leutenegger, S.: Simultaneous optical flow and intensity estimation from an event camera. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 884–892 (2016). https://doi.org/10.1109/CVPR.2016.102
  * [5] Benosman, R., Ieng, S.H., Clercq, C., Bartolozzi, C., Srinivasan, M.: Asynchronous frameless event-based optical flow. Neural Netw. 27, 32–37 (2012). https://doi.org/10.1016/j.neunet.2011.11.001
  * [6] Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Eur. Conf. Comput. Vis. (ECCV) (Oct 2012)
  * [7] Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Eur. Conf. Comput. Vis. (ECCV) (2012)
  * [8] Dosovitskiy, A., Fischer, P., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., Brox, T.: FlowNet: Learning optical flow with convolutional networks. In: Int. Conf. Comput. Vis. (ICCV). pp. 2758–2766 (2015). https://doi.org/10.1109/ICCV.2015.316
  * [9] Gadot, D., Wolf, L.: Patchbatch: A batch augmented loss for optical flow. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (June 2016)
  * [10] Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A., Conradt, J., Daniilidis, K., Scaramuzza, D.: Event-based vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020.3008413
  * [11] Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. Int. J. Comput. Vis. 104(3), 286–314 (2013)
  * [12] Gehrig, D., Rebecq, H., Gallego, G., Scaramuzza, D.: EKLT: Asynchronous photometric feature tracking using events and frames. Int. J. Comput. Vis. (2019). https://doi.org/10.1007/s11263-019-01209-w
  * [13] Gehrig, M., Aarents, W., Gehrig, D., Scaramuzza, D.: Dsec: A stereo event camera dataset for driving scenarios. IEEE Robot. Autom. Lett. (2021)
  * [14] Gehrig, M., Millhäusler, M., Gehrig, D., Scaramuzza, D.: E-RAFT: Dense optical flow from event cameras. In: 3D Vision (3DV) (2021)
  * [15] Godet, P., Boulch, A., Plyer, A., Besnerais, G.L.: Starflow: A spatiotemporal recurrent cell for lightweight multi-frame optical flow estimation. In: IEEE Int. Conf. Pattern Recog. (ICPR). pp. 2462–2469 (2021). https://doi.org/10.1109/ICPR48806.2021.9412269
  * [16] Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: Evolution of optical flow estimation with deep networks. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 1647–1655 (2017). https://doi.org/10.1109/cvpr.2017.179
  * [17] Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Conf. Neural Inf. Process. Syst. (NeurIPS). pp. 2017–2025 (2015)
  * [18] Jaegle, A., Borgeaud, S., Alayrac, J.B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., Hénaff, O., Botvinick, M.M., Zisserman, A., Vinyals, O., Carreira, J.: Perceiver io: A general architecture for structured inputs and outputs (2021)
  * [19] Janai, J., Guney, F., Ranjan, A., Black, M., Geiger, A.: Unsupervised learning of multi-frame optical flow with occlusions. In: Eur. Conf. Comput. Vis. (ECCV) (September 2018)
  * [20] Janai, J., Güney, F., Wulff, J., Black, M., Geiger, A.: Slow flow: Exploiting high-speed cameras for accurate and diverse optical flow reference data. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2017)
  * [21] Jiang, S., Campbell, D., Lu, Y., Li, H., Hartley, R.: Learning to estimate hidden motions with global motion aggregation. In: Int. Conf. Comput. Vis. (ICCV) (2021)
  * [22] Jonschkowski, R., Stone, A., Barron, J.T., Gordon, A., Konolige, K., Angelova, A.: What matters in unsupervised optical flow. In: Eur. Conf. Comput. Vis. (ECCV) (2020)
  * [23] Kerl, C., Stückler, J., Cremers, D.: Dense continuous-time tracking and mapping with rolling shutter RGB-D cameras. In: Int. Conf. Comput. Vis. (ICCV) (2015). https://doi.org/10.1109/ICCV.2015.261
  * [24] Lee, C., Kosta, A.K., Zhu, A.Z., Chaney, K., Daniilidis, K., Roy, K.: Spike-flownet: event-based optical flow estimation with energy-efficient hybrid neural networks. In: Eur. Conf. Comput. Vis. (ECCV) (2020)
  * [25] Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2021). https://doi.org/10.1109/cvpr46437.2021.00643
  * [26] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Int. Conf. Learn. Representations (ICLR) (2019), https://openreview.net/forum?id=Bkg6RiCqY7
  * [27] Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Int. Joint Conf. Artificial Intell. (IJCAI). pp. 674–679 (1981)
  * [28] Maurer, D., Bruhn, A.: Proflow: Learning to predict optical flow. In: British Mach. Vis. Conf. (BMVC) (06 2018)
  * [29] Mayer, N., Ilg, E., Häusser, P., Fischer, P., Cremers, D., Dosovitskiy, A., Brox, T.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2016), http://lmb.informatik.uni-freiburg.de/Publications/2016/MIFDB16, arXiv:1512.02134
  * [30] Mueggler, E., Forster, C., Baumli, N., Gallego, G., Scaramuzza, D.: Lifetime estimation of events from dynamic vision sensors. In: IEEE Int. Conf. Robot. Autom. (ICRA). pp. 4874–4881 (2015). https://doi.org/10.1109/ICRA.2015.7139876
  * [31] Mueggler, E., Gallego, G., Rebecq, H., Scaramuzza, D.: Continuous-time visual-inertial odometry for event cameras. IEEE Trans. Robot. 34(6), 1425–1440 (Dec 2018). https://doi.org/10.1109/tro.2018.2858287
  * [32] Niklaus, S., Liu, F.: Softmax splatting for video frame interpolation. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (June 2020)
  * [33] Pan, L., Liu, M., Hartley, R.: Single image optical flow estimation with an event camera. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2020)
  * [34] Paredes-Vallés, F., de Croon, G.C.: Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2021)
  * [35] Rebecq, H., Gehrig, D., Scaramuzza, D.: ESIM: an open event camera simulator. In: Conf. on Robotics Learning (CoRL) (2018)
  * [36] Ren, Z., Gallo, O., Sun, D., Yang, M.H., Sudderth, E.B., Kautz, J.: A fusion approach for multi-frame optical flow estimation. In: IEEE Winter Conf. Appl. Comput. Vis. (WACV). pp. 2077–2086 (2019). https://doi.org/10.1109/WACV.2019.00225
  * [37] Ricco, S., Tomasi, C.: Dense lagrangian motion estimation with occlusions. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 1800–1807 (2012). https://doi.org/10.1109/CVPR.2012.6247877
  * [38] Seok, H., Lim, J.: Robust feature tracking in DVS event stream using bézier mapping. In: IEEE Winter Conf. Appl. Comput. Vis. (WACV) (2020). https://doi.org/10.1109/wacv45572.2020.9093607
  * [39] Sun, D., Vlasic, D., Herrmann, C., Jampani, V., Krainin, M., Chang, H., Zabih, R., Freeman, W.T., Liu, C.: Autoflow: Learning a better training set for optical flow. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2021)
  * [40] Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2018)
  * [41] Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: Eur. Conf. Comput. Vis. (ECCV) (2020)
  * [42] Tulyakov, S., Gehrig, D., Georgoulis, S., Erbach, J., Gehrig, M., Li, Y., Scaramuzza, D.: TimeLens: Event-based video frame interpolation. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2021)
  * [43] Wulff, J., Black, M.J.: Efficient sparse-to-dense optical flow estimation using a learned basis and layers. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (June 2015)
  * [44] Xu, H., Yang, J., Cai, J., Zhang, J., Tong, X.: High-resolution optical flow from 1d attention and correlation. In: Int. Conf. Comput. Vis. (ICCV). pp. 10498–10507 (October 2021)
  * [45] Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Liu, C., Ramanan, D.: ViSER: Video-specific surface embeddings for articulated 3d shape reconstruction. In: Conf. Neural Inf. Process. Syst. (NeurIPS) (2021)
  * [46] Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d neural models from many casual videos. arXiv e-prints (2022)
  * [47] Ye, C., Mitrokhin, A., Fermüller, C., Yorke, J.A., Aloimonos, Y.: Unsupervised learning of dense optical flow, depth and egomotion with event-based sensors. In: IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS) (2020)
  * [48] Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (2014)
  * [49] Zhang, F., Woodford, O.J., Prisacariu, V.A., Torr, P.H.: Separable flow: Learning motion cost volumes for optical flow estimation. In: Int. Conf. Comput. Vis. (ICCV). pp. 10807–10817 (October 2021)
  * [50] Zhao, S., Sheng, Y., Dong, Y., Chang, E.I., Xu, Y., et al.: Maskflownet: Asymmetric feature matching with learnable occlusion mask. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2020)
  * [51] Zheng, Y., Zhang, M., Lu, F.: Optical flow in the dark. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (June 2020)
  * [52] Zhu, A.Z., Atanasov, N., Daniilidis, K.: Event-based feature tracking with probabilistic data association. In: IEEE Int. Conf. Robot. Autom. (ICRA) (2017). https://doi.org/10.1109/icra.2017.7989517
  * [53] Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In: Robotics: Science and Systems (RSS) (2018). https://doi.org/10.15607/RSS.2018.XIV.062
  * [54] Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: Unsupervised event-based learning of optical flow, depth, and egomotion. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2019)