11institutetext: Institute for Biomedical Informatics, Faculty of Medicine and
University Hospital Cologne, University of Cologne, Germany 22institutetext:
Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine and
University Hospital Cologne, University of Cologne, Germany 33institutetext:
Cologne Excellence Cluster on Cellular Stress Responses in Aging-Associated
Diseases (CECAD), University of Cologne, Germany 44institutetext: Institute of
Pathology, University Hospital Cologne, Germany

# Learning Nuclei Representations with Masked Image Modelling

Piotr Wójcik 11 2 2 Hussein Naji 11 2 2 Adrian Simon 44 Reinhard Büttner 44
Katarzyna Bożek 11 2 2 3 3

###### Abstract

Masked image modelling (MIM) is a powerful self-supervised representation
learning paradigm, whose potential has not been widely demonstrated in medical
image analysis. In this work, we show the capacity of MIM to capture rich
semantic representations of Haemotoxylin & Eosin (H&E)-stained images at the
nuclear level. Inspired by Bidirectional Encoder representation from Image
Transformers (BEiT) [1], we split the images into smaller patches and generate
corresponding discrete visual tokens. In addition to the regular grid-based
patches, typically used in visual Transformers, we introduce patches of
individual cell nuclei. We propose positional encoding of the irregular
distribution of these structures within an image. We pre-train the model in a
self-supervised manner on H&E-stained whole-slide images of diffuse large
B-cell lymphoma, where cell nuclei have been segmented. The pre-training
objective is to recover the original discrete visual tokens of the masked
image on the one hand, and to reconstruct the visual tokens of the masked
object instances on the other. Coupling these two pre-training tasks allows us
to build powerful, context-aware representations of nuclei. Our model
generalizes well and can be fine-tuned on downstream classification tasks,
achieving improved cell classification accuracy on PanNuke dataset by more
than $5\%$ compared to current instance segmentation methods.

###### Keywords:

nuclei classification masked image modelling self-supervised learning.

##  1 Introduction

In recent years, Transformer architectures [15] have demonstrated the
capability to achieve competitive results in computer vision [5]. However,
their data-hungry training paradigm hinders many potential applications,
especially in the field of digital pathology, where annotated data is limited.
Several self-supervised solutions have been proposed to tackle this problem
and proved to be very efficient in building good representations of visual
data based on Transformers (e.g. [2]). The work of H. Bao _et al._ [1] helped
to bridge the gap between training paradigms used in natural language
processing, e.g. in BERT [4], and computer vision tasks. BERT randomly masks
some word tokens and then reconstructs the missing tokens with the use of
encoded representations of visible portions of the text. The fundamental
difficulty in reproducing this pre-training schema for images is the magnitude
of potential vocabulary for even small image patches (e.g. $16\times 16$
pixels). At the same time, large portion of the low-frequency visual detail is
spurious for semantic decoding of an image. Bao et al. [1] addressed this
problem by using a semantic-aware discrete image tokenizer [12] which opened
possibilities for efficient self-supervised learning of high-level, context-
aware visual features via Transformers.

Concurrently, many approaches have been proposed to address a fundamental task
in digital pathology – segmentation and classification of nuclei. These
approaches can roughly be categorised according to their backbone into
Convolutional Neural Network (CNN)-based [8, 7] and, more recently,
Transformer-based [14]. For the task of nuclei segmentation, Generative
Adversarial Network was proposed as a method for synthesizing images with
labels [10]. Despite promising advances in detection, classification of nuclei
into various cell types is still considered as a by-product of instance
segmentation and needs to be improved. Moreover, CNN-based methods are
inherently limited by the locality of the view. This limitation poses a
problem for pathological analysis, since the shape, size and density are not
the sole indicators of a cell being cancerous. For example, in the diagnosis
of diffuse large B-cell lymphoma (DLBCL), a highly heterogeneous disease both
morphologically and clinically, neighboring cells as well as the subtle
relationships between nucleus and cytoplasm are important to distinguish
between healthy large leukocytes and lymphomatic cells. It is worth noting
that several Transformer-based solutions have been proposed for learning good
slide-level representations, with remarkable results achieved by R. Chen _et
al._ [3], albeit no research known to us directly addresses the challenge of
building context-aware representation of already segmented nuclei.

Motivated by this need, we introduce a method for learning segmented nuclei
representation which leverages the power of BEiT. Since its architecture is
instance-agnostic, i.e. divides the image into a fixed number of regular grid-
patches, we adopt the strategy presented in the work of S. Kim _et al._ [11].
Namely, we incorporate instance-level patches of varying sizes, containing
separate cell nuclei, into the sequence of patches fed to the backbone
Transformer. We demonstrate that the representations of nuclei indeed reflect
their respective cell types and require small amount of fine-tuning to achieve
improved classification results compared to current state-of-the-art
segmentation and classification methods.

##  2 Method

Figure 1: Pipeline for self-supervised pre-training. We follow closely
training schema proposed in [1], which converts an image into discrete tokens.
In addition to patches obtained by dividing an image into a regular grid we
also tokenize parts of the image inside bounding boxes $B_{i}$, where each box
encloses a single nucleus. We randomly mask a portion of an image, replacing
masked patches (gray squares) with a special token [M]. We then embed image
patches into a feature map $\mathbf{c}$. The same grid $\mathbf{c}$ is used to
extract representations of nuclei through RoI Align [9]. Both grid patches and
cell patches (represented by blue and red boxes, respectively) are fed into a
vision Transformer backbone after aggregation. The pretext task aims at
predicting discrete tokens for both input image and instance patches.

###  2.1 Image Representation

As input, we are given an image $\bm{x}$ and bounding boxes of individual
nuclei $B_{i}\in\mathcal{B}$, described by parameters
$B_{i}=[x^{i}_{\min},y^{i}_{\min},x^{i}_{\max},y^{i}_{\max}]$, i.e. by the
coordinates of the upper-left and lower-right corners and $i\in 1,\ldots,N$
where $N$ is the total number of bounding boxes contained within an image
$\bm{x}$. We apply the standard protocol for vision Transformers [5] which
involves splitting a 2D image $\bm{x}\in\mathbb{R}^{H\times W\times C}$ into a
sequence of grid patches $\bm{x}^{p}\in\mathbb{R}^{M\times(P^{2}C)}$, where
$C$ stands for the number of channels, $(H,W)$ represents the input size,
$(P,P)$ is the size of a single patch and $L=HW/P^{2}$. In this work we run
all experiments with $3$-channel images of size $448\times 448$, divided into
$28\times 28$ square patches.

##### Grid Patch Embedding and Masking

After obtaining grid patches, we use linear projection $\bm{E}\bm{x}^{p}$ (as
depicted in Fig. 1) to embed grid patches into $D$-dimensional feature map
$\bm{c}$, where $\bm{E}\in\mathbb{R}^{(P^{2}C)\times D}$. Next, we randomly
mask approximately $40\%$ of patches, employing blockwise masking described in
detail in [1]. Indices of patches chosen for masking are denoted by
$\mathcal{M}$. Then, we replace grid patch embeddings with a shared learnable
token $\bm{e}_{[\mathcal{M}]}$ of a size $1\times 1\times D$:
$\bm{e}^{p}_{i}$:
$\bm{x}_{i}^{\mathcal{M}}=\delta(i\in\mathcal{M})\odot\bm{e}_{[\mathcal{M}]}+(1-\delta(i\in\mathcal{M}))\odot\bm{x}_{i}^{p}$,
where $\delta(\cdot)$ is the indicator function. The choice of the masking
algorithm and the ratio was motivated by ablation studies in [1].
Specifically, the use of the blockwise masking algorithm was shown to improve
the model’s accuracy compared to selecting patches for masking independently
at random.

##### Cell Patch Embedding

So far, we closely followed steps described in [1]. To encode not only the
overall image structure but also the key semantic elements withing it - cells,
we introduce the following modification to the baseline architecture inspired
by the work [11]. Namely, given bounding boxes $B_{i}$ and the feature map
$\bm{c}$ (see Fig. 1), we extract cell instance features through RoI Align
module [9]:

$\bm{c}^{\tiny{\textrm{ins}}}_{i}=\textrm{RoIAlign}(\bm{c};B_{i})\in\mathbb{R}^{k\times
k\times D}$ (1)

where $k\times k$ is the size of extracted features. In our work we set $k=3$.
Note that some cell instance features may be sampled from masked patches if
their bounding boxes intersect with a masked portion of the image.
Subsequently, we process the $\bm{c}^{\tiny{\textrm{ins}}}_{i}$ to obtain
embedding vectors of dimension $1\times 1\times D$. We achieve this by
applying a convolution on $\bm{c}^{\tiny{\textrm{ins}}}_{i}$, as proposed in
[11]:

$\bm{p}^{\tiny{\textrm{ins}}}_{i}=\textrm{Conv}(\bm{c}^{\tiny{\textrm{ins}}}_{i})\in\mathbb{R}^{1\times
1\times D}$ (2)

##### Patch Aggregation

In our architecture, we concatenate a sequence of grid patch embeddings
$\\{\bm{c}^{\mathcal{M}}_{i}\\}_{i=1}^{L}$ (blue squares in Fig. 1) and a
sequence of $N$ cell patch embeddings
$\\{\bm{p}^{\tiny{\textrm{ins}}}_{i}\\}_{i=1}^{N}$. Following [1], we prepend
a special, learnable token [CLS] to the concatenated sequence. Since the
number of nuclei in a particular image may vary, we pad cell embedding
sequences to a maximum length using [PAD] token. Overall, in this module we
use three special tokens ([M], [PAD], [CLS]), each of them is shared,
learnable and has size $1\times 1\times D$. After aggregation, the input
sequence to the positional encoding module has the following form:

$\bm{H}^{0}=[\texttt{[CLS]},\bm{c}^{\mathcal{M}}_{1},\ldots,\bm{c}^{\mathcal{M}}_{L},\underbrace{\bm{p}^{\tiny{\textrm{ins}}}_{1},\ldots,\bm{p}^{\tiny{\textrm{ins}}}_{N},\texttt{[PAD]},\ldots,\texttt{[PAD]}}_{\text{fixed
maximum length}}]$ (3)

###  2.2 Positional Encoding

To encode the information about the positions of patches (both grid and cell),
we use the technique proposed in [11]. The position and the shape of any
patch, regardless of its regular grid or a bounding box identity, can be
represented by center coordinates $(x,y)$ and patch width $w$ and height $h$,
as exemplified in Fig. 2. We use a sinusoidal mapping
$\gamma\colon\mathbb{R}\to\mathbb{R}^{D/4}$ defined as
$\gamma(t):=[\sin(2^{0}\pi t),\cos(2^{0}\pi t),\ldots,\sin(2^{D/8-1}\pi
t),\cos(2^{D/8-1}\pi t)]$. For every patch token, positional encoding is
constructed by concatenating all spatial information embedded by $\gamma$,
namely
$\bm{\hat{E}}_{i}=[\gamma(x_{i}),\gamma(y_{i}),\gamma(w_{i}),\gamma(h_{i})]$.
Positional encoding is further added to the corresponding token embedding:
$\bm{H}_{i}^{0}+\bm{\hat{E}}_{i}$. We handle [CLS] token separately, with a
learnable positional encoding of shape $1\times 1\times D$.

Figure 2: Building positional encoding for both grid and cell patches. A tuple
$(x_{g},y_{g},h_{g},w_{g})$ describes center coordinates, width and height of
a grid patch (blue), while $(x_{n},y_{n},h_{n},w_{n})$ describes the position
and shape of a cell patch (red).

###  2.3 Discrete Tokenizer

The role of discrete tokenizer is to map an image to discrete tokens, _i.e._
natural numbers. Formally, an image $\bm{x}$ is transformed into a grid
$\bm{z}=[z_{i},\ldots,z_{T}]\in\mathcal{V}^{(H/R)\times(W/R)}$, where
$\mathcal{V}$ denotes the discrete vocabulary, $R$ is the resolution of the
discrete tokenizer. In our work, we use pre-trained weights of a publicly
available dVAE (https://github.com/openai/DALL-E) with $|\mathcal{V}|=8192$.
For the details of dVAE implementation and pre-training, please refer to [13].
We tokenize the input image to into $28\times 28$ grid. We then crop each
nucleus instance to its bounding box and resize obtained crops to a size
$32\times 32$. Nuclei views are subsequently tokenized into $4\times 4$ grids.

###  2.4 Backbone Transformer Encoder

Our encoder shares all architectural details with ViT [5], except for
additional attention masking for preventing attending to [PAD] tokens, which
we implement according to BERT [4]. Embedded patches with positional encoding
$(\bm{H}^{0}+\bm{\hat{E}})$ are fed into $12$ layers of Transformer blocks
with $12$ attention heads.

$\bm{H}^{12}=[\bm{h}_{\texttt{[CLS]}},\underbrace{\bm{h}^{g}_{1},\ldots,\bm{h}^{g}_{28\times
28}}_{\text{\makebox[0.0pt]{\footnotesize grid
representations}}},\overbrace{\bm{h}^{n}_{1},\ldots,\bm{h}^{n}_{N}}^{\text{\makebox[0.0pt]{\footnotesize
nuclei
representations}}},\bm{h}_{\texttt{[PAD]}},\ldots,\bm{h}_{\texttt{[PAD]}}].$
(4)

After discarding the representation of [CLS] and [PAD] tokens,
$\bm{h}_{\texttt{[CLS]}}$ and $\bm{h}_{\texttt{[PAD]}}$ respectively, we are
left with $28\times 28$ representations of regular grid patches
$\bm{h}^{g}_{i}$ and $N$ representations of nuclei $\bm{h}^{n}_{j}$.

####  2.4.1 Pre-training Loss Function

We use two fully-connected layers as output heads (see Fig. 1). The first one,
image head, predicts discrete tokens from the representation of every masked
grid patch $\\{\bm{h}^{g}_{i}:i\in\mathcal{M}\\}$. Since the input image was
tokenized into $28\times 28$ natural numbers, there is one-to-one
correspondence between grid patches and tokens. The second, cell instance
head, predicts $4$ tokens from $\bm{h}^{n}_{j}$, for each cell instance whose
bounding box $B_{j}$ intersects with masked portion of the image. The training
MIM loss is defined as:

$\mathcal{L}_{\text{MIM}}=-\sum_{(\bm{x},\mathcal{B})\in\mathcal{D}}\left(\overbrace{\sum_{i\in\mathcal{M}}\log(p(z^{g}_{i}|\bm{c_{i}}^{\mathcal{M}}))}^{\text{\makebox[0.0pt]{\footnotesize
BEiT
loss}}}+\overbrace{\sum_{j\in\mathcal{B}_{\mathcal{M}}}\sum_{k=1}^{4}\log(p(z^{n}_{j,k}|\bm{p_{j}}^{\text{ins}})}^{\text{\makebox[0.0pt]{\footnotesize$\mathcal{L}_{\text{inst}}$}}})\right)$
(5)

where $\mathcal{D}$ denotes pre-training set of images with bounding boxes,
$\mathcal{B}_{\mathcal{M}}$ indicates the set $\mathcal{B}$ of bounding boxes
that intersect with masked grid patches, $z^{g}_{i}$ the discrete token for
the $i$-th grid patch and $z^{n}_{j,1},\ldots,z^{n}_{j,4}$ discrete tokens for
$j$-th nuclei instance. Notice, that the first component of the loss function
comes directly from [1] while the second, dubbed $\mathcal{L}_{\text{inst}}$,
is proposed by us in order to sharpen the view on nuclei.

##  3 Experiments

Figure 3: Datasets augmentation strategy. Left: Self-supervised pre-training
dataset consists of one H&E-stained WSI of a lymph node, divided into tiles of
size $512\times 512$. For each four adjacent tiles, we randomly generate
overlapping crops. Right: To evaluate our model on the CoNSeP dataset, we
divide $1000\times 1000$ pixel-large images into four overlapping tiles of
$448\times 448$. Each nucleus is classified basing on its representation from
one tile only.

##### Datasets

We used three datasets in our work. For self-supervised pre-training, we used
our in-house dataset of $\numprint{37665}$ H&E images ($40$x magnification)
obtained from a single WSI of a DLBCL lymph node. Each $512\times 512$ tile
was segmented with a bounding box for each nucleus, albeit no cell labels were
provided. The segmentation was performed automatically, without further
verification. We resized every tile to $448\times 448$ and randomly cropped
additional tiles to increase the number of training examples, as demonstrated
in Fig. 3. After preprocessing, DLBCL dataset consists of $\numprint{160545}$
image tiles with segmented nuclei. For fine-tuning and testing we use CoNSeP
and PanNuke datasets [8, 6]. The CoNSeP dataset consists of $\numprint{24319}$
annotated nuclei from $41$ H&E images. Nuclei are grouped into four types. The
PanNuke dataset [6] contains $\numprint{205343}$ annotated nuclei of five
types. Images in both datasets have size of $256\times 256$ and originate from
$19$ different tissue types. We did not perform any pre-processing on these
images except for resizing them to $448\times 448$.

##### Pre-Training Setup

The proposed network was pre-trained with similar set of parameters as BEiT
[1]. We used $12$-layer Transformer with $768$ hidden size and $12$ attention
heads. We pre-trained the model for $800$ epochs with a batch size $96$. The
codebase used for experiments was that of the original BEiT implementation
https://github.com/microsoft/unilm/tree/master/beit. Instance embedding code
was built upon InstaFormer implementation https://github.com/KU-
CVLAB/InstaFormer. Around $550$-th epoch, we added PanNuke Fold $1$ dataset
and CoNSeP training dataset to the initial DLBCL dataset to boost performance.
Throughout the pre-training phase, we applied the same set of augmentations as
in [1], which included RandomResizeAndCrop, color jittering with a parameter
of 0.4 and RandomFlip. Additionally, all input images were normalized to the
mean and standard deviation of ImageNet.

##### Fine-tuning on Annotated Datasets

We fine-tuned our model and validated its performance on the two labelled
datasets, CoNSeP and PanNuke. In nuclei classification task, we discarded two
MIM pre-training linear layers and used softmax classifier on nuclei
representations: $\text{softmax}(\\{\bm{h}_{i}^{n}\\}^{N}_{i=1}\bm{W}_{C})$,
where $\bm{h}_{i}^{n}$ stands for $i$-th nuclei representation, $C$ is the
number of nuclei classes and $\bm{W}_{C}$ denotes a linear layer parameter
matrix.

##### Evaluation Protocol

As a baseline model, we chose the widely used state-of-art nuclei detection
and classification model HoVer-Net [8]. Since the authors only provided
metrics relevant to the combined tasks of nuclei detection and classification,
for a fair comparison, we modified the codebase to print both accuracy and F1
score for the subset of nuclei that were correctly segmented. Results
presented in Tab. 1 were obtained with a use of published pre-trained weights
https://github.com/vqdang/hover_net for the CoNSeP and PanNuke datasets.
Subsequently, we ran evaluation of our model on the exact shapes of correctly
detected nuclei.

Figure 4: Left: t-SNE visualisation of DLBCL nuclei representations without
fine-tuning on any labeled dataset. Right: Upper row shows self-attention map
for [CLS] token; lower row shows self-attention map for a reference point
indicated with an arrow.

##### Ablation Studies on $\mathcal{L}_{\text{inst}}$

We performed ablation studies to test the importance of
$\mathcal{L}_{\text{inst}}$. We pre-trained two variants of the model (with
and w/o $\mathcal{L}_{\text{inst}}$) for $250$ epochs on the DLBCL dataset and
compared linear probing results on Fold 3 of PanNuke dataset. We kept the
models frozen and added a linear head to evaluate the self-supervised models
[1]. The model pre-trained using the full $\mathcal{L}_{\text{MIM}}$ loss
function achieved a significantly higher accuracy of $0.613$ compared to the
model pre-trained with only the BEiT component of the loss function, which
achieved an accuracy of $0.517$.

##  4 Results and Conclusion

As demonstrated in Tab. 1, our model visibly outperforms HoVerNet in the task
of nuclei classification. The difference is especially pronounced on the
PanNuke dataset ($0.05$ on average) for all dataset which is much larger than
CoNSeP. We find striking the capacity of our model to generalize to images
from a pathological domain completely different from the DLBCL corpus used for
pre-training. Fig. 4 shows the ability of self-supervised training to separate
cells of different types while self-attention map demonstrates that our model
learns long-distance relations between nuclei. Although our proposed training
method requires a non-negligible amount of segmented images, these can be
obtained automatically at a low cost. It should be also noted that MIM methods
achieve competitive results on image segmentation [1], offering new
possibilities for performing both tasks of nuclei detection and segmentation
simultaneously in the future.

Table 1: Nuclei classification results on PanNuke dataset (Fold 1 - Fold 3
split) and CoNSeP. HoVerNet (HN) is used as a baseline model. PanNuke
---
| All | Neo. | Inflam. | Conn. | Dead | Epith.
HN F1 |  $0.780$ |  $0.856$ |  $0.714$ |  $0.719$ |  $0.399$ |  $0.765$
Ours F1 |  $\mathbf{0.831}$ |  $\mathbf{0.893}$ |  $\mathbf{0.776}$ |  $\mathbf{0.764}$ |  $\mathbf{0.571}$ |  $\mathbf{0.848}$
HN Acc |  $0.780$ |  $0.748$ |  $0.555$ |  $0.562$ |  $0.249$ |  $0.619$
Ours Acc |  $\mathbf{0.832}$ |  $\mathbf{0.807}$ |  $\mathbf{0.634}$ |  $\mathbf{0.618}$ |  $\mathbf{0.400}$ |  $\mathbf{0.735}$
CoNSeP
| All | Misc. | Inflam. | Epith. | Spindle |
HN F1 |  $0.872$ |  $0.734$ |  $0.817$ |  $0.939$ |  $0.856$ |
Ours F1 |  $\mathbf{0.885}$ |  $\mathbf{0.804}$ |  $\mathbf{0.822}$ |  $\mathbf{0.954}$ |  $\mathbf{0.864}$ |
HN Acc |  $0.872$ |  $0.580$ |  $0.690$ |  $0.884$ |  $0.749$ |
Ours Acc |  $\mathbf{0.885}$ |  $\mathbf{0.672}$ |  $\mathbf{0.698}$ |  $\mathbf{0.911}$ |  $\mathbf{0.761}$ |

## References

  * [1] Bao, H., Dong, L., Piao, S., Wei, F.: Beit: BERT pre-training of image transformers. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022)
  * [2] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 9630–9640. IEEE (2021)
  * [3] Chen, R.J., Chen, C., Li, Y., Chen, T.Y., Trister, A.D., Krishnan, R.G., Mahmood, F.: Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 16123–16134. IEEE (2022)
  * [4] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics (2019)
  * [5] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net (2021)
  * [6] Gamper, J., Koohbanani, N.A., Graham, S., Jahanifar, M., Khurram, S.A., Azam, A., Hewitt, K., Rajpoot, N.: Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778 (2020)
  * [7] Graham, S., Epstein, D.B.A., Rajpoot, N.M.: Dense steerable filter cnns for exploiting rotational symmetry in histology images. IEEE Trans. Medical Imaging 39(12), 4124–4136 (2020)
  * [8] Graham, S., Vu, Q.D., e Ahmed Raza, S., Azam, A., Tsang, Y., Kwak, J.T., Rajpoot, N.M.: Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Anal. 58 (2019)
  * [9] He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell. 42(2), 386–397 (2020)
  * [10] Hou, L., Agarwal, A., Samaras, D., Kurç, T.M., Gupta, R.R., Saltz, J.H.: Robust histopathology image analysis: To label or to synthesize? In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 8533–8542. Computer Vision Foundation / IEEE (2019)
  * [11] Kim, S., Baek, J., Park, J., Kim, G., Kim, S.: Instaformer: Instance-aware image-to-image translation with transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 18300–18310. IEEE (2022)
  * [12] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8821–8831. PMLR (2021)
  * [13] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8821–8831. PMLR (2021)
  * [14] Tang, Y., Zhang, N., Wang, Y., He, S., Han, M., Xiao, J., Lin, R.: Accurate and robust lesion RECIST diameter prediction and segmentation with transformers. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2022 - 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 13434, pp. 535–544. Springer (2022)
  * [15] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pp. 5998–6008 (2017)


denotes equal contribution.

Event cameras capture the world at high time resolution and with minimal bandwidth requirements.
However, event streams, which only encode changes in brightness, do not contain sufficient scene information to support a wide variety of downstream tasks.
In this work, we design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner.
We generalize event cameras in terms of when an event is generated and what information is transmitted.
To implement our designs, we turn to single-photon sensors that provide digital access to individual photon detections; this modality gives us the flexibility to realize a rich space of generalized event cameras.
Our single-photon event cameras are capable of high-speed, high-fidelity imaging at low readout rates.
Consequently, these event cameras can support plug-and-play downstream inference, without capturing new event datasets or designing specialized event-vision models.
As a practical implication, our designs, which involve lightweight and near-sensor-compatible computations, provide a way to use single-photon sensors without exorbitant bandwidth costs.

§ INTRODUCTION

Event cameras <cit.> sense the world at high speeds, providing visual information with minimal bandwidth and power.
They achieve this by transmitting only changes in scene brightness, when significant “events” occur. However, there is a cost. Raw event data, a sparse stream of binary values, does not hold sufficient information to be used directly with mainstream vision algorithms.
Therefore, while event cameras have been successful at certain tasks (, object tracking <cit.>, obstacle avoidance <cit.>, and high-speed odometry <cit.>), they are not widely deployed as general-purpose vision sensors, and often need to be supplemented with conventional cameras <cit.>. These limitations are holding back this otherwise powerful technology.

Is it possible to realize the promise of event cameras, , high temporal resolution at low bandwidth, while preserving rich scene intensity information? To realize these seemingly conflicting goals, we propose a novel family of generalized event cameras.
We conceptualize a space of event cameras along two key axes (<ref> (top)): (a) “when to transmit information,” formalized as a change detection procedure $\Delta$; and (b) “what information to transmit,” characterized by an integrator $\Sigma$ that encodes incident flux. Existing event cameras represent one operating point in this $(\Sigma, \Delta)$ space. Our key observation is that by exploring this space and considering new $(\Sigma,\Delta)$ combinations, we can design event cameras that preserve scene intensity.
We propose more general integrators, , that represent flux according to motion levels, that span spatial patches, or that employ temporal coding (<ref> (middle)). We also introduce robust change detectors that better distinguish motion from noise, by considering increased spatiotemporal contexts and modeling noise in the sensor measurements.

Despite their conceptual appeal, physically implementing generalized event cameras is a challenge. This is because the requisite computations must be performed at the sensor to achieve the desired bandwidth reductions. For example, existing event cameras perform simple integration and thresholding operations via analog in-pixel circuitry. However, more general $(\Sigma, \Delta)$ combinations are not always amenable to analog implementations; even feasible designs might require years of hardware iteration and production scaling. To build physical realizations of generalized event cameras, we leverage an emerging sensor technology: single-photon avalanche diodes (SPADs) that provide digital access to photon detections at extremely high frame rates ($\tildeNice 100$ kHz). This allows us to compose arbitrary software-level signal transformations, such as those required by generalized event cameras. Further, we are not locked to a particular event camera design and can realize multiple configurations with the same sensor.

Implications: extreme, bandwidth-efficient vision. Generalized event cameras support high-speed, high-quality image reconstruction, but at low bandwidths quintessential of current event cameras. For example, <ref> (middle, bottom) shows reconstructions at $3025$ FPS that have an effective readout of a $30$ FPS frame-based camera. Further, our methods have strong low-light performance due to the SPAD's single-photon sensitivity. As we show in <ref> (bottom), preserving scene intensity facilitates plug-and-play inference in challenging scenarios, with state-of-the-art vision algorithms. Critically, this does not require retraining vision models or curating dedicated datasets, which is a significant challenge for unconventional imagers.
This plug-and-play capability is vital to realizing universal event vision that retains the benefits of current event cameras.

Scope. We consider full-stack event perception: we conceptualize a novel space of event cameras, provide relevant single-photon algorithms, analyze their imaging capabilities and rate-distortion trade-offs, and show on-chip feasibility. We demonstrate imaging capabilities in <ref> using the SwissSPAD2 array <cit.>, and show viable implementations of our algorithms for UltraPhase <cit.>, a recent single-photon compute platform. All of these are critical to unlocking the promise of event cameras. However, our objective is not to develop an integrated system that incorporates all these components; this paper merely takes the first steps toward that goal.

§ RELATED WORK

Event camera designs. Perhaps most widespread is the DVS event camera <cit.>,
where each pixel generates an event in response to measured changes in (log) intensity.
The DAVIS event camera <cit.> couples DVS pixels with conventional CMOS pixels, providing access to image frames. However, the frames lack the dynamic range of DVS events.
A recent design, Celex-V <cit.>, provides log-intensity frames using an external trigger. ATIS <cit.>, a less prevalent design, features asynchronous intensity events, but its sophisticated circuitry reduces pixel fill factor.
The above designs are based on analog processing; we instead design event cameras on digital photon detections.

Intensity imaging with event cameras. Several approaches have been explored to obtain images from events, including Poisson solvers <cit.>, manifold regularization <cit.>, assuming knowledge of camera motion <cit.> or optical flow <cit.>, and learning-based methods <cit.>. However, because events often lack sufficient scene information,
they are often supplemented by conventional frames <cit.>, either from sensors such as DAVIS or using a multi-camera setup. Fusing events and frames presents challenges due to potential spatiotemporal misalignment and discrepancies in imaging modalities.
Even when these challenges are overcome, we show that fusion methods produce lower fidelity than our proposed generalized event cameras.

Passive single-photon imaging. In the past few years, SPADs have found compelling passive imaging applications; this includes high-dynamic range imaging <cit.>, motion deblurring <cit.>, high-speed tracking <cit.>, and ultra wide-band videography <cit.>. A particularly relevant method is proposed by <cit.>, which uses flux changepoint estimation to perform burst photography on single-photon sequences. This approach uses flux changepoints to estimate motion, then integrates along spatiotemporal motion trajectories to circumvent the noise-blur tradeoff. This spatiotemporal integration allows for high-quality reconstructions under challenging lighting and motion conditions. In contrast, our paper emphasizes changepoint estimation as a means to compress single-photon data. Further, since we aim to run our proposed techniques near sensor, where there are limited memory and compute capabilities, we focus on online changepoint estimation that processes photon detections in a single pass.

The fine granularity of passive single-photon acquisition makes it possible to emulate a diverse set of imaging modalities <cit.>, including event cameras, via post-capture processing. In this work, we go beyond emulating existing event cameras and design alternate event cameras that preserve high-fidelity intensity information.

§ WHAT IS AN EVENT CAMERA?

The defining characteristic of event cameras is that they transmit information selectively, in response to changes in scene content. This selectivity allows event cameras to encode scene information at high time resolutions required to capture scene dynamics, without proportionately high bandwidth. This is in contrast to frame-based cameras, where readout occurs at fixed intervals.

We characterize event cameras in terms of two axes: what the camera transmits and when it transmits.
As a concrete example, consider existing event cameras. They trigger events (“when to transmit”) based on a fixed threshold:

\begin{equation}
\label{eq:event-generation}
|\Phi(\vx, t) - \Phi_\text{ref}(\vx)| \geq \tau,
\end{equation}

where $\Phi(\vx, t)$ is a flux estimate[Event cameras such as the DVS <cit.> measure a temporally low-pass filtered estimate of log-flux; we absorb this into $\Phi(\vx, t)$ for brevity.] at pixel $\vx$ and time $t$, and $\tau$ is the threshold. $\Phi_\text{ref}(\vx)$ is a previously-recorded reference, set to $\Phi(\vx, t)$ whenever an event is triggered. Each event consists of a packet

\begin{equation}
\label{eq:binary-event}
\paren*{\vx, t, \text{sign}\paren*{\Phi(\vx, t) - \Phi_\text{ref}(\vx)}}
\end{equation}

that encodes the polarity of the change (“what to transmit”).

Event polarities, although adequate for some applications, do not retain sufficient information to support a general set of computer vision tasks. A stream of event polarities is an extremely lossy representation. Notably, it only defines scene intensity up to an initial unknown reference value, and it does not encode any information in regions not producing events, , regions with little or no motion.

Our key observation is that existing event cameras represent just one operating point in a broader space of generalized event cameras, which is defined by two axes: “what to transmit” and “when to transmit.” By considering alternate points in this space, we can design event cameras that preserve high-fidelity scene intensity. This enables plug-and-play inference with a host of algorithms developed by the mainstream vision community.

We begin with a conceptual exploration of this generalized space, before describing its physical implementation.

Generalizing “what to transmit.” As a first step, we can modify the event camera such that it transmits $n$-bit values instead of one-bit change polarities. When an event is triggered, we send the current value of $\Phi(\vx, t)$; if a pixel triggers no events, we transmit $\Phi$ during the final readout. As we show in <ref> (b), this simple change allows us to recover scene intensity, even in static regions.
It is important to note that, while the transmitted quantity differs from conventional events, we retain the defining feature of an event camera: selective transmission based on scene dynamics (where we transmit according to <ref>). Thus, the readout remains decoupled from the time resolution.

If $\Phi(\vx, t)$ were a perfect estimate of scene intensity, then the changes thus far would suffice. However, $\Phi$ is fundamentally noisy: to capture high-speed changes, $\Phi$ must encompass a shorter duration, which leads to higher noise. This is a manifestation of the classical noise-blur tradeoff.

To address this problem, we introduce the abstraction of an integrator (or $\Sigma$), that defines how we accumulate incident flux and, in turn, what we transmit.
Ideally, we want the integrator to adapt to scene dynamics, , accumulate over longer durations when there is less motion, and vice versa.
We observe that event generation, which is based on scene dynamics, can be used to formulate an adaptive integrator. Specifically, we propose an integrator $\Sigma_\text{cuml}$ that computes the cumulative flux since the last event:

\begin{equation}
\label{eq:cuml-integrator}
\Sigma_\text{cuml}(\vx, t) = \int_{T_0}^{t} \Phi(\vx, s) ds,
\end{equation}

where $T_0$ is the time of the last event. When an event is triggered at time $T_1$, we communicate the value[We can either transmit values of $\Sigma_\text{cuml}$ or differences (changes) to $\Sigma_\text{cuml}$; we treat this as an implementation detail here.] of $\Sigma_\text{cuml}(\vx, T_1)$, which we interpret as the intensity throughout $[T_0, T_1]$. This approach yields a piece-wise constant time series, with segments delimited by events. We note that a similar idea, of virtual exposures beginning and ending with change points, was also explored in <cit.> as part of a motion-adaptive deblurring pipeline. Adaptive exposures significantly reduce noise while preserving dynamic scene content, as we show in <ref> (c).

Generalizing “when to transmit.” The success of the adaptive integrator crucially depends on the reliability of events; for example, triggering false events in static regions causes unnecessary noise. We refer to the event-generation procedure as the change detector, denoted by $\Delta$. Current event cameras detect changes by applying a fixed threshold to measured intensity differences (<ref>).
This method has two key limitations: it only considers the value of $\Phi$ at pixel location $\vx$ and time $t$ and is not attuned to the stochasticity in $\Phi$.

We design more robust change detectors that (1) leverage enhanced spatiotemporal contexts, and (2) incorporate noise awareness, either explicitly by tuning thresholds, or implicitly by modulating the detector's behavior. Specifically, we improve reliability by using temporal forecasters (<ref>), by leveraging correlated changes in patches (<ref>), or by exploiting integrator statistics  (<ref>).

Realizing generalized event cameras. The critical detail remaining is how we implement our proposed designs in practice. We need direct access to flux estimates at a high time resolution.
Conventional high-speed cameras can provide such access,
however, they incur substantial per-frame read noise ($\tildeNice 20$–$40$$e^-$ <cit.>) that grows with frame rate <cit.>.

We turn to an emerging class of single-photon sensors, single-photon avalanche diodes (SPADs <cit.>), that has witnessed dramatic improvements in device practicality and key sensor characteristics (, array sizes and fill factors) in recent years <cit.>.
SPADs can operate at extremely high speeds ($\tildeNice 100$ kHz) without incurring per-frame read noise.
Each $\Phi(\vx, t)$ measured by a SPAD is limited only by the fundamental stochasticity of photon arrivals (shot noise). This allows SPADs to provide high timing resolution without a substantial noise penalty. In the next section, we describe the image formation model of SPADs and provide single-photon implementations of our designs.

§ SINGLE-PHOTON GENERALIZED EVENT CAMERAS

A SPAD array can operate as a high-speed photon detector, producing binary frames as output. Each binary value indicates whether at least one photon was detected during an exposure. The SPAD output response, $\Phi(\vx, t)$, can be modeled as a Bernoulli random variable, with

\begin{equation}
\label{eq:bernoulli}
P \left( \Phi(\vx, t) = 1 \right) = 1 - e^{-N(\vx, t)},
\end{equation}

where $N(\vx, t)$ is the average number of photo-electrons during an exposure, including any spurious detections. The inherently digital SPAD response allows us to compute software-level transformations on the signal $\Phi(\vx, t)$, including operations that may be challenging to realize via analog processing. These transformations can be readily reconfigured, which permits a spectrum of event camera designs, not just one particular choice. However, there is one consideration: our designs should be lightweight and computable on chip. As we show in <ref>, this is vital to implementing generalized event cameras without the practical costs associated with reading off raw SPAD outputs.

We now describe a set of SPAD-based event cameras (summarized in <ref>), beginning with the adaptive exposure method from the previous section.

Adaptive-exposure event camera.
We obtain a SPAD implementation of the adaptive exposure described in <ref> by replacing the integral with a sum over photons:

\begin{equation}
\label{eq:cuml-sum}
\Sigma_\text{cuml}(\vx, t) = \sum_{s = T_0}^{t} \Phi(\vx, s).
\end{equation}

To generate events, we can use a threshold-based change detector (<ref>). Differences between individual binary values are not sufficiently informative; therefore, we apply <ref> to an exponential moving average (EMA) computed on $\Phi$. We call this an “adaptive-EMA” event camera.

§ EXPERIMENTAL RESULTS

We demonstrate the capabilities of generalized event cameras using a SwissSPAD2 array <cit.> with resolution $512 \times 256$, which we use to capture one-bit frames at $96.8$ kHz.
We show the feasibility of our designs on UltraPhase <cit.>, a recent single-photon computational platform (<ref>).

Refinement model. For each of our event cameras, we train a refinement model that mitigates artifacts arising from the asynchronous nature of events. This model takes a periodic frame-based sampling of integrator values and outputs a video reconstruction. The sampling rate is configurable; in practice, we set it $\tildeNice 16$–$64\times$ lower than the SPAD rate. We use a densely-connected residual architecture <cit.>, trained on data generated by simulating photon detections on temporally interpolated <cit.> high-speed videos from the XVFI dataset <cit.>. See the supplement for training details.

§.§ Extreme Bandwidth-Efficient Videography

High-speed videography. In <ref>, we capture the dynamics of a deformable ball (a “stress ball”) using a SPAD, a high-speed camera (Photron Infinicam) operated at $500$ FPS, and a commercial event camera (Prophesee EVK4). The high-speed camera suffers from low SNR due to read noise, which manifests as prominent artifacts after on-camera compression. Meanwhile, conventional events captured by Prophesee, when processed by “intensity-from-events” methods such as E2VID+ <cit.> fail to recover intensities reliably, especially in static regions. We also evaluate EDI <cit.>, a hybrid event-frame method. We consider an idealized variant that operates on SPAD events (obtained via EMA thresholding), which gives perfect event-frame alignment and a precisely known event-generation model.
We refine the outputs of EDI using the same model as for our methods. We refer to this idealized, refined version of EDI as “EDI++.” While EDI++ recovers more detail than other baselines, there are considerable artifacts in its outputs.

Our method achieves high-quality reconstructions at $3025$ FPS ($96800 / 32$) that faithfully capture non-rigid deformations, with only $431$ bits per second per pixel (bps/pixel) readout, which is a $227\times$ compression ($96800 / 431$) of the raw SPAD capture. Viewed differently, for a $1$ MPixel array, we would obtain a bitrate of $431$ Mbps, implying that we can read off these $3025$ FPS reconstructions over USB 2.0 (which supports transfer speeds of up to $480$ Mbps).

Event imaging in low light. <ref> compares the low-light performance of frame-based, event-based, and a generalized event camera on an urban night-time scene at $7$ lux (lux measured at the sensor). For frame-based cameras, a short exposure that preserves motion may be too noisy, while a long exposure can be severely blurred. The Prophesee's performance deteriorates in low light, resulting in blurred temporal gradients. EDI++, benefiting from the idealized SPAD-based implementation, can image this scene, but finer details like the motorcyclist are lost. Our generalized event cameras, on the other hand, provide reconstructions with minimal noise, blur, or artifacts—while retaining the bandwidth efficiency of event-based systems. The compression here is $307\times$ with respect to raw SPAD outputs.

§.§ Plug-and-Play Inference

Generalized event cameras preserve scene intensity, which enables plug-and-play event-based vision. We consider a tennis sequence (of $8196$ binary frames) containing a range of object speeds.
We evaluate a range of tasks: pose estimation (HRNet <cit.>), corner detection <cit.>, optical flow (RAFT <cit.>), object detection (DETR <cit.>), and segmentation (SAM <cit.>). We compare against event-based methods applied to Prophesee events; we use Arc* <cit.> for corner detection and E-RAFT <cit.> for optical flow.
For the remaining tasks, which do not have equivalent event methods, we run HRNet, DETR, and SAM on E2VID+ reconstructions.

As <ref> (top left) shows, traditional events are bandwidth efficient ($331$ bps/pixel),
but do not provide sufficient information for successful inference.
Generalized events (bottom left) have a modestly higher readout ($520$ bps/pixel), but support accurate inference without requiring dedicated algorithms.
To provide context for these rates, we compare them against frame-based methods (right). A long exposure ($120$ bps/pixel) blurs out the racket. Burst methods <cit.> recover a sharp image from a stack of short exposures, but with a large readout of $15100$ bps/pixel.

§.§ Rate-Distortion Analysis

Each method in <ref> features a sensitivity parameter that controls the sensor readout rate (event rate), which in turn influences image quality.
In this subsection, we evaluate the impact of readout on image quality (PSNR) by performing a rate-distortion analysis.
For ground truth, we use a set of YouTube-sourced high-speed videos captured by a Phantom Flex4k at $1000$ FPS; see the supplement for thumbnails and links.
We upsample these videos to the SPAD's frame rate and then simulate $4096$ binary frames using the image formation model described in <ref>.
When computing readout for our methods, we assume that events encode $10$-bit values and account for the header bits of each event packet.

As baselines, we consider EDI++, a long exposure, compressive sensing with 8-bucket masks, and burst denoising <cit.> using $32$ short exposures.
As <ref> shows, generalized event cameras provide a pronounced $4$–$8$ dB PSNR improvement over baseline methods.
Further, our methods can compress the raw SPAD response by around $80\times$ before a noticeable drop-off in PSNR is observed.

Among our methods, the spatiotemporal chunk approach of <ref> gives the best PSNR, followed by the Bayesian method (<ref>) and coded-exposure events (<ref>). That said, all methods are fairly similar in terms of rate-distortion (, all three give comparable results for the scenes in <ref>).
The methods are better distinguished by their practical characteristics.
The Bayesian method gives single-photon temporal resolution; however, as we show in <ref>, it is the most expensive to compute on-chip. The chunk-based method occupies a middle ground in terms of latency and cost.
Coded-exposure events have the highest latency—events are generated only every $\tildeNice256$–$512$ binary frames—but the lowest on-chip cost. This provides an end user the flexibility to choose from the space of generalized event cameras based on the latency requirements and the compute constraints of the target application.

§.§ On-Chip Feasibility and Validation

A critical limitation of single-photon sensors is the exorbitant bandwidth and power costs involved in reading off raw photon detections. However, the lightweight nature of our event camera designs allows us to sidestep this limitation by performing computations on-chip.
We demonstrate that our methods are feasible on UltraPhase <cit.>, a SPAD compute platform. UltraPhase consists of 3$\times$6 compute cores, each of which is associated with 4$\times$4 pixels.

We implement our methods for UltraPhase using custom assembly code. Some methods require minor modifications due to instruction-set limitations; see the supplement for details. We process $2500$ SPAD frames from the tennis sequence used in <ref>, cropped to the UltraPhase array size of $12 \times 24$ pixels. We determine the number of cycles required to execute the assembly code and estimate the chip's power consumption and readout bandwidth.

All our proposed methods run comfortably within the chip's compute budget of $4202$ cycles per binary frame and its memory limit of $4$ Kibit per core. As seen in <ref>, compared to raw photon-detection readout, our techniques reduce both bandwidth and power costs by over two orders of magnitude.
The coded-exposure method is particularly efficient; on most binary frames, it only requires multiplying a binary code with incident photon detections.
Our proof-of-concept evaluation may pave the way for future near-sensor implementations of generalized event cameras, which with advances in chip-to-chip communication, could involve a dedicated “photon processing unit”, similar to a camera image signal processor (ISP).

§ LIMITATIONS AND DISCUSSION

Generalized events push the frontiers of event-based imaging; however, some scenarios lead to sub-optimal performance. As seen in <ref> (left), if the scene dynamics is entirely comprised of rigid motion, burst photography <cit.> gives better image quality, albeit with much higher readout. (middle) Similar to current event cameras, modulated light sources trigger unwanted events that reduce bandwidth savings. However, it may be possible to ignore some of these events, perhaps by modeling the lighting variations <cit.>.

Ego-motion events. Camera motion can trigger events in static regions, although our methods still yield substantial compression ($130\times$ over SPAD outputs, <ref> right). We analyze the impact of ego-motion on bandwidth savings further in the supplement. However, single-photon cameras can emulate sensor motion by integrating flux along alternate spatiotemporal trajectories <cit.>. We can imagine a generalized event camera that is “ego-motion compensated,” by computing events along a suitable trajectory.

Photon-stream compression. SPADs generate a torrent of data—, $12.5$ GBps for a MPixel array at $100$ kHz—that can easily overwhelm data interfaces. Generalized event cameras reduce readout by around two orders of magnitude, by decoupling readout from the SPAD's frame rate and instead basing it on scene dynamics. This could pave the way for practical, high-resolution single-photon sensors.

§ ACKNOWLEDGMENTS

This research was supported in parts by NSF CAREER award 1943149, NSF award CNS-2107060, and the Swiss National Science Foundation grant 200021_166289.
We thank Paul Mos for providing us access to SwissSPAD2 acquisition software.
We also thank Trevor Seets for valuable discussions during the early stages of our research.

Supplementary Material

§ GENERALIZED EVENT ALGORITHMS

§.§ Overview

<ref> shows an overview of recovering scene intensity in a bandwidth-efficient manner using generalized event cameras. This process begins from the SPAD's output response, $\Phi(\vx, t)$ at pixel location $\vx$ and (discrete) time $t$, and culminates in a high-fidelity, high-temporal resolution video reconstruction. In what follows, we go over the salient steps before laying down specific algorithms (in <ref>) and detailing any algorithm-specific modifications to these steps.

Event generation. Each of our algorithms (<ref>) takes as input the high-speed binary frames produced by the SPAD, $\Phi(\vx, t)$, and processes it in an online manner—without any buffering of photon detections. The output of our algorithms is an asynchronous (each pixel or patch can emit events independently) spatiotemporal stream of event packets.

Event packets. For our methods, we assume events are sparsely encoded using a coordinate list (COO) format. In other words, we represent each event as a tuple $(\vx, t, \Lambda)$, where $\vx$ is the spatial location, $t$ is the time, and $\Lambda$ is the event payload—which is typically the integrator's value ($\Sigma$), except in the case of coded-exposure events where we transmit a set of integrator values. For the adaptive-EMA and Bayesian methods, $\Lambda = \Sigma_\text{cuml}$, the adaptive integrator. For the spatiotemporal chunk method, $\Lambda = \bm{\Sigma}_\text{patch}$, the patch-wise cumulative mean (a vector). For the coded method, $\Lambda = (\Sigma_\text{long}, \{\Sigma_\text{coded}^j\})$, the long exposure and most recent coded exposures (note that we can exclude $\Sigma_\text{long}$ if this is the change detector's first timestep or if we produced an event on the timestep immediately preceding this one).

Backtracking. When an event is emitted at time $t=T_1$, the integrator's value, $\Sigma$, is taken to be the flux estimate (or representation) between $T_1$ and the previous event emission time ($T_0$)—if this is the first event emission, we assume $T_0=0$. Thus, from the spatiotemporal event stream, we can construct a piece-wise constant representation of the incident flux, by traversing the event stream backward in time; we term this process as backtracking.

Sampling. After backtracking, we obtain a spatiotemporal cube of intensity estimates. We can now sample this cube discretely to obtain one or more frame-based samples. The main motivation for sampling the intensity cube, rather than processing it in its entirety, is that existing video restoration models are not capable of inference on very long video sequences (most models infer on $32$–$64$ video frames at a time). In this work, we consider a very simple sampling strategy: temporally uniform sampling. There can be, however, more sophisticated ways to sample, potentially based on the rate of events across time <cit.>—we do not explore these more sophisticated variants in this paper.

Restoration. Having obtained a frame-based sampling (video representation) of our backtracked cube, we can now process it using video restoration techniques. The purpose of video restoration here is to remove artifacts arising from the independence of events between pixels (or spatial patches). For instance, neighboring pixels may fire events at different times, which tends to produce jagged artifacts around motion boundaries. We show these asynchronous artifacts (and their removal) in the insets of <ref>.

§.§ Adaptive-EMA Event Camera

In <ref>, we describe our simplest variant of the adaptive exposure technique, which was introduced in <ref> and further specified in <ref>. This technique uses a fixed threshold applied to exponential moving averages (EMAs) to determine changes in scene intensity. Such a choice of change detector is motivated by early works in change-point detection that have utilized exponential moving averages <cit.>. We show intermediate and final outputs of adaptive-EMA on the slingshot sequence in <ref>.

§.§ Adaptive-Bayesian Event Camera

In <ref>, we describe our generalized event camera (from <ref>), which uses (a variant of) restarted Bayesian online change detection (R-BOCPD <cit.>) as the per-pixel change detector. As mentioned in <ref>, our main modification is the use of extreme pruning <cit.>: instead of storing one forecaster for each timestep (or per binary frame), we only retain the top-$K$ forecasters. We found that the performance of the change detector is not significantly impacted even with substantial pruning where we retain just the top-$3$ forecasters; see <ref>. We now describe a few variants of <ref>.

Restarts and pseudo-distribution array. We can also consider the base variant of BOCPD <cit.>, which does not involve restarts or an array of pseudo-distribution values. In this case, we can remove the arrays in line $7$ of <ref>, and initialize a new forecaster as $\nu^\prime = \gamma \sum_k \nu_k l_k$. Removing restarts allows us to skip lines $35$–$38$ in <ref>, and replace line $31$ by $\argmax_k \nu_k \neq T_0$, where $T_0$ is the timestamp of the previous event at the same pixel location $\vx$. We use this simplification for our on-chip implementation for UltraPhase <cit.>.

Direct extensions to spatial patches. While BOCPD is a per-pixel temporal change detector, there are simple ways to exploit the correlated changes observed in a patch—although these modifications should be seen as simple extensions and not a more principled approach, such as what we describe in <ref>. To reduce detection delay, we can fire events whenever a changepoint is detected in any pixel in a patch. While this may be preemptive for pixels where change has not yet been detected, being preemptive may not be detrimental, as a change in one pixel indicates that changes may soon be observed at other pixels in a patch.

We show intermediate and final outputs of adaptive-Bayesian on the slingshot sequence in <ref>. Notice that the dynamics of the slingshot's elastic band and the propelled ping-pong ball are much better preserved in <ref> (as compared to <ref>), while entailing a reduced sensor readout as well.

§.§ Spatiotemporal-Chunk Event Camera

<ref> provides a formal description of the spatiotemporal-chunk event camera. Recall from <ref> that this method generates an event whenever

\begin{equation}
\norm{\mathbf{P}\paren{\tilde{\bm{\Phi}}_\text{chunk}(\vy, t) - \tilde{\bm{\Sigma}}_\text{patch}(\vy, t)}} \geq \tau.
\end{equation}

Let $p \times p$ be the patch size, with $q = p^2$ the number of pixels in a patch. We use a patch size of $4 \times 4$ in all our experiments (except for Fig. 4, where we use $8 \times 8$ for illustrative purposes). $\tilde{\bm{\Phi}}_\text{chunk}(\vy, t) \in \mathbb{R}^q$ is the normalized mean over a temporal chunk of $m$ binary frames; we use $m = 32$ throughout the paper. $\tilde{\bm{\Sigma}}_\text{patch}(\vy, t) \in \mathbb{R}^q$ is the normalized cumulative mean since the last event, excluding the current temporal chunk. $\bm{P} \in \mathbb{R}^{r \times q}$ is a feature matrix. We use $r = 16$ in our main experiments; for UltraPhase experiments, we reduce this to $r = 4$ due to on-chip memory constraints (see <ref>).

We normalize via

\begin{align}
\tilde{\bm{\Phi}}_\text{chunk}(\vy, t) &= \bm{\Phi}_\text{chunk}(\vy, t) \oslash \mathbf{c} \\
\tilde{\bm{\Sigma}}_\text{patch}(\vy, t) &= \bm{\Sigma}_\text{patch}(\vy, t) \oslash \mathbf{c},
\end{align}

where $\oslash$ indicates pointwise (Hadamard) division and $\bm{c}$ is given by

\begin{gather}
\bm{c}(\vy, t) = \sqrt{\hat{\bm{p}}(\vy, t) \, (1 - \hat{\bm{p}}(\vy, t)) \frac{1}{m} \left( 1 + \frac{1}{n} \right)}, \\
\hat{\bm{p}}(\vy, t) = \frac{n \bm{\Sigma}_\text{patch}(\vy, t) + \bm{\Phi}_\text{chunk}(\vy, t)}{n + 1},
\end{gather}

where $m$ is the temporal chunk size, $n$ is the number of temporal chunks comprising $\bm{\Sigma}_\text{patch}$, and the square root is pointwise. $\bm{c}^2$ is an empirical estimate of the variance in $\bm{\Phi}_\text{chunk} - \bm{\Sigma}_\text{patch}$ under a static assumption, in which case $\bm{\Phi}_\text{chunk}$ and $\bm{\Sigma}_\text{patch}$ are binomial random variables with the same probability. We use ghost sampling when computing $\hat{\bm{p}}$ to prevent numerical instability near $\hat{p} = 0$ and $\hat{p} = 1$; specifically, we add $8$ ghost Bernoulli measurements, $4$ of which are successes.

Training matrix $\bm{P}$. We use the procedure outlined in <ref> to generate outputs that can be used to train the matrix $\bm{P}$ via backpropagation through time (BPTT). <ref> does not involve any branched control flow, and gives outputs identical to <ref>. The event-generation decision is represented by a binary value $k$. On the forward pass, $k$ is computed using a Heaviside function. On the backward pass, we approximate the gradients of the Heaviside using a surrogate gradient approach. Specifically, we replace the Heaviside gradients with those of a logistic sigmoid.

Note that <ref> also includes an autodiff-compatible backtracking step, where we fill in the values of $\bm{B}$ in time-reverse order. Unlike an indexing operation, which is not differentiable with respect to its indices, this implementation is differentiable with respect to the values of $k$, which delimit the piecewise-constant segments of $\bm{B}$.

We train using interpolated videos from the XVFI dataset <cit.>. We interpolate from the native $1$ kHz frame rate to $16$ kHz using RIFE <cit.>, then linearly interpolate over time by another factor of $8$ (for a total binary frame rate of $128$ kHz). We use the interpolated frames as the ground truth during training. To generate SPAD frames, we treat the ground truth values (in the range $[0, 1]$) as the Bernoulli photon-detection probability.

We initialize $\bm{P}$ with uniformly-distributed random values in the range $[-0.0125, 0.0125]$. We train for $20$ epochs, using vanilla SGD with a learning rate of $0.06$, reducing the learning rate by a factor of $5$ after $10$ epochs. Each epoch consists of $200$ batches, with each batch containing $64$ patch time series. Throughout training, we randomly vary the contrast threshold via uniform sampling in the range $[1 / 1.3, 1 / 0.7]$.

§.§ Coded-Exposure Event Camera

Mask pattern (coding) details. We choose $J \geq 2$ random binary mask sequences of length $N$ each, , $\bm{C}^j \in \{0, 1\}^N$. Each of these $J$ sequences is non-overlapping, which is

\begin{equation}
\bm{C}^j(n) \bm{C}^k(n) = 0 \;\forall\; j \neq k, \;\forall\; 1 \leq n \leq N.
\end{equation}

The motivation behind such a choice is to ensure that the pseudo-inverse step (Moore-Penrose inverse), which is applied to the coded measurements as a processing step, can be computed efficiently. With these constraints, the pseudo-inverse step involves multiplication (of the coded measurements) by a diagonal matrix, which can be carried out efficiently.

One way to construct such a mask sequence is by choosing $j^* \sim \text{Uniform}(1, J)$ at each sequence location $n \in {1, 2, ..., N}$, and then setting $\bm{C}^{j^*}(n) = 1$ and $\bm{C}^{j}(n) = 0$ for all other $j \neq j^*$. In other words, we pick a random “bucket” at each subframe index $n$.

For instance, when $J=2$, this amounts to picking to mask sequences, $\bm{C}^1, \bm{C}^2 \in \{0, 1\}^N$, such that

\begin{equation}
\bm{C}^1(n) = 1 - \bm{C}^2(n) \;\forall\; 1\leq n \leq N.
\end{equation}

This choice corresponds to the “coded two-bucket” camera <cit.>. Thus, these random binary masks can be seen as a generalization of computing a single coded measurement to computing $J$ coded measurements <cit.>.

Finally, we remark that we do not consider the case of $J=1$ here, since we implement coded exposures computationally on single-photon sensors—hence, the “complementary” coded exposure (or the two-bucket measurement) is always readily available. In other words, there is no drastic compute or memory overhead of a (computational) two-bucket coded exposure over a single coded exposure. This would not be the case if an optical setup (, using digital micromirror devices, DMDs) were used to implement coded exposures—where using a two-bucket measurement would likely entail a beam splitter and a second DMD.

Pseudo-inverse step. After backtracking, we can sample a $J$ bucket coded-exposure measurement at any time $t$. Of course, if the pixel is static, we only get $1$ static measurement at the pixel location—so we repeat static measurements $J$ times. Before applying our video restoration module, we perform a pseudo-inverse step that is derived from the linear forward model of $J$-bucket coded exposures and is similar to the pseudo-inverse pre-processing step adopted for single-bucket compressive captures <cit.>. This pre-processing step involves computing

\begin{equation}
Y(\vx, n) = \sum_{j=1}^J \frac{1}{\sum_{m=1}^N \bm{C}^j(\vx, m)} \Sigma^j(\vx, t) \bm{C}^j(\vx, n),
\end{equation}

giving us $N$ subframes from the $J$ coded measurements.

To summarize, for coded-exposure events, we follow these steps:

\begin{equation}
\text{Backtrack and sample} \rightarrow \text{Repeat static regions by $J\times$} \rightarrow \text{Pseudo-inverse pre-processing} \rightarrow \text{Video restoration}.
\end{equation}

We show intermediate and final outputs of (two-bucket) coded-exposure events on the slingshot sequence in <ref>.

§ EXTENDED DISCUSSION

§.§ More Sophisticated Integrators

In this work, we propose two integrators: adaptive exposures and coded exposures. There remains an extensive, unexplored space of alternate integrators. Both adaptive and coded exposures are linear projections of a pixel's photon detections over time. We could consider more general linear mappings, such as continuous-valued temporal codes. Further, we are not restricted to projections over time; it may also be advantageous to consider spatial projections, , frequency-domain transforms. With these alternate projections, we might be able to reduce bitrates while maintaining reconstruction quality.

We have so far assumed that our objective is intensity reconstruction. However, there may be situations where we know that the final goal is a specific inference task. In such a scenario, we could design integrators that only encode information relevant to the task at hand; this may be significantly more bandwidth-efficient than transmitting a generic intensity encoding. This integrator could take the form of a learned module, , a neural network, that operates near-sensor and computes a compressed, task-specific scene representation. We could train this module end-to-end with the downstream layers that perform final inference. The resulting system would involve a neural network that spans multiple compute devices, with an event-based communication layer in the middle. This setup somewhat resembles spiking neural networks and other event-based networks, although the goal with such methods is generally reduced computation costs (arising from sparse layer inputs) and not reduced bandwidth along a data-transfer interface.

§.§ Entropy Coding and Quantization

Generalized event cameras encode intensity levels, in the range $[0, 1]$, representing the photon-detection rate. In our experiments, we apply uniform quantization to the transmitted values; this keeps our comparisons straightforward and fair.

However, in a practical deployment, it may be more bandwidth-efficient to encode changes rather than values and apply entropy coding to the changes. When transmitting changes, the first event encodes a value in $[0, 1]$, and subsequent events encode a difference in $[-1, +1]$. This approach is functionally equivalent to transmitting levels or values, assuming the event camera correctly tracks quantization effects (to avoid drift). For natural scenes, the distribution of changes is non-uniform. The shape of this distribution depends in part on the change-detection algorithm; in general, we would expect changes near zero to be unlikely, as these would not trigger an event. We can apply entropy coding (, Huffman coding) to exploit the nonuniformity in the distribution and achieve some additional compression. Entropy coding could give substantial bandwidth savings, albeit at the cost of some increased near-sensor computation.

In addition to entropy coding, we could apply non-uniform quantization, either to values or changes. For example, we could non-uniformly quantize values in $[0, 1]$ based on perceptual considerations (, human sensitivity to intensity differences). With changes, we could exclude portions of the range $[-1, +1]$ based on the characteristics of the change detector; for example, with a fixed contrast threshold $\tau$, there is no need to represent changes in $(-\tau, \tau)$.

§.§ Spatial Compression

One advantage of patch-wise events (, as described in Sec. 4.2) is that they permit some spatial compression. For example, we can apply JPEG block compression to the payload of an $8 \times 8$ patch-wise event. Such a change would bring our techniques more in line with existing video-compression algorithms, which compress in both space and time. The compression ratio we observe would depend on the sensor resolution; with higher resolution, we would expect image patches to be more uniform, and thus more easily compressible. Likewise, patches with more noise (, due to a short adaptive integration window) would be more difficult to compress. Under ideal conditions, patch-wise compression might give us an additional $\tildeNice5\times$ reduction in bitrate. We leave this idea as a topic for future work.

§.§ Alternate Sparse Formats

As described in <ref>, we assume a COO event format $(\vx, t, \Lambda)$ when evaluating the bandwidth requirements of our methods. We could improve the sparse format to reduce bandwidth costs further. One such improvement would be to use an implicit time encoding. Specifically, on each binary frame, we would transmit a header $(t, n)$ indicating the current timestamp and the number of events on this frame. We would then send $n$ event packets $(\vx, \Lambda)$. If $n \gg 1$, this approach would virtually eliminate the overhead associated with transmitting values of $t$. Note, however, that this approach assumes we communicate events in time order; , if $t_2 > t_1$, all events at time $t_1$ are sent before any events at time $t_2$.

To reduce the overhead associated with $\vx$, we could employ sparse matrix formats such as compressed sparse row (CSR) or compressed sparse column (CSC). These formats compress one of the two spatial coordinates (the row and column indices, respectively). The savings relative to COO would depend on the density of events on each frame, with denser events leading to more compression with CSR or CSC.

For pixel-wise methods (, adaptive-EMA, Bayesian, and coded), another potential optimization is to adaptively group events into patch-wise packets. Assume we are using the COO format $(\vx, t, \Lambda)$. If a spatial patch contains many events, it may be more efficient to transmit them in a single packet, as this amortizes the overheads of $\vx$ and $t$. To implement this optimization, we would add an indicator bit $b$ to each event packet. If $b = 0$, then the packet should be treated as a pixel-wise event; if $b = 1$, it is a patch-wise event, meaning $\Lambda$ encodes integrator information for an entire patch. Within a patch-wise event we would use dense format for $\Lambda$, marking the pixels that triggered an event with a one-bit mask. Each patch would adaptively determine whether to encode its events pixel-wise or patch-wise. This decision would be based on the number of events, with the optimal threshold depending on the number of bits required for $\vx$, $t$, and $\Lambda$, as well as the patch size.

§.§ Latency of the Adaptive Integrator

One limitation of the adaptive integrator $\Sigma_\text{cumul}(\vx, T_1)$ is that it creates some latency in the intensity estimates. The estimate for the duration $[T_0, T_1]$—which the adaptive integrator computes as the mean of $\Phi(\vx, t)$ over $[T_0, T_1]$—is not known until $T_1$. Thus, there is a latency of $T_1 - t$ to obtain an estimate for $t \in [T_0, T_1]$. The expected delay depends on the frequency of events, with lower delay in more dynamic regions. The latency of $\Sigma_\text{cumul}(\vx, T_1)$ is not an issue for offline reconstruction and inference (which we assume throughout this paper). However, it may be of concern for certain real-time applications.

As a potential solution, we could introduce a second type of event, which we call an eager event, in contrast to the change events we have considered thus far. Assume again an event is generated at time $T_1$. In addition to transmitting $\Sigma_\text{cumul}(\vx, T_1)$ at that moment, we could send an eager event encoding a noisy estimate of $\Phi(\vx, T_1)$. We could then send periodic eager events encoding refinements to the value of the adaptive integrator. Assuming constant flux after $T_1$, the adaptive integrator converges to the true flux value as time passes; transmitting refinements would allow downstream components to leverage this improved estimate in the absence of a subsequent change event. We could consider various schedules for sending refinements—, at exponentially increasing intervals, or at fixed intervals until some maximum time has passed. Note, however, that to keep the event camera's bandwidth coupled with (proportional to) the scene dynamics, there would need to be an upper bound to the number of eager refinement events, , with each change event triggering at most $N$ refinements.

The solution described above would involve some additional bandwidth costs. We can thus imagine a generalized event camera that operates in two modes: “online mode” and “offline mode,” with eager events only being used in the online mode, where low latency is necessary.

§ RESTORATION MODEL DETAILS

§.§ Model Architecture

While any video restoration model could be used, we choose the densely-connected residual network proposed in <cit.> (EfficientSCI) for our video restoration architecture—which was successful at restoring backtracked outputs of all of our generalized event cameras (adpative-EMA, adaptive-Bayesian, spatiotemporal chunk, and coded-exposure events). We also experimented with the spatial-temporal shift-based model of <cit.> (ShiftNet): while this architecture was successful at restoring three of our four proposed events, it did not succeed at restoring backtracked coded exposures. We attribute this to the fact that EfficientSCI was designed with video compressive sensing in mind, whereas ShiftNet is targeted for more general video-restoration tasks.

The memory cost of EfficientSCI scales with the number of input frames. Thus, we are constrained by the device memory in the number of frames we can reconstruct. For example, with $24$ GB of GPU memory and a resolution of $512 \times 256$, we can reconstruct about $96$ video frames—which can cover a temporal extent of $3000$–$9000$ binary frames ($30$–$90$ ms). Increasing the temporal extent requires sampling the backtracked cube at an increased temporal stride, which can lead to blurring and lower-quality results. One solution to this problem might be to employ a recurrent architecture with a fixed-size memory. However, forward-mode recurrent inference must be causal; , the predicted frame at time $t$ may only consider backtracked samples from times before $t$.
An efficient recurrent model may enable reconstructing videos at frame-rates faster than what we show in this work ($\tildeNice3000$ FPS)—indeed, our methods that operate the granularity of individual binary frames (, <ref>) can, in principle, provide reconstructions at the frame-rate of SPAD photon-detections, , $96.8$ kHz.

§.§ Dataset

We use $4403$ high-speed videos from the training split of the XVFI dataset <cit.> to train our restoration models. We temporally interpolate these $1000$ FPS to $16000$ FPS using
RIFE <cit.> and treat each video frame, normalized between $0...1$ as the photon-detection probability. We then draw $6$ binary frames per video frame (as a per-pixel Bernoulli random variable, based on the photon-detection probabilities)—thereby giving us binary-valued responses at $96$ kHz. We remark that our dataset generation approach here (unlike in <ref>) is not physically accurate, since we directly treat the video's values as photon-detection probabilities instead of average photon-arrival rates. We adopt this approach for its simplicity and note that the difference (between detection probabilities and Poisson rates) manifests as a tone-mapping operation by the SPAD's response function ($f(x) = 1-e^{-x}$). We did not see any generalization issues when applying our models trained this synthetic dataset to real SPAD data—in both ambient and low-light scenarios.

§.§ Training Parameters

All restoration models were trained until convergence ($40$–$60$ epochs) using the Adam optimizer <cit.> and the mean squared error (MSE) loss objective. We used an initial learning rate of $10^{-5}$, which was decayed as per a cosine-annealed scheduler <cit.> to a minimum value of $10^{-8}$. When training the spatiotemporal chunk method, we randomly choose $\tau$ on each training iteration from a uniform distribution covering the range $[0.76, 1.43]$. We also clip the gradient norm to $1.0$ to resolve instability during training.

§ IMAGING SETUP

<ref> shows the cameras we used for demonstrating the capabilities of generalized events: our algorithms process the outputs of the SwissSPAD2 array. We compare the performance of generalized events against the imaging capabilities of a commercial event camera (Prophesee EVK4) and a high-speed camera that is capable of real-time streaming (Photron Infinicam).

§.§ Imager Specifications

SwissSPAD2 Array. The sensor has a resolution of $512\times 512$ pixels: however, we operate the device in its “half-array mode”, meaning that only one subarray of resolution $512\times 256$ pixels is used. Each pixel has a pixel pitch of $16.4$ $\mu$m and a fill factor of less than $10\%$. The fill factor can be improved by the inclusion of microlens arrays which this prototype lacks.

The SPAD arrays features a certain number ($\tildeNice 5$%) of “hot pixels”: pixels whose dark count rate (DCR) is abnormally high, and as a result, almost always return $1$s. We calibrate a mask of hot pixels by using sensor responses that were taken in the dark. We find that hot pixels do not impact the generation of generalized events—thus, we inpaint hot pixels (either using the Telea algorithm <cit.> or nearest-neighbor replacement) using the calibrated mask after backtracking is performed, , post-sensor readout.

Prophesee EVK4. This commercially available event camera has a sensor resolution of $1280 \times 720$ pixels. Each pixel has a pixel pitch of $4.86$ $\mu$m and a fill factor of $>77\%$.

Photron Infinicam. This high-speed camera provides a USB-C data interface, which is enabled by performing compression on the fly. The camera SDK does not provide access to parameters that control this online compression—as a result, we find that when imaging scenes where the camera is per-frame SNR limited, such as the indoor scene of <ref>, read-noise induced artifacts translate to severe compression artifacts. The camera has a resolution of $1246 \times 1024$ pixels.

§.§ Scene Acquisition Details

We list the (focal lengths of C-mount) lenses used to capture each of the scenes shown in our main paper, along with the acquisition time of each scene (in terms of the number of binary frames captured by the SPAD at $96.8$ kHz).

* Slingshot sequence in <ref>: $25$ mm lens, $4000$ binary frames.

* Tennis sequence in <ref>: $50$ mm lens, $3000$ binary frames.

* Vertical wheel in <ref>: $12$ mm lens, $3000$ binary frames.

* Nighttime traffic sequence in <ref>: $25$ mm lens, $3000$ binary frames.

* Dartboard sequence in <ref>: $16$ mm lens, $3000$ binary frames.

* Jack-in-the-box sequence in <ref>: $16$ mm lens, $4000$ binary frames.

* Rotating hole-saw bit sequence in <ref>: $35$ mm lens, $6000$ binary frames.

* Casino roulette sequence in <ref>: $25$ mm lens, $4000$ binary frames.

* Stress ball sequence in <ref>: $25$ mm lens, $4096$ binary frames.

* Nighttime traffic sequence in <ref>: $25$ mm lens, $8000$ binary frames.

* Tennis sequence in <ref>: $100$ mm lens, $8192$ binary frames.

§ EXTENDED EXPERIMENTS

In this supplementary note, we provide extended versions of our experiments in the main paper. We include detailed descriptions of our constructed baselines and make fine-grained comparisons across our proposed methods.