# Learning to Zoom and Unzoom

Chittesh Thavamani1 Mengtian Li 1 Francesco Ferroni 2 Deva Ramanan1
1Carnegie Mellon University 2Argo AI
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Now at Waymo.Now at Nvidia.

###### Abstract

Many perception systems in mobile computing, autonomous navigation, and AR/VR
face strict compute constraints that are particularly challenging for high-
resolution input images. Previous works propose nonuniform downsamplers that
"learn to zoom" on salient image regions, reducing compute while retaining
task-relevant image information. However, for tasks with spatial labels (such
as 2D/3D object detection and semantic segmentation), such distortions may
harm performance. In this work (LZU), we "learn to zoom" in on the input
image, compute spatial features, and then "unzoom" to revert any deformations.
To enable efficient and differentiable unzooming, we approximate the zooming
warp with a piecewise bilinear mapping that is invertible. LZU can be applied
to any task with 2D spatial input and any model with 2D spatial features, and
we demonstrate this versatility by evaluating on a variety of tasks and
datasets: _object detection_ on Argoverse-HD, _semantic segmentation_ on
Cityscapes, and _monocular 3D object detection_ on nuScenes. Interestingly, we
observe boosts in performance even when high-resolution sensor data is
unavailable, implying that LZU can be used to "learn to upsample" as well.
Code and additional visuals are available at https://tchittesh.github.io/lzu/.

##  1 Introduction

In many applications, the performance of perception systems is bottlenecked by
strict inference-time constraints. This can be due to limited compute (as in
mobile computing), a need for strong real-time performance (as in autonomous
vehicles), or both (as in augmented/virtual reality). These constraints are
particularly crippling for settings with high-resolution sensor data. Even
with optimizations like model compression [4] and quantization [23], it is
common practice to downsample inputs during inference.

However, running inference at a lower resolution undeniably destroys
information. While some information loss is unavoidable, the usual solution of
uniform downsampling assumes that each pixel is equally informative towards
the task at hand. To rectify this assumption, Recasens  _et al_. [20] propose
Learning to Zoom (LZ), a nonuniform downsampler that samples more densely at
salient (task-relevant) image regions. They demonstrate superior performance
relative to uniform downsampling on human gaze estimation and fine-grained
image classification. However, this formulation warps the input image and thus
requires labels to be invariant to such deformations.

Figure 1: LZU is characterized by "zooming" the input image, computing spatial
features, then "unzooming" to revert spatial deformations. LZU can be applied
to any task and model that makes use of internal 2D features to process 2D
inputs. We show visual examples of output tasks including 2D detection,
semantic segmentation, and 3D detection from RGB images.

Adapting LZ downsampling to tasks with spatial labels is trickier, but has
been accomplished in followup works for semantic segmentation (LDS [11]) and
2D object detection (FOVEA [22]). LDS [11] does not unzoom during learning,
and so defines losses in the warped space. This necessitates additional
regularization that may not apply to non-pixel-dense tasks like detection.
FOVEA [22] does unzoom bounding boxes for 2D detection, but uses a special
purpose solution that avoids computing an inverse, making it inapplicable to
pixel-dense tasks like semantic segmentation. Despite these otherwise elegant
solutions, there doesn’t seem to be a general task-agnostic solution for
intelligent downsampling.

Our primary contribution is a general framework in which we zoom in on an
input image, process the zoomed image, and then unzoom the output back with an
inverse warp. Learning to Zoom and Unzoom (LZU) can be applied to any network
that uses 2D spatial features to process 2D spatial inputs (Figure 1) with no
adjustments to the network or loss. To unzoom, we approximate the zooming warp
with a piecewise bilinear mapping. This allows efficient and differentiable
computation of the forward and inverse warps.

To demonstrate the generality of LZU, we demonstrate performance a variety of
tasks: _object detection_ with RetinaNet [17] on Argoverse-HD [14], _semantic
segmentation_ with PSPNet[29] on Cityscapes [7], and _monocular 3D detection_
with FCOS3D [26] on nuScenes [2]. In our experiments, to maintain favorable
accuracy-latency tradeoffs, we use cheap sources of saliency (as in [22]) when
determining where to zoom. On each task, LZU increases performance over
uniform downsampling and prior works with minimal additional latency.

Interestingly, for both 2D and 3D object detection, we also see performance
boosts even when processing low resolution input data. While prior works focus
on performance improvements via intelligent downsampling [20, 22], our results
show that LZU can also improve performance by intelligently upsampling
(suggesting that current networks struggle to remain scale invariant for small
objects, a well-known observation in the detection community [18]).

##  2 Related Work

We split related work into two sections. The first discusses the broad class
of methods aiming to improve efficiency by paying "attention" to specific
image regions. The second delves into works like LZU that accomplish this by
differentiably resampling the input image.

###  2.1 Spatial Attentional Processing

By design, convolutional neural networks pay equal "attention" (perform the
same computations) to all regions of the image. In many cases, this is
suboptimal, and much work has gone into developing attentional methods that
resolve this inefficiency.

One such method is Dynamic Convolutions [24], which uses sparse convolutions
to selectively compute outputs at only the salient regions. Similarly, gated
convolutions are used in [12, 28]. Notably, these methods implement "hard"
attention in that the saliency is binary, and non-salient regions are ignored
completely.

Deformable Convolutions [8, 30] provides a softer implementation of spatial
attention by learning per pixel offsets when applying convolutions, allowing
each output pixel to attend adaptively to pixels in the input image. SegBlocks
[25] also provides a softer attention mechanism by splitting the image into
blocks and training a lightweight reinforcement learning policy to determine
whether each block should be processed at a high or low resolution. This is
akin to our method, which also has variable resolution, albeit in a more
continuous manner. Our method is also generalizable to tasks in which it’s
infeasible to "stitch" together outputs from different blocks of the image
(e.g. in detection where an object can span multiple blocks).

###  2.2 Spatial Attention via Differentiable Image Resampling

Spatial Transformer Networks [10] introduces a differentiable method to
resample an image. They originally propose this to invert changes in
appearance due to viewpoint, thereby enforcing better pose invariance.

Learning to Zoom (LZ) [20] later adapts this resampling operation to "zoom" on
salient image regions, acting as a spatial attention mechanism. Their key
contribution is a transformation parameterized by a saliency map such that
regions with higher saliency are more densely sampled. However, this deforms
the image, requiring the task to have non-spatial labels.

Followup works [19, 22, 11] adapt LZ downsampling to detection and semantic
segmentation. For object detection, FOVEA [22] exploits the fact that image
resampling is implemented via an inverse mapping to map predicted bounding
boxes back into the original image space. This allows all processing to be
done in the downsampled space and the final bounding box regression loss to be
computed in the original space. However, when there are intermediate losses,
as is the case with two-stage detectors containing region proposal networks
(RPNs) [21], this requires more complex modifications to the usual delta loss
formulation, due to the irreversibility of the inverse mapping. For semantic
segmentation, Jin  _et al_. [11] apply LZ downsampling to both the input image
and the ground truth and computes the loss in the downsampled space. This is
elegant and model-agnostic but leads to misalignment between the training
objective and the desired evaluation metric. In the extreme case, the model
learns degenerate warps that sample "easy" parts of the image to reduce the
training loss. To address this, they introduce additional regularization on
the downsampler. Independently, [19] handcraft an energy minimization
formulation to sample more densely at semantic boundaries.

In terms of warping and unwarping, the closest approach to ours is Dense
Transformer Networks [13], which also inverts deformations introduced by
nonuniform resampling. However, their warping formulation is not saliency-
based, which makes it hard to work with spatial or temporal priors and also
makes it time-consuming to produce the warping parameters. Additionally, they
only show results for semantic segmentation, whereas we show that our
formulation generalizes across spatial vision tasks.

##  3 Background

Since our method is a generalization of previous works [20, 22, 11], we
include this section as a condensed explanation of prerequisite formulations
critical to understanding LZU.

###  3.1 Image Resampling

Suppose we want to resample an input image $\mathbf{I}({\mathbf{x}})$ to
produce an output image $\mathbf{I}^{\prime}({\mathbf{x}})$, both indexed by
spatial coordinates ${\mathbf{x}}\in[0,1]^{2}$. Resampling is typically
implemented via an inverse map $\mathcal{T}:[0,1]^{2}\to[0,1]^{2}$ from output
to input coordinates [1]. For each output coordinate, the inverse map computes
the source location from which to "steal" the pixel value, i.e.
$\mathbf{I}^{\prime}({\mathbf{x}})=\mathbf{I}(\mathcal{T}({\mathbf{x}}))$. In
practice, we are often given a discretized input image
$\mathbf{I}\in\mathbb{R}^{H\times W\times C}$ and are interested in computing
a discretized output $\mathbf{I}^{\prime}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times C}$. To do so, we compute $\mathbf{I}^{\prime}({\mathbf{x}})$
at grid points ${\mathbf{x}}\in\mathrm{Grid}(H^{\prime},W^{\prime})$, where
$\mathrm{Grid}(H,W):=\mathrm{Grid}(H)\times\mathrm{Grid}(W)$ and
$\mathrm{Grid}(D):=\\{\frac{d-1}{D-1}:d\in[D]\\}$. However,
$\mathcal{T}({\mathbf{x}})$ may return non-integer pixel locations at which
the exact value of $\mathbf{I}$ is unknown. In such cases, we use bilinear
interpolation to compute $\mathbf{I}(\mathcal{T}({\mathbf{x}}))$. As proven in
[10], such image resampling is differentiable with respect to $\mathcal{T}$
and $\mathbf{I}$.

###  3.2 Saliency-Guided Downsampling

Figure 2:  Illustration of $\mathcal{T}_{\mathrm{LZ}}$ [20]. Suppose we have a
saliency map $\mathbf{S}\in\mathbb{R}^{h\times w}$ (visualized in the
background) and want a warped image of size $H^{\prime}\times W^{\prime}$. (1)
We start with a uniform grid of sample locations $\mathrm{Grid}(h,w)$. (2)
Grid points are "attracted" to nearby areas with high saliency. (3) Applying
this "force" yields $\mathcal{T}_{\mathrm{LZ}}[\mathrm{Grid}(h,w)]$. (4)
Bilinear upsampling yields
$\widetilde{\mathcal{T}}_{\mathrm{LZ}}[\mathrm{Grid}(H^{\prime},W^{\prime})]$.
Figure 3:  Examples of the anti-cropping ($\mathrm{ac}$) and separable
($\mathrm{sep}$) variants of $\mathcal{T}_{\mathrm{LZ}}$ from [22].

When using nonuniform downsampling for information retention, it is useful to
parameterize $\mathcal{T}$ with a saliency map $\mathbf{S}({\mathbf{x}})$
representing the desired sample rate at each spatial location
${\mathbf{x}}\in[0,1]^{2}$ [20]. Recasens  _et al_. [20] go on to approximate
this behavior by having each sample coordinate $\mathcal{T}({\mathbf{x}})$ be
"attracted" to nearby areas ${\mathbf{x}}^{\prime}$ with high saliency
$\mathbf{S}({\mathbf{x}}^{\prime})$ downweighted according to a distance
kernel $k({\mathbf{x}},{\mathbf{x}}^{\prime})$, as illustrated in Figure 2.
Concretely,
$\mathcal{T}_{\mathrm{LZ}}({\mathbf{x}})=(\mathcal{T}_{\mathrm{LZ},x}({\mathbf{x}}),\mathcal{T}_{\mathrm{LZ},y}({\mathbf{x}}))$,
where

$\mathcal{T}_{\mathrm{LZ},x}({\mathbf{x}})=\frac{\int_{{\mathbf{x}}^{\prime}}\mathbf{S}({\mathbf{x}}^{\prime})k({\mathbf{x}},{\mathbf{x}}^{\prime}){\mathbf{x}}^{\prime}_{x}\,d{\mathbf{x}}^{\prime}}{\int_{{\mathbf{x}}^{\prime}}\mathbf{S}({\mathbf{x}}^{\prime})k({\mathbf{x}},{\mathbf{x}}^{\prime})\,d{\mathbf{x}}^{\prime}},$
(1)
$\mathcal{T}_{\mathrm{LZ},y}({\mathbf{x}})=\frac{\int_{{\mathbf{x}}^{\prime}}\mathbf{S}({\mathbf{x}}^{\prime})k({\mathbf{x}},{\mathbf{x}}^{\prime}){\mathbf{x}}^{\prime}_{y}\,d{\mathbf{x}}^{\prime}}{\int_{{\mathbf{x}}^{\prime}}\mathbf{S}({\mathbf{x}}^{\prime})k({\mathbf{x}},{\mathbf{x}}^{\prime})\,d{\mathbf{x}}^{\prime}}.$
(2)

[22] proposes _anti-cropping_ and _separable_ variants of this downsampler.
The anti-cropping variant $\mathcal{T}_{\mathrm{LZ,ac}}$ prevents the
resampling operation from cropping the image. The separable variant
marginalizes the saliency map $\mathbf{S}({\mathbf{x}})$ into two 1D saliency
maps $\mathbf{S}_{x}(x)$ and $\mathbf{S}_{y}(y)$, and replaces the kernel
$k({\mathbf{x}},{\mathbf{x}}^{\prime})$ with a two 1D kernels $k_{x}$ and
$k_{y}$ (although generally $k_{x}=k_{y}$). Then,
$\mathcal{T}_{\mathrm{LZ,sep}}({\mathbf{x}})=\left(\mathcal{T}_{\mathrm{LZ,sep,x}}({\mathbf{x}}_{x}),\mathcal{T}_{\mathrm{LZ,sep,y}}({\mathbf{x}}_{y})\right)$
where

$\mathcal{T}_{\mathrm{LZ,sep,x}}(x)=\frac{\int_{x^{\prime}}\mathbf{S}_{x}(x^{\prime})k_{x}(x,x^{\prime})x^{\prime}\,dx^{\prime}}{\int_{x^{\prime}}\mathbf{S}_{x}(x^{\prime})k_{x}(x,x^{\prime})\,dx^{\prime}},$
(3)
$\mathcal{T}_{\mathrm{LZ,sep,y}}(y)=\frac{\int_{y^{\prime}}\mathbf{S}_{y}(y^{\prime})k_{y}(y,y^{\prime})y^{\prime}\,dy^{\prime}}{\int_{y^{\prime}}\mathbf{S}_{y}(y^{\prime})k_{y}(y,y^{\prime})\,dy^{\prime}}.$
(4)

This preserves axis-alignment of rectangles, which is crucial to object
detection where bounding boxes are specified via corners. We refer to the
above method and all variants as _LZ downsamplers_ , after the pioneering work
"Learning to Zoom" [20]. Examples of each variant are shown in Figure 3.

##  4 Method

We begin by discussing our general technique for warp inversion. Then, we
discuss the LZU framework and how we apply warp inversion to efficiently
"unzoom".

###  4.1 Efficient, Differentiable Warp Inversion

Figure 4: Given a warp $\mathcal{T}$, we construct an approximation
$\widetilde{\mathcal{T}}$ designed for efficient inversion. As illustrated,
$\widetilde{\mathcal{T}}$ is a piecewise tiling of simpler invertible maps.
This allows us to approximate the inverse $\widetilde{\mathcal{T}}^{-1}$, even
when $\mathcal{T}^{-1}$ lacks a closed form.

Suppose we have a continuous map $\mathcal{T}:[0,1]^{2}\to[0,1]^{2}$. Our
primary technical innovation is an efficient and differentiable approximation
of $\mathcal{T}^{-1}$, even in cases where $\mathcal{T}$ has no closed-form
inverse.

Since $\mathcal{T}$ is potentially difficult to invert, we first approximate
it as $\widetilde{\mathcal{T}}$, a piecewise tiling of simpler invertible
transforms (illustrated in Figure 4). Formally,

$\widetilde{\mathcal{T}}=\bigcup_{\begin{subarray}{c}i\in[h-1]\\\
j\in[w-1]\end{subarray}}\widetilde{\mathcal{T}}_{ij},$ (5)

where the $ij$-th tile $\widetilde{\mathcal{T}}_{ij}$ is any bijective map
from the rectangle formed by corners
$R_{ij}=\\{\frac{i-1}{h-1},\frac{i}{h-1}\\}\times\\{\frac{j-1}{w-1},\frac{j}{w-1}\\}$
to quadrilateral $\mathcal{T}[R_{ij}]$. For our purposes, we choose bilinear
maps as our tile function, although homographies could work just as well.
Then, so long as $\widetilde{\mathcal{T}}$ is injective (if each of the tiles
$\widetilde{\mathcal{T}}_{ij}$ is nondegenerate and no two tiles overlap), we
are guaranteed a well-defined left inverse
$\widetilde{\mathcal{T}}^{-1}:[0,1]^{2}\to[0,1]^{2}$ given by

$\widetilde{\mathcal{T}}^{-1}({\mathbf{x}})=\begin{cases}\widetilde{\mathcal{T}}^{-1}_{ij}({\mathbf{x}})&\text{if
}{\mathbf{x}}\in\mathrm{Range}(\widetilde{\mathcal{T}}_{ij})\\\
0&\text{else}\end{cases}.$ (6)

Equation 6 is efficient to compute, since determining if
${\mathbf{x}}\in\mathrm{Range}(\widetilde{\mathcal{T}}_{ij})$ simply involves
checking if ${\mathbf{x}}$ is in the quadrilateral $\mathcal{T}[R_{ij}]$ and
computing the inverse $\widetilde{\mathcal{T}}^{-1}_{ij}$ of a bilinear map
amounts to solving a quadratic equation [27]. This efficiency is crucial to
maintaining favorable accuracy-latency tradeoffs.
$\widetilde{\mathcal{T}}^{-1}$ is guaranteed to be differentiable with respect
to $\mathcal{T}$, since for each
${\mathbf{x}}\in\widetilde{\mathcal{T}}[R(i,j)]$, the inverse bilinear map can
be written as a quadratic function of the four corners of tile $ij$ (see
Appendix A.1 for exact expression). This allows gradients to flow back into
$\mathcal{T}$, letting us learn the parameters of the warp.

In the case of LZ warps, $\mathcal{T}_{\mathrm{LZ}}$ has no closed form
inverse to the best of our knowledge. Because
$\mathcal{T}_{\mathrm{LZ}}[\mathrm{Grid}(h,w)]$ has no foldovers [20],
$\widetilde{\mathcal{T}}_{\mathrm{LZ}}$ must be injective, implying its
inverse $\widetilde{\mathcal{T}}_{\mathrm{LZ}}^{-1}$ is well-defined.

When applying an LZ warp, saliency can be learned (with trainable parameters)
or unlearned (with fixed parameters), and fixed (invariant across frames) or
adaptive (changes every frame). Adaptive saliency maps require efficient warp
inversion since a different warp must be applied on every input. Learned
saliency maps require differentiability. We note that fixed unlearned saliency
maps do not technically require efficiency or differentiability, and most of
our current results show that such saliency maps are already quite effective,
outperforming prior work. We posit that LZU would shine even more in the
learned adaptive setting, where it could make use of temporal priming for top-
down saliency.

###  4.2 Learning to Zoom and Unzoom

In the Learning to Zoom and Unzoom (LZU) framework, we use existing LZ
downsamplers (see Section 3.2) to "zoom" in on the input image, compute
spatial features, and then use our warp inversion formulation to "unzoom" and
revert any deformations in the feature map, as shown in Figure 1. This
framework is applicable to all tasks with 2D spatial input and all models with
some intermediate 2D spatial representation.

Notice that a poorly approximated inverse warp $\widetilde{\mathcal{T}}^{-1}$
would lead to misaligned features and a drop in performance. As a result, we
use the approximate forward warp $\widetilde{\mathcal{T}}$ instead of the true
forward warp $\mathcal{T}$, so that the composition of forward and inverse
warps is actually the identity function. See Appendix A.3 for a discussion of
the associated tradeoff.

Figure 5: Inverting each axis of a separable warp. LZU first evaluates the
forward warp $\mathcal{T}_{\mathrm{LZ,sep,x}}$ (solid blue arrows) at a
uniform grid of target locations (blue points). The resulting source locations
are shown as red points. LZU then approximates the warp in between these
samples via a linear transform; this piecewise linear map is
$\widetilde{\mathcal{T}}_{\mathrm{LZ,sep,x}}$ (dotted blue arrows). To
evaluate the inverse $\widetilde{\mathcal{T}}^{-1}_{\mathrm{LZ,sep,x}}$
(dotted green arrows), we must determine for each green point which red points
it falls between and invert the corresponding linear transform. An example is
shown in the top-right.

To maintain favorable accuracy-latency tradeoffs, we make several
optimizations to our forward and inverse warps. As done in previous works [11,
20, 22], for the forward warp or "zoom," instead of computing
$\mathcal{T}_{\mathrm{LZ}}[\mathrm{Grid}(H^{\prime},W^{\prime})]$, we compute
$\mathcal{T}_{\mathrm{LZ}}[\mathrm{Grid}(h,w)]$ for smaller $h\ll H^{\prime}$
and $w\ll W^{\prime}$ and bilinearly upsample this to get
$\widetilde{\mathcal{T}}_{\mathrm{LZ}}[\mathrm{Grid}(H^{\prime},W^{\prime})]$.
This also reduces the complexity of computing the inverse, by reducing the
number of cases in our piecewise bilinear map from $H^{\prime}\cdot
W^{\prime}$ to $h\cdot w$.

We explore efficient implementations of both separable and nonseparable warp
inversion, but we find experimentally that nonseparable warps perform no
better than separable warps for a strictly higher latency cost, so we use
separable warps for our experiments. Details for efficiently inverting
nonseparable warps are given in Appendix A.2. For separable warps
$\mathcal{T}_{\mathrm{LZ,sep}}$, we invert each axis separately and take the
Cartesian Product:

$\displaystyle\widetilde{\mathcal{T}}_{\mathrm{LZ,sep}}^{-1}[\mathrm{Grid}$
$\displaystyle(H^{\prime},W^{\prime})]=$ (7)
$\displaystyle\widetilde{\mathcal{T}}_{\mathrm{LZ,sep,x}}^{-1}[\mathrm{Grid}(H^{\prime})]\times\widetilde{\mathcal{T}}_{\mathrm{LZ,sep,y}}^{-1}[\mathrm{Grid}(W^{\prime})].$

This further reduces our problem from inverting a piecewise bilinear map with
$h\cdot w$ pieces to inverting two piecewise _linear_ maps with $h$ and $w$
pieces each. Figure 5 visualizes how to invert each axis.

When unwarping after feature pyramid networks (FPNs) [16], we may have to
evaluate the inverse $\widetilde{\mathcal{T}}_{\mathrm{LZ}}^{-1}$ at multiple
resolutions $\mathrm{Grid}(H^{\prime},W^{\prime})$,
$\mathrm{Grid}(H^{\prime}/2,W^{\prime}/2)$, etc. In practice, we evaluate
$\widetilde{\mathcal{T}}_{\mathrm{LZ}}^{-1}[\mathrm{Grid}(H^{\prime},W^{\prime})]$
and then approximate the inverse at lower resolutions via bilinear
downsampling. This is surprisingly effective (see Appendix A.3) and leads to
no observable loss in performance.

Finally, as introduced in [22], we can also use a fixed warp to exploit
dataset-wide spatial priors, such as how objects are concentrated around the
horizon in many autonomous driving datasets. This allows us to cache forward
and inverse warps, greatly reducing additional latency.

##  5 Experiments

First, we compare LZU to naive uniform downsampling and previous works on the
tasks of 2D object detection and semantic segmentation. We include ablations
to evaluate the effectiveness of training techniques and explore the
upsampling regime. Then, we evaluate LZU on monocular 3D object detection, a
task which no previous works have applied "zooming" to. We perform all timing
experiments with a batch size of 1 on a single RTX 2080 Ti GPU. Figure 6
contains qualitative results and analysis across all tasks. Full
implementation details and hyperparameters are given in Appendix A.6.

###  5.1 2D Object Detection

Scale | Method | AP |  AP50 |  AP75 |  APS |  APM |  APL | Lat (ms)
---|---|---|---|---|---|---|---|---
0.25x | Uniform | 10.5 | 18.0 | 9.9 | 0.3 | 5.2 | 38.6 | 23.3
0.25x | LZU, fixed | 12.4 | 22.6 | 11.2 | 1.0 | 7.1 | 39.2 | 23.6
0.25x | LZU, adaptive | 12.3 | 22.8 | 11.3 | 1.4 | 6.6 | 38.0 | 26.4
0.5x | Uniform | 22.6 | 38.7 | 21.7 | 3.7 | 22.1 | 53.1 | 36.0
0.5x |  FOVEA [22] | 24.9 | 40.3 | 25.3 | 7.1 | 27.7 | 50.6 | 37.9
0.5x | LZU, fixed | 25.2 | 42.1 | 24.8 | 5.5 | 26.7 | 51.8 | 36.4
0.5x | LZU, adaptive | 25.3 | 43.0 | 24.6 | 6.1 | 25.9 | 52.6 | 39.3
0.5x | LZU, adaptive | 22.8 | 39.3 | 22.3 | 5.1 | 22.7 | 48.9 | 39.3
| w/o cascade sal. |  |  |  |  |  |  |
0.75x | Uniform | 29.5 | 48.4 | 29.6 | 9.1 | 32.4 | 55.1 | 62.9
0.75x | LZU, fixed | 30.8 | 50.4 | 31.8 | 10.9 | 33.5 | 54.1 | 63.5
0.75x | LZU, adaptive | 26.5 | 44.6 | 26.7 | 8.3 | 28.7 | 48.7 | 66.3
1x | Uniform | 31.9 | 51.5 | 33.1 | 11.4 | 35.9 | 54.5 | 98.3
1x | LZU, fixed | 32.6 | 52.8 | 34.0 | 13.2 | 36.0 | 54.7 | 99.3
1x | LZU, adaptive | 32.0 | 52.4 | 33.1 | 12.5 | 36.3 | 52.9 | 102.0

Table 1: 2D object detection results of RetinaNet [17] on Argoverse-HD [14].
Fixed LZU uses a dataset-wide spatial prior, and adaptive LZU uses a temporal
prior based on previous frame detections. LZU consistently outperforms the
uniform downsampling baseline and prior work across all scales, with
additional latency less than $4$ms. We hypothesize that the drop in APL is
because objects that are already large benefit less from zooming. Still, this
drawback is offset by larger improvements on small and medium objects.

2D Object Detection  Uniform Resampling LZU Resampling From From To 0.25x 0.5x
0.75x 1x To 0.25x 0.5x 0.75x 1x 0.25x 10.5 10.5 10.5 10.5 0.25x 11.7 12.4 12.4
12.4 0.5x 17.0 22.6 22.6 22.6 0.5x 20.9 24.8 24.8 25.2 0.75x 23.5 28.5 29.5
29.5 0.75x 22.5 29.4 30.0 30.8 1x 13.5 28.4 30.9 31.9 1x 22.1 30.7 31.2 32.6
Monocular 3D Object Detection  Uniform Resampling LZU Resampling From From To
0.25x 0.5x 0.75x 1x To 0.25x 0.5x 0.75x 1x 0.25x 21.8 21.8 21.8 21.8 0.25x
22.5 23.5 23.4 23.4 0.5x 25.4 27.5 27.5 27.5 0.5x 27.0 29.2 29.1 29.3 0.75x
27.6 30.3 30.5 30.5 0.75x 29.0 31.6 31.6 31.8 1x 28.4 30.7 31.1 31.2 1x 30.1
32.5 32.7 32.6

Table 2: 2D and 3D object detection results in the upsampling and downsampling
regimes, using the "Uniform" and "LZU, fixed" models from Tables 1 and 5. LZU
is surprisingly effective even in the upsampling regime! This demonstrates
that simply allocating more pixels to small objects (without retaining extra
information) can help performance, suggesting that detectors still struggle
with scale invariance for small objects. Figure 6:  Examples of the success
and failure cases of LZU. Rows A and E show examples where zooming in on the
horizon helps the detector pick up smaller objects. On the other hand,
sometimes zooming leads to false negatives, such as the black car in Row B and
objects near the edge in Row F. For segmentation, LZU consistently improves
quality near the center of the image. The last column shows the saliency map
used in each case and the resulting spatial magnification ratios. For the
Argoverse-HD [14] dataset, the magnification ratio at the center is nearly
$2$x, meaning the "zoom" is preserving nearly all information in that region,
at the cost of information at the corners.

For 2D object detection, we evaluate LZU using RetinaNet [17] (with a
ResNet-50 backbone [9] and FPN [16]) on Argoverse-HD [14], an object detection
dataset for autonomous driving with high resolution $1920\times 1200$ videos.
For our baselines, we compare to uniform downsampling and FOVEA [22], a
previous work that applies LZ downsampling to detection by unwarping bounding
boxes. We keep the same hyperparameters and setup as in FOVEA [22].
Experiments are run at $0.25$x, $0.5$x, $0.75$x, and $1$x scales, to measure
the accuracy-latency tradeoff.

Our LZU models "unzoom" the feature map at each level after the FPN [16]. We
adopt the low-cost saliency generators introduced in [22] — a "fixed" saliency
map exploiting dataset-wide spatial priors, and an "adaptive" saliency map
exploiting temporal priors by zooming in on detections from the previous
frame. When training the adaptive version, we simulate previous detections by
jittering the ground truth for the first two epochs. For the last epoch, we
jitter detections on the current frame to better simulate previous detections;
we call this "cascaded" saliency. To determine saliency hyperparameters, we
run grid search at $0.5$x scale on splits of the training set (details in
Appendix A.4, A.6). We use a learning rate of $0.01$ and keep all other
training settings identical to the baseline. Latency is measured by timing
only the additional operations (the "zoom" and "unzoom") and adding it to the
baseline. This is done to mitigate the impact of variance in the latency of
the backbone and detector head.

Figure 7:  Plotting the accuracy-latency/FLOPs tradeoffs reveals the Pareto optimal methods for each task. Fixed LZU is Pareto optimal for both 2D and 3D object detection, outperforming uniform downsampling and FOVEA [22]. For semantic segmentation, we use FLOPs in lieu of latency to enable fair comparisons (ES [19] only reports FLOPS and LDS [11] has an unoptimized implementation). Although LDS boasts large improvements in raw accuracy at each scale, it also incurs a greater cost due to its expensive saliency generator. Overall, the Pareto frontier for segmentation is very competitive, with ES dominating at $64\times 64$, LDS at $128\times 128$, and LZU at $256\times 256$.  |  |  | IOU |
---|---|---|---|---
Crop Size | Method | mIOU | road | swalk | build. | wall | fence | pole | tlight | sign | veg. | terr. | sky | person | rider | car | truck | bus | train | mbike | bike |  Latency (ms)
64 | Uniform | 26.4 | 93.9 | 35.6 | 68.6 | 3.5 | 2.9 | 0.5 | 0.0 | 0.1 | 72.5 | 21.1 | 76.0 | 26.8 | 0.9 | 57.1 | 8.9 | 16.9 | 8.0 | 0.0 | 8.2 | 15.5
64 | LZU, fixed | 26.7 | 93.4 | 36.1 | 68.9 | 5.8 | 2.3 | 0.4 | 0.0 | 0.0 | 72.6 | 23.4 | 75.9 | 29.4 | 1.2 | 56.7 | 15.0 | 10.2 | 4.1 | 0.0 | 11.7 | 16.9
128 | Uniform | 39.3 | 96.3 | 54.0 | 78.4 | 15.0 | 7.9 | 8.1 | 8.5 | 16.6 | 81.2 | 34.4 | 86.7 | 42.9 | 13.8 | 74.4 | 22.9 | 41.6 | 24.4 | 10.2 | 29.6 | 16.1
128 | LZU, fixed | 41.7 | 96.4 | 55.2 | 78.7 | 12.7 | 13.4 | 8.1 | 11.4 | 19.0 | 81.7 | 39.0 | 86.5 | 45.7 | 17.9 | 76.8 | 21.9 | 48.2 | 31.7 | 11.6 | 36.3 | 18.0
256 | Uniform | 53.6 | 97.5 | 64.0 | 84.7 | 20.0 | 19.0 | 22.1 | 34.8 | 41.6 | 87.0 | 41.9 | 91.2 | 59.3 | 33.7 | 84.1 | 39.2 | 62.9 | 57.9 | 27.7 | 49.1 | 19.1
256 | LZU, fixed | 55.1 | 97.7 | 67.0 | 84.9 | 24.4 | 24.4 | 21.3 | 35.2 | 42.9 | 87.0 | 44.5 | 90.7 | 61.5 | 35.7 | 85.7 | 40.8 | 67.9 | 52.8 | 29.3 | 53.4 | 21.2
512 | Uniform | 63.8 | 98.3 | 73.3 | 88.8 | 29.2 | 34.3 | 40.6 | 54.4 | 61.6 | 90.7 | 47.7 | 94.0 | 72.7 | 50.6 | 89.1 | 45.6 | 72.1 | 59.1 | 44.5 | 64.9 | 32.3
512 | LZU, fixed | 64.2 | 98.3 | 73.4 | 88.6 | 30.0 | 35.7 | 38.8 | 56.0 | 63.8 | 90.4 | 47.0 | 93.4 | 72.4 | 43.9 | 90.1 | 50.5 | 76.4 | 59.6 | 45.4 | 65.3 | 34.4
Table 3: Full semantic segmentation results of PSPNet [29] on Cityscapes [7].
At each resolution, LZU outperforms uniform downsampling.

Results are given in Table 1. We outperform both uniform downsampling and
FOVEA in all but one case, while incurring an additional latency of less than
$4$ms. The one exception is adaptive LZU at $0.75$x, which is evidence that
our adaptive saliency hyperparameters, chosen at $0.5$x scale, struggle to
generalize to other resolutions. We also confirm that using cascaded saliency
to train adaptive LZU is crucial. Although adaptive LZU outperforms fixed LZU
at $0.5$x scale, plotting the accuracy-latency curves (Figure 7) reveals that
fixed LZU is Pareto optimal at all points.

Finally, we explore how LZU performs in the upsampling regime. We reuse the
same models trained in our previous experiments, testing them with different
pre-resampling resolutions. Results are shown in Table 2. In this regime, LZU
consistently outperforms uniform downsampling, even though information
retention is no longer a factor.

###  5.2 Semantic Segmentation

For our semantic segmentation experiments, we compare to previous works ES
[19] and LDS [11], so we adopt their setup. We test the PSPNet [29] model
(with a ResNet-50 backbone [9] and FPN [16]) on Cityscapes [7]. Cityscapes is
an urban scene dataset with high resolution $1024\times 2048$ images and $19$
classes. We perform our experiments at several image scales ($64\times 64$,
$128\times 128$, $256\times 256$, and $512\times 512$), taken by resizing a
centered square crop of the input image. Our simple baseline trains and tests
PSPNet with uniform downsampling. To reduce overfitting, we allot 500 images
from the official training set into a mini-validation split. We train our
model on the remaining training images and evaluate at 10 equally spaced
intervals on the mini-validation split. We choose the best performing model
and evaluate it on the official validation set.

For our LZU model, we unzoom spatial features after the FPN and use a fixed
saliency map. Inspired by the idea of zooming on semantic boundaries [19], we
generate our fixed saliency by averaging the ground truth semantic boundaries
over the train set. Notably, our saliency hyperparameters are chosen
qualitatively (for producing a reasonably strong warp) and tested one-shot.

We report our full results in Table 3 and compare to previous works in Table
4. Since our baseline results are slightly different than reported in previous
works [19, 11], we compare results using a percent change relative to the
corresponding baseline. We find increased performance over the baseline at all
scales, and at $256\times 256$, we beat both previous works with only $2.3$ms
of additional latency. Plotting the accuracy-FLOPs tradeoff (Figure 7) reveals
that the large improvements of LDS [11] at $64\times 64$ and $128\times 128$
input scales come at significant cost in FLOPs. In actuality, ES [19] is
Pareto optimal at $64\times 64$ and $128\times 128$, LDS [11] at $128\times
128$, and LZU at $256\times 256$. We hypothesize that further improvements
might be possible using an adaptive, learned formulation for saliency.

###  5.3 Monocular 3D Object Detection

| Downsampled Resolution
---|---
Method | $64\times 64$ | $128\times 128$ | $256\times 256$
Uniform (theirs) | 29 | 40 | 54
Uniform (ours) | 26.4 | 39.3 | 53.6
ES [19] | 32 | (+10.3%) | 43 | (+7.5%) | 54 | (+0.0%)
LDS [11] | 36 |  (+24.1%) | 47 |  (+17.5%) | 55 | (+1.9%)
LZU, fixed | 26.7 | (+1.1%) | 41.7 | (+6.1%) | 55.1 |  (+2.9%)
Table 4: Semantic segmentation results of PSPNet [29] on Cityscapes [7], in
mIOU. Due to differing implementation, the performance of our baseline varies
from reported values, so we report relative improvements. At $256\times 256$,
we outperform prior works. At $64\times 64$ and $128\times 128$, LZU performs
worse than prior work, perhaps because "unzooming" features at such small
scales is more destructive. We posit the performance losses from such
aggressive downsampling factors (across all methods) may be too impractical
for deployment, and so focus on the $256\times 256$ regime.

Scale | Method | NDS | mAP | mATE | mASE | mAOE | mAVE | mAAE | Lat (ms)
---|---|---|---|---|---|---|---|---|---
0.25x | Uniform | 21.8 | 11.4 | 96.7 | 32.6 | 90.1 | 125.0 | 19.8 | 54.7
0.25x | LZU, fixed | 23.4 | 13.1 | 96.8 | 31.9 | 82.7 | 129.4 | 20.0 | 55.1
0.5x | Uniform | 27.5 | 17.5 | 90.1 | 28.8 | 75.5 | 131.6 | 17.8 | 58.1
0.5x | LZU, fixed | 29.3 | 20.1 | 88.9 | 28.3 | 73.9 | 130.6 | 16.7 | 58.5
0.75x | Uniform | 30.5 | 21.0 | 87.3 | 27.9 | 67.0 | 132.8 | 17.5 | 59.2
0.75x | LZU, fixed | 31.8 | 22.4 | 83.8 | 27.5 | 67.2 | 134.6 | 15.9 | 59.7
1x | Uniform | 31.2 | 22.4 | 84.2 | 27.4 | 70.9 | 129.6 | 17.4 | 88.7
1x | LZU, fixed | 32.6 | 24.8 | 84.6 | 27.5 | 68.2 | 131.6 | 18.3 | 89.4

Table 5:  3D object detection results of FCOS3D [26] on nuScenes [2]. Higher
NDS and mAP is better, and lower is better on all other metrics. Intuitively,
size is an important cue for depth, and image deformations would stifle this
signal. Suprisingly, this is not the case. LZU improves upon the uniform
downsampling baseline at all scales with less than $1$ms of additional
latency. Notably, LZU at $0.75$x scale even outperforms uniform downsampling
at $1$x.

Finally, we evaluate LZU on monocular 3D object detection. To the best of our
knowledge, no previous work has applied LZ downsampling to this task. The
closest existing solution, FOVEA [22], cannot be extended to 3D object
detection, because 3D bounding boxes are amodal and cannot be unwarped in the
same manner as 2D bounding boxes. For our base model, we use FCOS3D [26], a
fully convolutional model, with a ResNet-50 backbone [9] and FPN [16]. For our
dataset, we use nuScenes [2], an autonomous driving dataset with multi-view
$1600\times 900$ RGB images for 1000 scenes and 3D bounding box annotations
for 10 object classes. As is standard practice, we use the nuScenes Detection
Score (NDS) metric, which is a combination of the usual mAP and measures of
translation error (mATE), scale error (mASE), orientation error (mAOE),
velocity error (mAVE), and attribute error (mAAE). We run experiments at
$0.25$x, $0.5$x, $0.75$x, and $1$x scales and test against a uniform
downsampling baseline. We train for 12 epochs with a batch size of 16 with
default parameters as in MMDetection3D [5].

For our LZU model, again we unzoom post-FPN features and use a fixed saliency
map. Inspired by FOVEA [22], our fixed saliency is generated by using kernel
density estimation on the set of projected bounding boxes in the image space.
We reuse the same saliency hyperparameters from 2D detection. All other
training settings are identical to the baseline.

Results are given in Table 5. LZU performs consistently better than uniform
downsampling, with less than $1$ms of additional latency. Specifically, LZU
improves mAP and the aggregate metric NDS, with mixed results on mATE, mASE,
mAOE, mAVE, and mAAE. Since the latter five metrics are computed on only true
positives, this demonstrates that LZU increases overall recall, while
maintaining about equal performance on true positives. Plotting the accuracy-
latency curves (Figure 7) shows that LZU is Pareto optimal. We also repeat the
same upsampling experiments as performed in 2D object detection. Results,
shown in Table 2, reaffirm the viability of LZU in the upsampling regime.

##  6 Conclusion

We propose LZU, a simple attentional framework consisting of "zooming" in on
the input image, computing spatial features, and "unzooming" to invert any
deformations. To unzoom, we approximate the forward warp as a piecewise
bilinear mapping and invert each piece. LZU is highly general and can be
applied to any task with 2D spatial input and any model with 2D spatial
features. We demonstrate the versatility of LZU empirically on a variety of
tasks and datasets, including monocular 3D detection which has never been done
before. We also show that LZU may even be used when high-resolution sensor
data is unavailable. For future work, we can consider alternatives to the
"unzoom" formulation that are perhaps less destructive than simple resampling
of features.

Broader impact. Our work focuses on increasing the efficiency and accuracy of
flagship vision tasks (detection, segmentation, 3D understanding) with high-
resolution imagery. We share the same potential harms of the underlying tasks,
but our approach may increase privacy concerns as identifiable information may
be easier to decode at higher resolutions (e.g., facial identities or license
plates). Because our approach is agnostic to the underlying model, it is
reproducible with minimal changes to existing codebases.

Acknowledgements: This work was supported by the CMU Argo AI Center for
Autonomous Vehicle Research.

## References

  * [1] Thaddeus Beier and Shawn Neely.  Feature-based image metamorphosis.  ACM SIGGRAPH computer graphics, 26(2):35–42, 1992.
  * [2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * [3] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [4] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang.  A survey of model compression and acceleration for deep neural networks.  arXiv preprint arXiv:1710.09282, 2017.
  * [5] MMDetection3D Contributors.  MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * [6] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [10] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  Advances in neural information processing systems, 28, 2015.
  * [11] Chen Jin, Ryutaro Tanno, Thomy Mertzanidou, Eleftheria Panagiotaki, and Daniel C Alexander.  Learning to downsample for segmentation of ultra-high resolution images.  arXiv preprint arXiv:2109.11071, 2021.
  * [12] Shu Kong and Charless Fowlkes.  Pixel-wise attentional gating for scene parsing.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1024–1033. IEEE, 2019.
  * [13] Jun Li, Yongjun Chen, Lei Cai, Ian Davidson, and Shuiwang Ji.  Dense transformer networks.  arXiv preprint arXiv:1705.08881, 2017.
  * [14] Mengtian Li, Yu-Xiong Wang, and Deva Ramanan.  Towards streaming perception.  In European Conference on Computer Vision, pages 473–488. Springer, 2020.
  * [15] Mengtian Li, Ersin Yumer, and Deva Ramanan.  Budgeted training: Rethinking deep neural network training under resource constraints.  arXiv preprint arXiv:1905.04753, 2019.
  * [16] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [17] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [19] Dmitrii Marin, Zijian He, Peter Vajda, Priyam Chatterjee, Sam Tsai, Fei Yang, and Yuri Boykov.  Efficient segmentation: Learning downsampling near semantic boundaries.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2131–2141, 2019.
  * [20] Adria Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba.  Learning to zoom: a saliency-based sampling layer for neural networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 51–66, 2018.
  * [21] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [22] Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan.  Fovea: Foveated image magnification for autonomous navigation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15539–15548, 2021.
  * [23] Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao.  Improving the speed of neural networks on cpus.  In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 2011.
  * [24] Thomas Verelst and Tinne Tuytelaars.  Dynamic convolutions: Exploiting spatial sparsity for faster inference.  In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 2320–2329, 2020.
  * [25] Thomas Verelst and Tinne Tuytelaars.  Segblocks: Block-based dynamic resolution networks for real-time segmentation.  arXiv preprint arXiv:2011.12025, 2020.
  * [26] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  Fcos3d: Fully convolutional one-stage monocular 3d object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913–922, 2021.
  * [27] George Wolberg.  Digital image warping, volume 10662.  IEEE computer society press Los Alamitos, CA, 1990.
  * [28] Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, and Stephen Lin.  Spatially adaptive inference with stochastic feature sampling and interpolation.  In European conference on computer vision, pages 531–548. Springer, 2020.
  * [29] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [30] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9308–9316, 2019.

##  Appendix A Appendix

###  A.1 Bilinear Transformations

Figure 8:  Geometric interpretation of bilinear transformations. Suppose we
have such a transformation from the unit square to an arbitrary quadrilateral.
Given coordinates $(u,v)$ in the unit square, if we draw lines in the
quadrilateral such that
$u=\frac{|{\mathbf{x}}_{b}{\mathbf{x}}_{bl}|}{|{\mathbf{x}}_{br}{\mathbf{x}}_{bl}|}=\frac{|{\mathbf{x}}_{t}{\mathbf{x}}_{tl}|}{|{\mathbf{x}}_{tr}{\mathbf{x}}_{tl}|}$
and
$v=\frac{|{\mathbf{x}}_{l}{\mathbf{x}}_{bl}|}{|{\mathbf{x}}_{tl}{\mathbf{x}}_{bl}|}=\frac{|{\mathbf{x}}_{r}{\mathbf{x}}_{br}|}{|{\mathbf{x}}_{tr}{\mathbf{x}}_{br}|}$,
they will intersect at $\mathrm{BilinearTransformation}(u,v)$.

Our construction in Section 4.1 assumes prior knowledge of bilinear
transformations. Bilinear transformations have actually been widely studied in
the context of computer graphics [27]. Here, for unfamiliar readers, we
outline its definition and inverse formulation.

For simplicity, consider a bilinear transformation that maps the unit square
to the quadrilateral with corners
${\mathbf{x}}_{bl},{\mathbf{x}}_{br},{\mathbf{x}}_{tl},{\mathbf{x}}_{tr}$.
True to its name, the transformation is defined within the square via bilinear
interpolation:

$\displaystyle\mathrm{BilinearTransformation}(u,v)$
$\displaystyle={\mathbf{x}}_{bl}+({\mathbf{x}}_{br}-{\mathbf{x}}_{bl})u$
$\displaystyle+({\mathbf{x}}_{tl}-{\mathbf{x}}_{bl})v+({\mathbf{x}}_{tr}-$
$\displaystyle{\mathbf{x}}_{br}-{\mathbf{x}}_{tl}+{\mathbf{x}}_{bl})uv.$ (8)

Interestingly, this transformation also has a geometric interpretation, shown
in Figure 8.

Now, consider the inverse of this mapping. Given a point $(x,y)$ in the
quadrilateral, we want to find the point $(u,v)$ in the unit square that maps
to it. A full derivation is given in [27], but if we define the following
scalars

$\displaystyle(a_{0},b_{0})$ $\displaystyle={\mathbf{x}}_{bl}$ (9)
$\displaystyle(a_{1},b_{1})$
$\displaystyle={\mathbf{x}}_{br}-{\mathbf{x}}_{bl}$ (10)
$\displaystyle(a_{2},b_{2})$
$\displaystyle={\mathbf{x}}_{tl}-{\mathbf{x}}_{bl}$ (11)
$\displaystyle(a_{3},b_{3})$
$\displaystyle={\mathbf{x}}_{tr}-{\mathbf{x}}_{br}-{\mathbf{x}}_{tl}+{\mathbf{x}}_{bl}$
(12) $\displaystyle c_{0}$ $\displaystyle=a_{1}(b_{0}-y)+b_{1}(x-a_{0})$ (13)
$\displaystyle c_{1}$
$\displaystyle=a_{3}(b_{0}-y)+b_{3}(x-a_{0})+a_{2}b_{1}-a_{2}b_{1}$ (14)
$\displaystyle c_{2}$ $\displaystyle=a_{3}b_{2}-a_{2}b_{3},$ (15)

then the solution $(u,v)$ must satisfy

$c_{2}v^{2}+c_{1}v+c_{0}=0$ (16)

and

$u=\frac{x-a_{0}-a_{2}v}{a_{1}+a_{3}v}.$ (17)

Applying the quadratic formula on Equation 16, we can solve for $v$. Then, we
can substitute into Equation 17 to find $u$. Given a point $(x,y)$ in the
quadrilateral, this will produce exactly one pair of solutions $(u,v)$ in the
unit square (there may be extraneous solutions with $u$ or $v$ negative or
greater than $1$).

Although these results assume a mapping from the unit square, they extend
naturally to our use case. Recall from Section 4.1 that
$\mathrm{\widetilde{\mathcal{T}}}_{ij}$ is a bilinear transformation from
rectangle $R_{ij}$ to quadrilateral $\mathcal{T}[R_{ij}]$. We can apply all
previous results, simply by normalizing the coordinates within $R_{ij}$.

###  A.2 Efficient Inversion of Nonseparable Warps

Figure 9:  Unzooming in the nonseparable case. $\mathcal{T}_{\mathrm{LZ}}$ is
approximated as $\widetilde{\mathcal{T}}_{\mathrm{LZ}}$, which is a
$(h-1)\times(w-1)$ tiling of bilinear transformations (left). We wish to
compute $\widetilde{\mathcal{T}}_{\mathrm{LZ}}^{-1}({\mathbf{x}})$ at each
green point ${\mathbf{x}}\in\mathrm{Grid}(H^{\prime\prime},W^{\prime\prime})$,
where $H^{\prime\prime}\times W^{\prime\prime}$ is the desired output size
(middle). For the $ij$-th tile, we consider the set of all candidate green
point ${\mathbf{x}}$ within the enclosing blue box (right) and apply the
corresponding inverse bilinear transformation. We set
$\widetilde{\mathcal{T}}_{\mathrm{LZ}}^{-1}({\mathbf{x}})=\mathrm{BilinearTransformation}_{ij}^{-1}({\mathbf{x}})$
only if it falls in the $ij$-th grid rectangle $R(i,j)$.

In Section 4.2, we detail how to efficiently invert separable zooms
$\mathcal{T}_{\mathrm{LZ,sep}}$. To invert nonseparable zooms
$\mathcal{T}_{\mathrm{LZ}}$, it no longer suffices to invert each axis. We
must instead reason in the full 2D space.

Suppose we have a nonseparable zoom $\mathcal{T}_{\mathrm{LZ}}$. We compute
$\mathcal{T}_{\mathrm{LZ}}[\mathrm{Grid}(h,w)]$ for small $h,w$ and use this
to approximate the forward zoom as $\widetilde{\mathcal{T}}_{\mathrm{LZ}}$, an
$(h-1)\times(w-1)$ piecewise tiling of bilinear maps. Now, to unzoom to a
desired output resolution of $H^{\prime\prime}\times W^{\prime\prime}$, we
must evaluate
$\widetilde{\mathcal{T}}_{\mathrm{LZ}}[\mathrm{Grid}(H^{\prime\prime},W^{\prime\prime})]$.
That is, for each
${\mathbf{x}}\in\mathrm{Grid}(H^{\prime\prime},W^{\prime\prime})$, we must
determine which of the $(h-1)(w-1)$ quadrilateral pieces it falls in and apply
the corresponding inverse bilinear map. Recall from Appendix A.1 that applying
an inverse bilinear map amounts to solving a quadratic.

In our actual implementation, we parallelize operations as much as possible.
For the $ij$-th tile, instead of first determining which points
${\mathbf{x}}\in\mathrm{Grid}(H^{\prime\prime},W^{\prime\prime})$ are inside
of it and then applying the $ij$-th inverse bilinear map, we implement it the
other way around. We consider a set of candidate interior points, apply the
$ij$-th inverse bilinear map to all of them, and keep only those with a valid
solution. The candidate points are those falling inside the axis-aligned
rectangle enclosing that tile. The full procedure is described in Algorithm 1
and visualized in Figure 9.

Our implementation takes about $12.6$ms to invert a nonseparable warp with
$(h,w)=(31,51)$ and an output shape
$(H^{\prime\prime},W^{\prime\prime})=(600,960)$, as in our Argoverse-HD [14]
experiments. While this is not fast enough to support favorable accuracy-
latency tradeoffs, we believe that further optimization (_e.g_. using custom
CUDA operations) may change this.

Algorithm 1 Inverting nonseparable zooms $\mathcal{T}_{\mathrm{LZ}}$.
In practice, we make the following optimizations. We vectorize the loop on
line 13. We also fix $B_{ij}$ to be the max size over choices of $(i,j)$,
allowing us to implement the loop on line 4 using batch-processing.

1:$\triangleright$ See Appendix A.2 for the algorithm setup and meaning of
variables.

2:function
Unzoom($\mathcal{T}_{\mathrm{LZ}}[\mathrm{Grid}(h,w)],(H^{\prime\prime},W^{\prime\prime})$)

3: Initialize $\mathcal{T}_{\mathrm{LZ}}^{-1}({\mathbf{x}})=(0,0)$ for all
${\mathbf{x}}\in\mathrm{Grid}(H^{\prime\prime},W^{\prime\prime})$

4: for $(i,j)\in[h-1]\times[w-1]$ do

5: $\triangleright$ corners of $R(i,j)$

6:
${\mathbf{x}}_{tl}^{\prime},{\mathbf{x}}_{tr}^{\prime}=\left(\frac{i}{h-1},\frac{j-1}{w-1}\right),\left(\frac{i}{h-1},\frac{j}{w-1}\right)$

7:
${\mathbf{x}}_{bl}^{\prime},{\mathbf{x}}_{br}^{\prime}=\left(\frac{i}{h-1},\frac{j-1}{w-1}\right),\left(\frac{i}{h-1},\frac{j}{w-1}\right)$

8: $\triangleright$ corners of $ij$-th quadrilateral tile

9:
${\mathbf{x}}_{tl},{\mathbf{x}}_{tr}=\mathcal{T}_{\mathrm{LZ}}^{-1}\left({\mathbf{x}}_{tl}^{\prime}\right),\mathcal{T}_{\mathrm{LZ}}^{-1}\left({\mathbf{x}}_{tr}^{\prime}\right)$

10:
${\mathbf{x}}_{bl},{\mathbf{x}}_{br}=\mathcal{T}_{\mathrm{LZ}}^{-1}\left({\mathbf{x}}_{bl}^{\prime}\right),\mathcal{T}_{\mathrm{LZ}}^{-1}\left({\mathbf{x}}_{br}^{\prime}\right)$

11: $\triangleright$ top-left and bottom-right corners of rectangle

12: $\triangleright$ enclosing the quadrilateral tile

13:
$\mathbf{c}_{tl}=\min({\mathbf{x}}_{tl},{\mathbf{x}}_{tr},{\mathbf{x}}_{bl},{\mathbf{x}}_{br})$

14:
$\mathbf{c}_{br}=\max({\mathbf{x}}_{tl},{\mathbf{x}}_{tr},{\mathbf{x}}_{bl},{\mathbf{x}}_{br})$

15: $\triangleright$ set of all candidate points in the $ij$-th tile

16:
$B_{ij}=\left\\{{\mathbf{x}}\in\mathrm{Grid}(H^{\prime\prime},W^{\prime\prime}):\mathbf{c}_{tl}\leq{\mathbf{x}}\leq\mathbf{c}_{br}\right\\}$

17: for ${\mathbf{x}}\in B_{ij}$ do

18:
${\mathbf{x}}^{\prime}=\mathrm{BilinearTransformation}^{-1}_{ij}({\mathbf{x}})$

19: if
${\mathbf{x}}_{tl}^{\prime}\leq{\mathbf{x}}^{\prime}\leq{\mathbf{x}}_{br}^{\prime}$
then

20: $\mathcal{T}_{\mathrm{LZ}}^{-1}({\mathbf{x}})={\mathbf{x}}^{\prime}$

21: return $\mathcal{T}^{-1}_{\mathrm{LZ}}$

###  A.3 Analysis of our Warping Approximations

We make two approximations in our formulation. First, to ensure that the
composition of forward and inverse warps is truly the identity function, we
use the approximate forward warp $\widetilde{\mathcal{T}}$ in place of the
true forward warp $\mathcal{T}$. This trades latency for how well
$\widetilde{\mathcal{T}}$ zooms in on the intended regions of interest. See
Figure 10 for a visualization of this effect.

Figure 10: Quality of the approximate forward warp $\widetilde{\mathcal{T}}$
as we decrease the dimensions $h\times w$ of our piecewise bilinear
approximation. $\widetilde{\mathcal{T}}$ degrades noticeably once $h,w$
decrease beyond a certain threshold.

Second, we use bilinear downsampling to approximate the inverse at lower
resolutions after feature pyramid networks [16]. This approximation is
surprisingly effective! For our fixed LZU model on 2D detection, we calculate
that the error between $\widetilde{\mathcal{T}}^{-1}$ and the doubly
approximated inverse $\widehat{\mathcal{T}}_{d}^{-1}$ given by

$\underset{{\mathbf{x}}\in\mathrm{Grid}(H^{\prime}/d,W^{\prime}/d)}{\mathrm{avg}}\left\|\widetilde{\mathcal{T}}^{-1}({\mathbf{x}})-\widehat{\mathcal{T}}_{d}^{-1}({\mathbf{x}})\right\|_{2}$
(18)

is only $0.0274$, $0.0065$, $0.0028$ pixels at $d=2$, $4$, and $8$!

###  A.4 Sensitivity to Saliency

Fixed LZU is quite robust to choice of saliency. Details on how we computed
our saliency maps are given in Appendix A.6. For 2D detection, we performed a
grid search at $0.5$x scale to determine the saliency hyperparameters. The
results show that LZU is quite robust to choice of saliency, as long as the
warp is not too strong (see Table 6). For semantic segmentation and 3D
detection, we chose saliency one-shot, which suggests that these fixed LZU
models are also robust to choice of saliency.

Our grid search suggests that adaptive LZU is also robust to choice of
hyperparameters than fixed LZU. However, these saliency hyperparameters,
chosen at $0.5$x scale, struggle to generalize to $0.75$x and $1$x scale (see
Table 1). As a result, for adaptive warps, it may be necessary to tune
saliency at each scale.

LZU, fixed  Attraction Kernel fwhm Amp. 4 10 16 22 1 34.6 34.8 35.4 35.7 5
34.5 34.7 32.6 34.6 10 34.8 34.4 32.3 34.5 50 35.0 33.5 31.4 33.7 100 35.0
33.8 31.3 33.2

LZU, adaptive  Attraction Kernel fwhm Amp. 4 10 16 22 1 34.9 36 35.6 35.5 5
34.3 35.1 33.3 35.1 10 33.4 34.3 32.2 34.0 50 33.8 33.2 32.3 34.2 100 34.0
33.6 32.3 33.7

Table 6: Grid search over hyperparameters for 2D object detection (see
Appendix A.6) shows that LZU is quite robust to choice of saliency. For
comparison, uniform downsampling yields an AP of $32.2$.

###  A.5 Additional Results

Figure 11: Visualizations of the per-pixel accuracy difference between the
uniform and the LZU models at each resolution. The largest gains occur at the
central horizontal strip, largely corresponding with the saliency map used for
segmentation. However, performance also decreases at the top and bottom of the
image. Thus, using a fixed saliency map must be conscious decision to trade
performance at low saliency regions for high saliency regions. We hypothesize
that adaptive saliency maps could lead to more uniform gains.

In Figure 11, we plot the spatial accuracy of our CityScapes [7] models.

###  A.6 Implementation Details

Our experiments are implemented using open-source libraries MMDetection [3],
MMSegmentation [6], and MMDetection3D [5], all released under the Apache 2.0
License. We use GeForce RTX 2080 Ti’s for training and training, which takes
at most $5$ GPU-days for any given model, but the precise amount varies by
model and task. We perform all timing experiments with a batch size of 1 on a
single GPU.

#### Argoverse-HD [14] Detection

As done in FOVEA [22], for our uniform downsampling experiments, we finetune a
COCO-pretrained model for 3 epochs with the random left-right image flip
augmentation, a batch size of 8, momentum SGD with a learning rate of 0.005,
momentum of 0.9, weight decay of 1e-4, learning rate warmup for 1000
iterations, and a per-iteration linear learning rate decay [15].

For both LZU models, we use a learning rate of $0.01$ and keep all other
hyperparameters identical to the baseline. To "zoom", we use a $31\times 51$
saliency map and the separable anti-cropping formulation
$\mathcal{T}_{\mathrm{LZ,sep,ac}}$ (as proposed in [22] and discussed in
Section 3.2).

For the fixed saliency LZU model, we use Gaussian distance kernels $k_{x}$ and
$k_{y}$ of full-width-half-maximum (fwhm) $22$. To generate the fixed saliency
map, we use kernel density estimation (KDE) on all training bounding boxes
with hyperparameters amplitude $a=1$ and bandwidth $b=64$. For details on the
effects of $a$ and $b$, refer to [22].

For the adaptive saliency LZU model, we use Gaussian distance kernels $k_{x}$
and $k_{y}$ of fwhm 10. To generate adaptive saliency, we use KDE on
detections from the previous frame with $a=1$ and $b=64$. When training, to
simulate motion, we jitter bounding boxes by $\mathcal{N}(0,7.5)$ pixels
horizontally and $\mathcal{N}(0,3)$ pixels vertically.

For each LZU experiment, we run a grid search at $0.5$x scale over
separable/nonseparable, amplitude $a=1,5,10,50,100$, and distance kernel’s
$\text{fwhm}=4,10,16,22$ to determine optimal settings. This is done using an
$80/20$ split of the train set, so as to not overfit on the real validation
set. We generate this split such that locations between splits are disjoint.
All other hyperparameters are chosen one-shot.

#### Cityscapes [7] Segmentation

We train the uniform downsampling baseline using mostly the default
hyperparameters from MMSegmentation [6]. The only changes are to the data
augmentation pipeline and the evaluation frequency. Comprehensively, we train
with just the photometric distortion augmentation (random adjustments in
brightness, contrast, saturation, and hue), a batch size of 16, momentum SGD
with a learning rate of 0.01, momentum of 0.9, weight decay of 5e-4, and a
polynomial learning rate schedule with power 0.9. To account for overfitting,
we validate our performance at 10 equally spaced intervals on a 500-image
subset of the training dataset (and train on the others). For all experiments,
we train for 80K iterations, with the exception of $64\times 64$, which we
train for 20K iterations due to rapid overfitting.

For the fixed LZU model, we use the same training hyperparameters as the
baseline. To "zoom", we use the separable anti-cropping formulation
$\mathcal{T}_{\mathrm{LZ,sep,ac}}$ with a $45\times 45$ saliency map and
Gaussian distance kernels $k_{x}$, $k_{y}$ of fwhm $15$. To generate the fixed
saliency, we aggregate ground truth semantic boundaries over the train set.
Precisely, we define boundaries to be pixels which differ from at least one of
its eight neighbors. We compute semantic boundaries for each $256\times 256$
ground truth segmentation, assign boundaries an intensity of $200$ and
background an intensity of $1$, and average pool down to a $45\times 45$
saliency map. The semantic boundary intensity value was chosen qualitatively
(for producing a reasonably strong warp) and tested one-shot.

#### nuScenes [2] 3D Detection

We train the uniform downsampling baseline using all default hyperparameters
from MMDetection3D [5], except the learning rate, which we reduce for
stability. Specifically, we train for 12 epochs with the random left-right
flip augmentation, a batch size of 16, momentum SGD with a learning rate of
0.001, momentum of 0.9, weight decay of 1e-4, doubled learning rate on bias
parameters with no weight decay, L2 gradient clipping, a step learning rate
schedule with drops at epochs 8 and 11, and a linear learning rate warmup for
the first 500 iterations.

For the fixed LZU model, we use the same training hyperparameters as the
baseline. To "zoom", we use the separable anti-cropping formulation
$\mathcal{T}_{\mathrm{LZ,sep,ac}}$ with a $27\times 48$ saliency map and
Gaussian distance kernels $k_{x}$, $k_{y}$ of fwhm $10$. To generate the fixed
saliency, we project 3D bounding boxes into the image plane and reuse the same
KDE formulation with the same hyperparameters ($a=1$ and $b=64$) as used in 2D
detection. These are all chosen and evaluated one-shot.