Content-Adaptive Downsampling in Convolutional Neural Networks
Many convolutional neural networks (CNNs) rely on progressive downsampling of
their feature maps to increase the network's receptive field and decrease
computational cost. However, this comes at the price of losing granularity in
the feature maps, limiting the ability to correctly understand images or
recover fine detail in dense prediction tasks. To address this, common practice
is to replace the last few downsampling operations in a CNN with dilated
convolutions, allowing to retain the feature map resolution without reducing
the receptive field, albeit increasing the computational cost. This allows to
trade off predictive performance against cost, depending on the output feature
resolution. By either regularly downsampling or not downsampling the entire
feature map, existing work implicitly treats all regions of the input image and
subsequent feature maps as equally important, which generally does not hold. We
propose an adaptive downsampling scheme that generalizes the above idea by
allowing to process informative regions at a higher resolution than less
informative ones. In a variety of experiments, we demonstrate the versatility
of our adaptive downsampling strategy and empirically show that it improves the
cost-accuracy trade-off of various established CNNs.
Robin Hesse, Simone Schaub-Meyer, Stefan Roth
# Content-Adaptive Downsampling in Convolutional Neural Networks

###### Abstract

Many convolutional neural networks (CNNs) rely on progressive downsampling of their feature maps to increase the network's receptive field and decrease computational cost. However, this comes at the price of losing granularity in the feature maps, limiting the ability to correctly understand images or recover fine detail in dense prediction tasks. To address this, common practice is to replace the last few downsampling operations in a CNN with dilated convolutions, allowing to retain the feature map resolution without reducing the receptive field, albeit increasing the computational cost. This allows to trade off predictive performance against cost, depending on the output feature resolution. By either regularly downsampling or not downsampling the entire feature map, existing work implicitly treats all regions of the input image and subsequent feature maps as equally important, which generally does not hold. We propose an adaptive downsampling scheme that generalizes the above idea by allowing to process informative regions at a higher resolution than less informative ones. In a variety of experiments, we demonstrate the versatility of our adaptive downsampling strategy and empirically show that it improves the cost-accuracy trade-off of various established CNNs.

+
Footnote †: 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

## 1 Introduction

When humans are exposed to a complex task, they focus on relevant aspects to make optimal use of the brain's limited capacities [2]. For instance, when categorizing the bird's species in Fig. 1, our brain allocates significantly more computational resources for the bird than for the background. While naturally occurring in humans, this adaptive allocation of resources is not utilized in most of today's deep learning architectures. In this work, we propose an adaptive downsampling scheme for Convolutional Neural Networks (CNNs) [23] that mimics above adaptive allocation of resources by _simultaneously_ processing different regions of an image or feature map _at different resolutions_.

Many CNNs used as backbones in computer vision rely on a progressive application of pooling or strided convolution [17, 22, 37] to increase the network's receptive field and decrease computational cost [46]. However, this progressive, regular downsampling comes at the price of losing fine detail in the feature maps, limiting the network's ability to correctly understand images [46] or recover fine detail in dense prediction tasks [4], such as at object boundaries. To approach this problem, Yu _et al_. [46] replace the last few downsampling operations in a CNN with dilated convolutions [45]. Although increasing the computational cost, this allows to keep the resolution of feature maps without reducing the network's receptive field. However, by regu

Figure 1: _Illustration of our content-adaptive method compared to regular downsampling. (a)_ In regular downsampling, every second pixel is sampled (repeated twice here). Note how the result is almost in recognizable and important detail like the beak, claws, and tail of the bird is lost. (b) In our adaptive downsampling, a precomputed downsampling mask defines the number of downsampling operations applied to each pixel. The resulting representation (zoom in to avoid aliasing) contains information of locally varying resolution. This allows to semi-continuously adjust the representation size to keep more detail where it matters.
larly downsampling, respectively _not_ downsampling [46], _entire_ feature maps, most existing CNNs implicitly assume that all regions of the input image and subsequent feature maps are equally important, which generally does not hold [21, 29, 31] as, _e.g_., also shown by locally salient attribution maps [36, 39, 18].

In contrast, our _locally adaptive_ downsampling scheme makes a more realistic assumption and can be considered as a generalization of both regular downsampling in CNNs and dilated convolutions [46]. By processing task-relevant regions at a higher resolution than unimportant ones, we can maintain the fine granularity of important regions while efficiently processing unimportant regions at a lower resolution, ultimately leading to an improved trade-off between the accuracy and computational expense of various popular CNNs. Contrary to existing adaptive downsampling methods [21, 29] that adaptively downsample the input image _before_ passing it through a CNN, our approach allows for adaptive downsampling of feature maps _within_ a CNN.

Specifically, we make the following contributions: _(1)_ We present a novel adaptive downsampling scheme for CNNs, allowing to process different feature map regions at different resolutions. _(2)_ As different amounts of computational resources are being allocated for different regions, our method can be considered as a novel realization of "focus" in CNNs. This is fundamentally different from existing attention mechanisms, _e.g_., in transformers [8], where attention is implemented via adaptive weighting of input features. As a consequence, an uninformative black image and an informative natural image would still require the same inference time in a transformer, while the black image would be processed faster in our proposed adaptive downsampling. _(3)_ We provide an accompanying modified instantiation of submanifold sparse convolution [14] that allows to efficiently perform standard convolution on multi-resolution grids that occur in our adaptive downsampling. _(4)_ Thanks to carefully designing the proposed method to satisfy certain guarantees, it can be used in a plug-and-play fashion within existing backbones even without retraining, making it exceptionally versatile and practical. _(5)_ We further empirically show with two computer vision tasks that our method is an effective and application-agnostic tool to improve the cost-accuracy trade-off of different established CNNs that build on regular downsampling.

## 2 Related Work

**Adaptive (down)sampling.** The goal of adaptive sampling approaches is to sample an image or feature map spatially to obtain optimal performance or properties of a CNN. An early instance are Spatial Transformer Networks [19], which aim to increase invariance to different geometric transformations by learning to spatially transform feature maps. Deformable convolutions [7] use two-dimensional offsets to the regular sampling locations of standard convolutions to allow for more flexible handling of geometric variations and different receptive field sizes within one layer. CF-ViT [5] entails a multi-stage approach where in each stage the input patch resolution of the most important patches is increased to refine the result of a vision transformer. Talebi and Milanfar [41] show that resizing images with learned resizers, instead of linear ones, can improve the accuracy of recognition models. Recasens _et al_. [31] use an auxiliary saliency network to estimate the most important image regions that are then used for non-uniformly downsampling the input image, leading to an amplification of salient regions. Marin _et al_. [29] train an auxiliary network to predict a non-uniform sampling grid that is denser near semantic boundaries and use it to non-uniformly downsample input images to retain fine detail for segmentation tasks. Jin _et al_. [21] propose a similar idea as [29], particularly for ultra-high-resolution images, additionally including the segmentation accuracy in the training objective of the sampling grid estimator. Our proposed approach is fundamentally different from the above methods that adaptively downsample the image _before_ passing it through a CNN [21, 29, 31], instead of _within_ the CNN. As a result, they severely sacrifice accuracy, making them primarily appropriate for very high-resolution images.

**Detail-preserving pooling.** While above adaptive downsampling methods are concerned with the question of _how many_ features to sample per region, detail-preserving pooling is concerned with the question of _which_ features or feature combinations to sample to better retain important detail. Mixed Pooling [44] randomly selects max or average pooling. Lee _et al_. [24] learn a weighted combination of max and average pooling that is dependent on the pooling region. In \(L_{p}\) pooling [15], orders \(p_{j}\) are learned to interpolate between different pooling operators with \(p_{j}=1\) corresponding to average pooling and \(p_{j}=\infty\) to max pooling. Detail-preserving pooling [35] is a learnable adaptive pooling method that magnifies important detail, making use of inverse bilateral filters. Local importance-based pooling [11] preserves discriminative detail by training an auxiliary network to predict adaptive importance maps that are used to aggregate features for downsampling. SoftPool [38] minimizes information loss using a softmax-weighted sum of feature activations. Note that adaptive downsampling methods [21, 31], including our approach, also incorporate regular pooling layers, and thus, advanced pooling operations such as the above can be used complementarily.

**Layer aggregation and architecture.** Besides the above, one can also use early high-resolution feature maps of CNNs and more complex network architectures to improve the granularity of feature maps or the output. Hypercolumns [16] build a feature vector for any pixel by concatenating activations from all feature maps above that pixel.
Fully convolutional networks [26] refine semantic segmentations by combining earlier layers of higher resolution with later layers of lower resolution. The U-Net [34] architecture extends the previous idea by introducing skip connections between all layers of corresponding resolutions. Pinheiro [30] propose to gradually refine the output segmentation by adding information from earlier layers in a top-down fashion. Feature pyramid networks [25] produce feature pyramids by combining high-level and low-level features of a CNN. Instead of simple one-step skip-connections, Yu [47] incorporate more depth and sharing in their layer aggregation. HRNet [43] goes even further and processes high-resolution and low-resolution streams in parallel while repeatedly exchanging information between the streams. Yu [46] improve the granularity of feature maps by substituting downsampling operations with dilated convolutions [45], allowing to retain the feature map resolution while keeping the original receptive field of the CNN. Generally, high-level feature maps yield semantically stronger features [17, 25], and thus, using auxiliary information from low-level layers might not be optimal. Further, combining layers of different resolutions or keeping higher resolutions can potentially introduce new parameters, increase computational cost, and/or raise memory usage.

**Sparse convolution.** While sparse convolution is not directly related to adaptive downsampling, it plays an essential role in our method. Initial works on sparse convolution [10, 12, 13, 33] improve the computational cost of standard convolution by processing only active elements in sparse inputs, such as point clouds [10], handwritten digits [12], or 3D grids [33]. Submanifold sparse convolution [14] avoids a growing number of active elements caused by standard convolutions that dilate the sparse data with each layer. Contrary to sparse convolution, which is only used to completely ignore inactive elements, we adapt sparse convolution to work with our proposed multi-resolution feature maps.

## 3 Content-adaptive Downsampling in CNNs

In this work, we postulate that not all feature map regions of a CNN are equally important and that, therefore, different regions should be processed at _different resolutions_. By processing only a subset of the most important feature map regions at higher resolution while downsampling less important ones, we can combine the smaller representation size of regular downsampling with the higher feature map granularity of dilated convolutions [46]. As a consequence, we can gain relatively large improvements in predictive performance while only moderately increasing the computational cost compared to regular downsampling.

The high-level idea behind our approach is illustrated in Fig. 1(b). Contrary to regular downsampling (Fig. 1(a)), where the image or feature map is downsampled uniformly by sampling every second pixel, we utilize an adaptive downsampling scheme (Fig. 1(b)) that retains higher resolution in areas with fine detail while downsampling less informative ones. In the following, we will outline the details of our method with a focus on two-dimensional feature maps with channels,, in CNNs for vision tasks. However, the extension to other dimensionalities, such as 1D signals and 3D temporal or volumetric data, works analogously.

**Regular downsampling.** Let \(f\in\mathbb{R}^{H\times W\times C}\) be a feature map (or image) of height \(H\), width \(W\), and depth \(C\). We denote regular downsampling by a factor of \(d\in\mathbb{N}_{>1}\) as the process of reducing the spatial dimension of \(f\) by outputting one element \(e\in\mathbb{R}^{C}\) from each non-overlapping \(d\times d\) patch, such that the resulting feature map \(f_{d}\) is of shape \(\frac{H}{d}\times\frac{W}{d}\times C\). Generally, regular downsampling \(\psi\colon\mathbb{R}^{H\times W\times C}\mapsto\mathbb{R}^{H/d\times W/d\times C}\) in CNNs is realized by applying a patch-wise downsampling function \(\psi^{\prime}\colon\mathbb{R}^{d\times d\times C}\mapsto\mathbb{R}^{C}\) to all \(HW/d^{2}\) non-overlapping \(d\times d\) patches \(f^{\prime}_{i,j}\) in \(f\):

\[f_{d}=\psi\left(f\right)=\left(\psi^{\prime}(f^{\prime}_{1,1}),\ldots,\psi^{ \prime}(f^{\prime}_{H/d,W/d})\right). \tag{1}\]

The above definition of regular downsampling generalizes to many commonly used downsampling methods used in CNNs. For example, \(\psi^{\prime}\) can take the form of various pooling operations, [15, 35, 44], or it can select an element based on its spatial location to implement uniform downsampling,, sampling every \(d\)-th element. When following a convolution of stride 1, this corresponds conceptually to a strided convolution with a stride of \(d\).

**Adaptive downsampling.** In this work, we generalize the above regular downsampling scheme by allowing some patches \(f^{\prime}_{i,j}\) to _not be downsampled_, and therefore, retaining their _original resolution_ of \(d\times d\). This is accomplished by providing an additional downsampling mask \(m\in\{0,1\}^{H/d\times W/d}\) as input, specifying which patches \(f^{\prime}_{i,j}\) should be downsampled (\(m_{i,j}=1\)) and which not (\(m_{i,j}=0\)). For now, we assume that this downsampling mask is given, and go later into more detail about how to compute it in a content-adaptive way. Formally, our novel adaptive downsampling \(\psi_{a}\) is defined as

\[\psi_{a}\left(f,m\right) =\left(\psi^{\prime}_{a}\big{(}f^{\prime}_{1,1}),\ldots,\psi^{ \prime}_{a}(f^{\prime}_{H/d,W/d})\right), \tag{2}\] \[\text{with }\psi^{\prime}_{a}\big{(}f^{\prime}_{i,j}\big{)}= \begin{cases}f^{\prime}_{i,j}&\text{if }m_{i,j}=0\\ \psi^{\prime}(f^{\prime}_{i,j})&\text{if }m_{i,j}=1\,,\end{cases}\]

denoting a patch-wise adaptive downsampling as indicated by the downsampling mask \(m\).

**Multi-resolution grid convolution.** As the resulting feature map \(\psi_{a}\left(f,m\right)\) consists of multiple resolutions, we lose the regular grid structure of our data necessary to work with standard convolutional layers. To address this, we project
the lower-resolution features onto their corresponding location in the high-resolution grid, as illustrated in Fig. 2(c). Naturally, high-resolution patches \(f^{\prime}_{i,j}\) that are filled with a lower-resolution element will have empty cells (Fig. 2(c) - shaded cells) and, thus, are sparse. We exploit the resulting sparsity by processing our feature representations in subsequent layers using sparse convolution [10, 13, 33]. This allows to only consider _active_ elements, and hence, uses less computation in regions of lower effective resolution. To avoid submanifold dilation [14], _i.e._, that inactive elements become active after convolution, we suggest a modified instance of submanifold sparse convolution [14] (SSC). SSC ensures that input and output contain the same set of active elements and thus enables us to keep the resolution of each respective feature map or image region unchanged. To retain the same receptive field as the convolutional layer would have with regular downsampling, we increase the dilation factor to \(d\). A remaining challenge is that at specific feature map locations (see, _e.g._, Fig. 2(c) - cell with \(6\)) non-central elements of the convolutional kernel could fall onto inactive elements, causing the result to be corrupted. To avoid this, we modify SSC to assume that all inactive elements take the value of the corresponding active element within their patch (shown in Fig. 2(c) as shaded cells).

Extension to CNNs.Similarly to Yu _et al_. [46], our proposed adaptive downsampling is incorporated into a standard CNN with regular downsampling, _i.e._, pooling or strided convolution, by substituting the last \(n\) downsampling operations with adaptive downsampling. Further, we substitute all convolutional layers that follow the \(i\in\{1,\dots,n\}\) adaptive downsampling operations with our proposed sparse convolution with a dilation factor of \(d^{i}\) to keep the same receptive field as the original network. To perform multiple adaptive downsampling operations in succession, we only consider the elements belonging to the currently lowest available resolution in our multi-resolution feature map and convert them into a dense representation. We then perform adaptive downsampling on these elements according to Eq. (2). As before, the resulting features are projected to their corresponding location in the multi-resolution feature map. Consequently, patches already containing higher-resolution elements are excluded from the downsampling process. This implies that areas, where resolution is retained, cannot be downsampled at a later downsampling step, and thus, for \(d=2\), one obtains a quadtree-like resolution pattern as seen in Fig. 1(b). A more detailed explanation of this process can be found in the supplementary.

Guarantees.The resulting CNN can further be seen as a generalization of both regular CNNs with downsampling as well as of the dilated convolution of Yu _et al_. [46]. When setting all elements of the downsampling mask to one, we obtain a standard CNN with regular downsampling. When setting all elements of the downsampling mask to zero, we obtain exactly dilated convolution [46] for higher-resolution feature maps. By simultaneously processing different feature map regions at _different_ resolutions, controlled by the downsampling mask, we establish a combination of the above methods that allows us to "interpolate" between different output strides in a _locally adaptive_ fashion. An important desideratum behind our approach is that - like dilated convolution [46] - additional training of the backbone is not mandatory. Therefore, it is exceptionally easy to use

Figure 2: _Illustration of different methods to handle feature map resolution in CNNs. \(\star\)_ denotes the two-dimensional cross-correlation operator (w/ zero padding). _(a)_ Dilated convolution increases the receptive field (equally to downsampling) while retaining the feature resolution [46]. _(b)_ Standard strided convolution in CNNs (with stride=2) corresponds to uniformly downsampling the feature map by sampling every second element. This is typically followed by a standard convolution. _(c)_ In our adaptive scheme, the downsampling mask is used to downsample a subset of the feature map and low-resolution entries are projected into the high-resolution grid; shaded cells (hatched) denote inactive elements. Afterward, sparse convolution is applied to the active elements (black font) of the multi-resolution feature map to produce an output with the same, multiple resolutions as the input. Note how the (loop) invariant of all algorithms (Guarantee 2) ensures that the elements highlighted in blue, both before and after convolution, are the same across the different methods.
in a plug-and-play fashion. This is possible due to the following two guarantees, which result from our design and hold when substituting regular downsampling, _i.e_., strided convolution or pooling, with our adaptive downsampling:

**Guarantee 1:** Due to projecting all elements into the highest-resolution grid and adjusting the dilation factors of subsequent sparse convolutions, we can guarantee that the receptive fields of the CNN and each intermediate layer remain unchanged compared to the corresponding CNN with regular downsampling.
**Guarantee 2:** All output features of a standard CNN with strided convolution equal the output features at the corresponding high-resolution locations of the same CNN with adaptive downsampling (see Fig. 2(c) - blue cells).

This means that a CNN with adaptive downsampling is not modifying feature map values of a standard CNN with strided convolution, but instead only _adds_ information at pixel locations where there was none before, and therefore, refines the feature map. Note that while Guarantee 2 formally only holds for strided convolutions and not for pooling, we show in Sec. 4.3 that applying our adaptive downsampling in a CNN with pooling is still feasible in practice without necessarily requiring retraining.

**Content-adaptive downsampling masks.** A remaining question is how to obtain the downsampling mask to indicate what regions to process at which resolution in a locally content-adaptive way. In Sec. 4, we show that traditional algorithms, like high-frequency detection or keypoint estimates, can serve as an effective basis to identify important image regions, and thus, to estimate downsampling masks.

Additionally, we demonstrate how to learn an appropriate downsampling mask from data. Specifically, we utilize a shallow CNN that takes as input the feature map of the main model before the adaptive downsampling step, and outputs the downsampling mask. In order to train it end-to-end with the main model, we make the discrete mask estimation differentiable with Gumbel-Softmax [20]. As training our adaptive downsampling end-to-end can result in a downsampling mask of only zeros, _i.e_., full resolution, we control the amount of invested resources with an additional hyperparameter \(\gamma\in[0,1]\) that defines the desired proportion of active mask elements. Its squared difference to the actual proportion \(\hat{m}\in[0,1]\) of active elements in our downsampling mask is included in the final loss function

\[\mathcal{L}=\alpha\mathcal{L}_{1}+\beta(\gamma-\hat{m})^{2}, \tag{3}\]

with \(\alpha\) and \(\beta\) being weighting factors, and \(\mathcal{L}_{1}\) denoting the loss of the main task, _e.g_., the segmentation loss.

Limitations.Our approach allows to interpolate between different feature map resolutions, which can drastically improve the cost-accuracy trade-off as shown in Sec. 4. However, some limitations arise that could hinder its effective usage in some applications. First, our method's benefit depends on the given downsampling mask. If the mask does not capture important regions, retaining higher-resolution regions will not help. Second, there may be backbones, especially those with advanced layer aggregation schemes [43, 47], or datasets without fine details where higher-resolution feature maps and, therefore, also our method generally do not yield advantages. However, our analysis and related work [9, 46] show that the highly impactful and widely used CNN backbones, VGG [37] and ResNet [17], benefit from higher-resolution feature representations in various applications.

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Different backbones with DeepLabv3 [3]**} \\ \cline{2-10}  & \multicolumn{3}{c}{ResNet-50 [17]} & \multicolumn{3}{c}{ResNet-101 [17]} & \multicolumn{3}{c}{ResNet-152 [17]} \\ \cline{2-10} OS & \(16\) & \(8\) & \(8{\rightarrow}32\) & \(16\) & \(8\) & \(8{\rightarrow}32\) & \(16\) & \(8\) & \(8{\rightarrow}32\) \\ \hline mIoU & \(0.7496\) & \(0.7655\) & \(0.7658\) & \(0.7591\) & \(0.7775\) & \(0.7748\) & \(0.7683\) & \(0.7852\) & \(0.7848\) \\ \#MA & \(2.6e11\) & \(8.0e11\) & \(3.8e11\) & \(4.2e11\) & \(1.4e12\) & \(6.7e11\) & \(5.7e11\) & \(1.9e12\) & \(1.0e12\) \\ Time & \(0.05\) & \(0.11\) & \(0.05/0.08\) & \(0.07\) & \(0.22\) & \(0.08/0.15\) & \(0.1\) & \(0.31\) & \(0.11/0.25\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Oracle experiment for semantic segmentation._ For each model and output stride (OS), we report the mean IoU (mIoU) on the Cityscapes [6] evaluation split, the theoretical (average) number of multiply-adds (#MA), and the min./max. inference times in seconds. Our adaptive downsampling (OS=\(8{\rightarrow}32\)) improves the cost-accuracy trade-off of all models, indicated by the on par mIoU scores and the significantly reduced #MA and inference times, compared to OS=8.
## 4 Experiments

To demonstrate the practicality of our proposed adaptive downsampling scheme, we conduct various experiments on two different computer vision tasks confirming the following points: _(1)_ As a generalization of [46] and standard CNNs, our method allows to interpolate between different output strides, enabling a more granular control of how many resources should be allocated for the task and input image at hand. _(2)_ By selecting task-relevant regions for a specific input, our method can drastically improve the cost-accuracy trade-off of standard CNNs. _(3)_ The guarantees satisfied by our method allow to incorporate it in a plug-and-play fashion into pre-trained networks at test time. _(4)_ Our method is not limited to a single application and finding appropriate downsampling masks is feasible in practice. _(5)_ Our approach generalizes to different backbones, segmentation heads, and extensions.

We start our evaluation with an oracle experiment, demonstrating the potential of our method given a high-quality downsampling mask. Afterward, we present two case studies that show how our method can be used in practice, demonstrating the advantages summarized above.

**Experimental setup (segmentation).** Experiments for semantic segmentation (Secs. 4.1 and 4.2) have been conducted on the Cityscapes dataset [6]. We report the mean intersection over union (mIoU) on the validation set.

**Experimental setup (keypoint description).** For the keypoint description experiment (Sec. 4.3), we use the established D2-Net descriptor [9], which takes an image as input and outputs a dense feature map that can be used to locate and describe keypoints. Since D2-Net keypoint localization tends to be imprecise [42], we follow Uzpak _et al_. [42] and instead use SIFT [27] to first detect the 512 most salient keypoints and employ D2-Net only to describe them. Following common practice [28, 32], we report the mean matching accuracy, _i.e_., the ratio between correct and possible matches, at a three-pixel threshold (MMA@3) on the HPatches dataset [1].

**Experimental setup (general).** As the focus of this work is the cost-accuracy trade-off of the examined methods, for all experiments we report the respective task-specific metrics, _i.e_., mIoU and MMA@3, over the required theoretical (average) number of multiply-adds as a proxy of computational cost. As baselines, we consider the original models with varying output strides obtained by using regular downsampling [17, 37] with the standard factor \(d=2\), respectively _not_ downsampling and using dilated convolutions [46]. To draw a more conclusive picture, we also report the required inference time in seconds. However, the inference time depends on the used implementation and hardware - here an Nvidia RTX A6000 GPU. For a fair comparison we use the same implementation for our method and the baselines; please refer to the supplementary for different implementations.

### Oracle experiment: Feature resolution in semantic segmentation

In our first experiment, we examine the role of feature map resolution in semantic segmentation and whether processing more informative regions at a higher resolution offers advantages and is generally possible. To this end, we first conduct an oracle experiment in which we assume a high-quality downsampling mask to be given. As baselines we use a segmentation model with regular downsampling and two different output strides (OS) of 16 (lower feature resolution) and 8 (higher feature resolution). For our proposed adaptive downsampling with output stride \(8\) to \(32\) (OS=\(8\)\(\rightarrow\)\(32\)), we define the downsampling mask as the pixels that are misclassified by a regular model with OS=\(32\) but correctly classified by a model with OS=\(8\), followed by a dilation with a square kernel size of \(K\times K\) (see Fig. 3 and supplementary). This gives us exactly the image regions that benefit from higher resolutions, and thus, can be considered our oracle downsampling mask. To demonstrate the generalizability of our approach, we investigate a variety of different backbones, segmentation heads, and extensions.

Results for our adaptive downsampling and baselines can be seen in Tab. 1. Confirming our assumption and prior work [4, 46], a smaller output stride, respectively higher resolution, increases both the mIoU and computational cost of all the examined baseline models (OS=\(8\)_vs_. OS=\(16\)). Moreover, using our adaptive downsampling leads to an mIoU that is on par with the respective regular model with OS=\(8\) while for some images requiring less than 50% of the time and computational cost (_e.g_., ResNet-101+DeepLabv3). This clearly confirms our hypothesis that only a fraction of the feature map must be processed at a higher resolution to obtain strong predictive performance.

Figure 3: _Example downsampling masks._ A Cityscapes [6] image and the corresponding estimated downsampling masks using our oracle setup, edge detection, and a shallow learned network. Note how low-frequency regions, _e.g_., the sky and road, belong to the same class, and thus, can be processed at low resolution (bright) to decrease the computational cost.
Further, it shows that our proposed method is applicable to various different ResNet backbones (VGG [37] is shown in Sec. 4.3) and segmentation heads. Also, our approach is a novel and orthogonal research direction, which can be used _together_ with existing extensions like deformable convolution [7] and advanced pooling strategies [38].

### Case study 1: Semantic segmentation

In our first case study, we again consider semantic segmentation with ResNet101+DeepLabv3 models but now with realistically estimated downsampling masks. We propose two strategies to estimate useful downsampling masks:

**Edge mask.** From analyzing the problem at hand, we know that segmentation boundaries are likely to align with edges. For this reason, we perform edge detection on the input image to create a downsampling mask. Specifically, we apply a Sobel filter to the original grayscale images and dilate the thresholded result with a square kernel of size \(11\times 11\). Computational cost for detecting the edges is neglectable (\(3\times 10^{-3}\%\) of the total multiply-adds).

**Learned mask.** As described in Sec. 3, we use a shallow CNN to estimate downsampling masks from an intermediate layer of the feature extractor. The mask estimator is trained end-to-end with the segmentation model (see supplementary). The computational cost for the mask estimator is included in the reported multiply-adds and times.

The resulting downsampling masks for a single example image can be seen in Fig. 3. Dark areas denote keeping the higher resolution of an output stride of 8 while bright areas correspond to an output stride of 16 (OS=\(8{\rightarrow}16\)). In Fig. 4, we report results for our adaptive downsampling with the two proposed strategies. We again observe that by "focusing" on more relevant image regions, _e.g_., edges, we can improve the cost-accuracy trade-off of an established model. Further, by adjusting the hyperparameters of the mask estimators, we can granularly control the amount of invested resources. Remarkably, adaptive downsampling with a learned mask achieves on par mIoU as regular downsampling with an output stride of 8 (0.775 vs. 0.776) while reducing the required computational cost by approximately 35%. Looking at the annotated minimum and maximum times, we can observe time improvements of up to \(40\%\). However, we observe that for images with a lot of "interesting" content, the worst-case inference time is similar to regular downsampling with OS=8. This case study also demonstrates how naively and easily we can estimate downsampling masks that together with our adaptive downsampling still yield advantages for the task at hand. A visual example of the estimated segmentations is shown in Fig. 5.

Contrary to our work, existing methods that adaptively downsample the input _before_ passing it into a CNN [21, 29] lose important image information, resulting in lower mIoU scores of \(0.5\) or \(0.65\) on Cityscapes [6]. Hence, they are mainly beneficial for processing ultra-high resolution images and are not considered as competitive baselines here.

### Case study 2: Keypoint description

In our second realistic case study, we consider the problem of keypoint description. To this end, we investigate the established SIFT detector [27] and D2-Net descriptor [9] as described in the experimental setup. D2-Net utilizes a VGG16 [37] backbone that progressively applies max pooling to reduce the resolution of the feature map, and thus, computational load. To demonstrate the simplicity and versatility of our method, we use the original model weights provided by the authors [9] and do _not_ perform additional training for any of the following setups. Again confirming prior work [9], in Fig. 6 we see that a higher feature map resolution (_i.e_., lower OS) increases both the MMA@3 and the computational cost of the baseline model.

Next, we substitute the last 3, 2, or 1 max pooling op

Figure 4: _Semantic segmentation results._ mIoU on the Cityscapes [6] evaluation set over the number of multiply-adds for a ResNet-101 [17] backbone with a DeepLabv3 [3] segmentation head, using regular downsampling [46] and our novel adaptive downsampling scheme with a learned mask, respectively an edge detection mask, with varying hyperparameters. Min. and max. inference times for each setup are annotated. To reduce variance, we report the maximum mIoU over three runs. X-confidence intervals show 2 times the standard deviation of the required multiply-adds.

Figure 5: _Qualitative semantic segmentation results._ Estimated segmentation maps using regular downsampling with OS=\(8\) and adaptive downsampling with an output stride of 8 to 16 (OS=\(8{\rightarrow}16\)) and our edge downsampling (DS) mask.
erations with our adaptive downsampling with max pooling (OS=\(\{1,2,4\}\)\(\rightarrow\)8). As only image regions around the keypoints are important for our task at hand, we estimate the downsampling masks by dilating keypoints obtained from SIFT with filters of increasing sizes for each of the up to three downsampling levels (see Fig. 7). We demonstrate a fine-grained control of computational cost by varying the dilation sizes for the downsampling levels. Larger dilation sizes will lead to larger areas being processed at higher resolution, and thus, increase cost as well as MMA@3. The results in Fig. 6 clearly show that, compared to the fixed output strides of the baseline model, our adaptive downsampling yields a significant reduction of the computational cost while achieving the same MMA@3, _e.g._, we achieve an \(\sim\)\(70\%\) reduction of the computational cost to reach an MMA@3 that is on par with a regular output stride of 1.

**Inadequate masks.** To evaluate how our method performs with "bad" downsampling masks, in Fig. 6 we additionally report the MMA@3 for adaptive downsampling with an output stride of \(4\) to \(8\) using random and unreasonable masks, _i.e._, the inverse of reasonable masks. Confirming our stated limitations, we see that inadequate downsampling masks do not capture important image regions, and thus, we process unimportant regions at high resolution, leading to an increased computational cost compared to regular downsampling with OS=\(8\) without yielding significant improvements in predictive performance. Note, however, that thanks to our guarantees in Sec. 3, poor downsampling masks still yield comparable predictive performance as regular downsampling with OS=\(8\), showing that these masks do not negatively affect predictive performance and that the model still behaves in an expected way.

## 5 Conclusion and Discussion

In this work, we propose - to the best of our knowledge - the first downsampling scheme that changes the operating resolution _within_ CNNs in a _locally adaptive_ fashion. We do so by generalizing standard CNNs [17, 22, 37] and Yu _et al._'s [46] dilated convolutional networks, allowing us to process feature maps with spatially-varying resolutions. By selecting an appropriate content-adaptive downsampling mask, indicating locally the most important regions that are to be processed at a higher resolution, we can enable CNNs to "focus" more strongly on task-relevant regions. Besides substantially improving the cost-accuracy trade-off in two computer vision tasks, our novel adaptive downsampling enables a more continuous control of the invested computational resources, giving practitioners another degree of freedom to best adapt their model to the available resources. Thanks to carefully designing the proposed method, it satisfies two important guarantees, allowing adaptive downsampling even to be used at test time in a plug-and-play fashion within standard CNNs pre-trained with _regular_ downsampling. As our approach improves the cost-accuracy trade-off of various established models, it contributes to saving valuable scarce resources and to the important research direction of more (energy) efficient deep learning [40].

**Acknowledgements.** This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 866008). The project has also been supported in part by the State of Hesse through the cluster project "The Adaptive Mind (TAM)" and the Hessian research priority programme LOEWE within the project "WhiteBox".

Figure 6: _Keypoint description results._ MMA@3 over the required number of multiply-adds for a SIFT [27] detector with a pre-trained D2-Net [9] descriptor using regular downsampling [46] with different output strides (OS), and our novel adaptive downsampling scheme using different masks. Given a reasonable downsampling mask, our adaptive downsampling can achieve on par MMA@3 while significantly reducing the computational cost. Actual inference times for a randomly chosen image are annotated.

Figure 7: Qualitative results of keypoints matched with our adaptive downsampling scheme and the corresponding downsampling masks. Note how only a fraction of the image is important, and thus, processed at high resolution (dark) to increase the accuracy while saving resources in bright areas of the downsampling mask.