Deformable kernel networks for guided depth map upsampling
Beomjun Kim Jean Ponce Bumsub Ham








Deformable kernel networks for guided depth map upsampling
    Beomjun Kim

Yonsei University








Jean Ponce

Inria / PSL Research University



Bumsub Ham
 
Yonsei University




    December 30, 2023
============================================================================================================================








We address the problem of upsampling a low-resolution (LR) depth map using a registered high-resolution (HR) color image of the same scene. Previous methods based on convolutional neural networks (CNNs) combine nonlinear activations of spatially-invariant kernels to estimate structural details from LR depth and HR color images, and regress upsampling results directly from the networks. In this paper, we revisit the weighted averaging process that has been widely used to transfer structural details from hand-crafted visual features to LR depth maps. We instead learn explicitly sparse and spatially-variant kernels for this task. To this end, we propose a CNN architecture and its efficient implementation, called the deformable kernel network (DKN), that outputs sparse sets of neighbors and the corresponding weights adaptively for each pixel. We also propose a fast version of DKN (FDKN) that runs about 17 times faster (0.01 seconds for a HR image of size 640 × 480). Experimental results on standard benchmarks demonstrate the effectiveness of our approach. In particular, we show that the weighted averaging process with 3 × 3 kernels (i.e., aggregating 9 samples sparsely chosen) outperforms the state of the art by a significant margin.









§ INTRODUCTION



Acquiring depth information is one of the fundamental tasks in computer vision, for scene recognition <cit.>, pose estimation <cit.> and 3D reconstruction <cit.>, for example. Recent stereo matching methods based on convolutional neural networks (CNNs) <cit.> give high-quality depth maps, but still require a huge computational cost, especially in the case of a large search range. Consumer depth cameras (, the ASUS Xtion Pro <cit.> and the Microsoft Kinect <cit.>), typically coupled with RGB sensors, are practical alternatives to obtain depth maps at low cost. Although they provide dense depth maps, these typically offer limited spatial resolution and depth accuracy. To address this problem, registered high-resolution (HR) color images can be used as guidance to enhance the spatial resolution of low-resolution (LR) depth maps <cit.>. The basic idea behind this approach, called guided or joint image filtering, is to exploit their statistical correlation to transfer structural details from the guidance HR color image to the target LR depth maps, typically by estimating spatially-variant kernels from the guidance. Concretely, given the target image f and the guidance image g, the filtering output f̂ at position p=(x,y) is expressed as a weighted average <cit.>:



    f̂_p = ∑_q∈𝒩(𝓅) W_pq(g)f_q,

where we denote by 𝒩(𝓅) a set of neighbors (defined on a discrete regular grid) near the position p. The filter kernel W is typically a function of the guidance image g <cit.>, normalized so that 



    ∑_q∈𝒩(𝓅) W_pq(g) = 1.
 


Classical approaches to depth map upsampling mainly focus on designing the filter kernels W and the set of neighbors 𝒩 (, sampling locations q). They use hand-crafted kernels and predefined neighbors without learning <cit.>. For example, the guided filter <cit.> uses spatially-variant matting Laplacian kernels <cit.> to encode local structures from the HR color image. These methods use regularly sampled neighbors for aggregating pixels, and do not handle inconsistent structures in the HR color and LR depth images, causing texture-copying artifacts <cit.>. To address the problem, both HR color and LR depth images have been used to extract common structures <cit.>. Recently, learning-based approaches using CNNs <cit.> have also become increasingly popular. The networks are trained using large quantities of data, capturing natural image priors and often outperforming traditional methods by large margins. These methods do not use a weighted averaging process. They combine instead nonlinear activations of spatially-invariant kernels learned from the networks. That is, they approximate spatially-variant kernels by mixing the activations of spatially-invariant ones nonlinearly (, via the ReLU function <cit.>). 

 In this paper, we propose to exploit spatially-variant kernels explicitly to encode the structural details from both HR color and LR depth images as in classical approaches, but learn the kernel weights in a supervised manner. We also learn the set of neighbors, building an adaptive and sparse neighborhood system for each pixel. This also allows sub-pixel information aggregation, which may be difficult to achieve by hand. To implement this idea, we propose a CNN architecture and its efficient implementation, called a deformable kernel network (DKN), for learning sampling locations of the neighboring pixels and their corresponding kernel weights at every pixel. We also propose a fast version of DKN (FDKN), achieving a 17 times speed-up compared to the plain DKN for a HR image of size 640 × 480, while retaining its superior performance. We show that the weighted averaging process, even trained with the LR depth map only without any guidance (, g=f in (<ref>)), with 9 points sparsely sampled, is sufficient to obtain a new state of the art (Fig. <ref>). Our code and models are available online: <https://cvlab-yonsei.github.io/projects/DKN>



Contributions. The main contributions of this paper can be summarized as follows:



  
  ∙ We introduce a novel variant of the classical guided weighted averaging process for depth map upsapling and its implementation, the DKN, that computes the set of neighbors and their corresponding weights adaptively for individual pixels.


  
  ∙ We propose a fast version of DKN (FDKN) that runs about 17 times faster than the DKN while retaining its superior performance.


  
  ∙ We achieve a new state of the art, outperforming all existing methods we are aware of by a large margin, and clearly demonstrating the advantage of our approach to learning both kernel weights and sampling locations. We also provide an extensive experimental analysis to investigate the influence of all the components and parameters of our model. 








§ RELATED WORK



Here we briefly describe the context of our approach, and review representative works related to ours. 



Depth map upsampling.
We categorize depth map upsampling into explicit/implicit weighted-average methods and learning-based ones. First, explicit weighted-average methods compute the output at each pixel by a weighted average of neighboring pixels in the LR depth image, where the weights are estimated from the HR color image <cit.> to transfer fine-grained structures. The bilateral <cit.> and guided <cit.> filters are representative methods that have been successfully adapted to depth map upsampling. They use hand-crafted kernels to estimate the weights, which may transfer erroneous structures to the target image <cit.>. Second, implicit weighted-average methods formulate depth map upsampling as an optimization problem, and minimize an objective function that usually involves fidelity and regularization terms <cit.>. The fidelity term encourages the output to be close to the LR depth image, and the regularization term gives the output having a structure similar to that of the HR color image. Although, unlike explicit ones, implicit weighted-average methods exploit global structures in the HR color image, hand-crafted regularizers may not capture structural priors. Finally, learning-based methods can further be categorized into dictionary- and CNN-based approaches. Dictionary-based methods exploit the relationship between paired LR and HR depth patches, additionally coupled with the HR color image <cit.>. In CNN-based methods <cit.>, an encoder-decoder architecture is typically used to learn features from the HR color and/or LR depth images, and the output is then regressed directly from the network. Other methods <cit.> integrate a variational optimization into CNNs by unrolling the optimization steps of the primal-dual algorithm, which requires two stages in training and a number of iterations in testing. Similar to implicit weighted-average methods, they use hand-crafted regularizers, which may not capture structural priors.  


Our method borrows from both explicit weighted-average methods and CNN-based ones. Unlike existing explicit weighted-average methods <cit.>, that use hand-crafted kernels and neighbors defined on a fixed regular grid, we leverage CNNs to learn the set of sparsely chosen neighbors and their corresponding weights adaptively. Our method differs from previous CNN-based ones <cit.> in that we learn sparse and spatially-variant kernels for each pixel to obtain upsampling results as a weighted average. The bucketing stretch in single image super-resolution <cit.> can be seen as a non-learning-based approach to filter selection. It assigns a single filter by solving a least-squares problem for a set of similar patches (buckets). In contrast, our model learns different filters using CNNs even for similar RGB patches, since we learn them from a set of multi-modal images (., pairs of RGB/D images). 

Variants of the spatial transformer <cit.>.
Recent works introduce more flexible and effective CNN architectures. Jaderberg et al. propose a novel learnable module, the spatial transformer <cit.>, that outputs the parameters of the desired spatial transformation (, affine or thin plate spline) given a feature map or an input image. The spatial transformer makes a standard CNN for classification invariant to a set of geometric transformation, but it has a limited capability of handling local transformations. Most similar to ours are the dynamic filter network <cit.> and its variants (the adaptive convolution network <cit.> and the kernel prediction networks <cit.>), where a set of local transformation parameters is generated adaptively conditioned on the input image. The main differences between our model and these works are two-fold. First, our network is more general in that it is not limited to learning spatially-variant kernels, but it also learns the sampling locations of neighbors. This allows to aggregate sparse but highly related samples only, enabling an efficient implementation in terms of speed and memory and achieving state-of-the-art results even with aggregating 9 samples sparsely chosen. For comparison, the adaptive convolution and kernel prediction networks require lots of samples (, 21 × 21 in <cit.>, 41 × 41 in <cit.>, and 8 × 5 × 5 in <cit.>). As will be seen in our experiments, learning sampling locations of neighbors clearly boosts the performance significantly compared to learning kernel weights only. Second, as other guided image filtering approaches <cit.>, our model is easily adapted to other tasks such as saliency map upsampling, cross-modality image restoration, texture removal, and semantic segmentation. We focus here on depth upsampling but see the supplement for some examples. In contrast, the adaptive convolution network is specialized to video frame interpolation, and kernel prediction networks are applicable to denoising Monte Carlo renderings <cit.> or burst denoising <cit.> only. Our work is also related to the deformable convolutional network <cit.>. The basic idea of deformable convolutions is to add  offsets to the sampling locations defined on a regular grid in standard CNNs. The deformable convolutional network samples features directly from learned offsets, but shares the same weights for different sets of offsets as in standard CNNs. In contrast, we use spatially-variant weights for each sampling location. 








§ APPROACH



In this section, we briefly describe our approach, and present a concrete network architecture. We then describe a fast version of DKN.




 §.§ Overview



Our network mainly consists of two parts (Fig. <ref>): We first learn spatially-variant kernel weights and spatial sampling offsets w.r.t the regular grid. To this end, a two-stream CNN <cit.>, where each sub-network has the same structure (but different parameters), uses the guidance (HR color) and target (LR depth) images to extract features that are used to estimate the kernel weights and the offsets. We then compute a weighted average using the learned kernel weights and sampling locations computed from the offsets to obtain a residual image. Finally, the upsampling result is obtained by combining the residual image with the LR depth map. Note that we can train DKN without the residual connection, by directly computing the upsampling result as a weighted average. Note also that we can train our model without the guidance of the HR color image. In this case, we use a single-stream CNN to extract features from the LR depth map only in both training and testing. Our network is fully convolutional, does not require fixed-size input images, and it is trained end-to-end. 


Weight and offset learning. Dual supervisory information for the weights and offsets is typically not available. We learn instead these parameters by minimizing directly the discrepancy between the output of the network and a reference HR depth map. In particular, constraints on weight and offset regression (sigmoid and mean subtraction layers in Fig. 2) specify how the kernel weights and offsets behave and guide the learning process. For weight regression, we apply a sigmoid layer that makes all elements larger than 0 and smaller than 1. We then subtract the mean value from the output of the sigmoid layer so that the regressed weights should be similar to high-pass filters with kernel weights adding to 0. For offset regression, we do not apply the sigmoid layer, since relative offsets (for x, y positions) from locations on a regular grid can have negative values. 


Residual connection. The main reason behind using a residual connection is that the upsampling result is largely correlated with the LR depth map, and both share low-frequency content <cit.>. Focussing on learning the residuals also accelerates training speed while achieving better performance <cit.>. Note that contrary to <cit.>, we obtain the residuals by a weighted averaging process with the learned kernels, instead of regressing them directly from the network output. Empirically, the kernels learned with the residual connection have the same characteristics as the high-pass filters widely used to extract important structures (, object boundaries) from images (See the supplemental material). 





 §.§ DKN architecture
  


We design a fully convolutional network to learn the kernel weights and the sampling offsets for individual pixels. We show in Table <ref> the detailed description of the network structure.


Feature extraction.
We adapt an architecture similar to <cit.> for feature extraction. It consists of 7 convolutional layers, two of which use convolutions with multiple strides (“DownConv" in Table <ref>), that enlarge a receptive field size with a small number of network parameters to estimate. We input the HR color and LR depth images to each of the sub-networks, resulting in a feature map of size 128 × 1 × 1 for a receptive field of size 51 × 51. The LR depth map is initially upsampled using bicubic interpolation. We use the ReLU <cit.> as an activation function. Batch normalization <cit.> is used for speeding up training and regularization. 

Weight regression.
For each sub-network, we add a 1 × 1 convolutional layer on top of the feature extraction layer. It gives a feature map of size k^2 × 1 × 1, where k is the size of the filter kernel, which is used to regress the kernel weights. To estimate the weights, we apply a sigmoid layer to each feature map of size k^2 × 1 × 1, and then combine the outputs by element-wise multiplication (see Fig. <ref>). We could use a softmax layer as in <cit.>, but empirically find that it does not perform as well as the sigmoid layer. The softmax function encourages the estimated kernel to have only a few non-zero elements, which is not appropriate for estimating the weights for sparsely sampled pixels. The estimated kernels should be similar to high-pass filters, with kernel weights adding to 0. To this end, we subtract the mean value from the combined output of size k^2 × 1 × 1. For our model without a residual connection, we apply instead L1 normalization to the output of size k^2 × 1 × 1. Since the sigmoid layer makes all elements in the combined output larger than 0, applying L1 normalization forces the kernel weights to add to 1 as in (<ref>). 


Offset regression.
Similar to the weight regression case, we add a 1 × 1 convolutional layer on top of the feature extraction layer. The resulting two feature maps of size 2k^2 × 1 × 1 are combined by element-wise multiplication. The final output contains relative offsets (for x, y positions) from locations on a regular grid. In our implementation, we use 3× 3 kernels, but the output is computed by aggregating 9 samples sparsely chosen from a much larger neighborhood. The two main reasons behind the use of small-size kernels are as follows: (1) This enables an efficient implementation in terms of speed and memory. (2) The reliability of samples are more important than the total number of samples aggregated. As will be seen in Sec. <ref>, our model outperforms the guided filter <cit.> using kernels of size 17 × 17 by a large margin. A similar finding is noted in <cit.>, which shows that only high-confidence samples should be chosen when estimating foreground and background images in image matting. Note that offset regression is closely related to nonlocal means <cit.> in that both select which pixels to aggregate instead of immediate neighbors. Likewise, learning offsets is related to “self-supervised" correspondence models in stereo matching <cit.> and optical flow estimation <cit.>. For example, in the case of stereo matching, a model is trained to produce a flow field such that a right image is reconstructed by a left one according to that flow field. Our model with filter kernels of size k× k computes k^2 correspondences for each pixel within input images, and also learns the corresponding matching confidence (, the kernel weights).




Weighted average.
Given the learned kernel K and sampling offsets Δq, we compute the residuals f̂_p - f_p as a weighted average: 



    f̂_p  =  f_p + ∑_q∈𝒩(𝓅) K_ps(f,g)f_s(q),

where 𝒩(𝓅) is a local 3 × 3 window centered at the location p on a regular grid (Fig. <ref>(a)). We denote by s(q) the sampling position computed from the offset Δq (Fig. <ref>(b)) of the location q as follows.



    s(q) = q + Δq.

The sampling position s(q) predicted by the network is irregular and typically fractional (Fig. <ref>(c)). We use a sampler to compute corresponding (sub) pixel values f_s(q) as 



    f_s(q) = ∑_t∈ℛ(𝓈(𝓆)) G(s, t) f_t,

where ℛ(𝓈(𝓆)) enumerates all integer locations in a local 4-neighborhood system to the fractional position s(q), and G is a sampling kernel. Following <cit.>, we use a two-dimensional bilinear kernel, and split it into two one-dimensional ones as



    G(s, t) = g(s_x, t_x) g(s_y, t_y),

where g(a,b) = max(0,1-|a-b|). Note that the residual term in (<ref>) is exactly the same as the explicit weighted average, but we aggregate pixels from the sparsely chosen locations s(q) with the learned kernels K, which is not feasible in current methods.

When we do not use a residual connection, we compute the upsampling result f̂_p directly as a weighted average using the learned kernels and offsets:



    f̂_p = ∑_q∈𝒩(𝓅) K_ps(f,g)f_s(q).


Loss.
We train our model by minimizing the L_1 norm of the difference between the network output f̂ and ground-truth HR reference depth map f^gt as follows.



    L(f^gt, f̂) =  ∑_p| f^gt_p -  f̂_p |_1.





Testing.
Two principles have guided the design of our learning architecture:
(1) Points from a large receptive field in the original guidance and
target images should be used to compute the weighted averages
associated with the value of the upsampled depth map at each one of
its pixels; and (2) inference should be fast. The second principle is
rather self-evident. We believe that the first one is also rather
intuitive, and it is justified empirically by the ablation study
presented later. In fine, it is also the basis for our approach, since
our network learns where, and how to sample a small number of points
in a large receptive field.

A reasonable compromise between receptive field size and speed is to
use one or several convolutional layers with a multi-pixel stride,
which enlarges the image area pixels are drawn from without increasing
the number of weights in the network. This is the approach we have
followed in our base architecture, DKN, with two stride-2 “DownConv"
layers. The price to pay is a loss in spatial resolution for the final
feature map, with only 1/16 of the total number N of pixels in the
input images. One could of course give as input to our network the
receptive fields associated with all N of the original guidance and
target image pixels, at the cost of N forward passes during
inference. DKN implements a much more efficient method where 16
shifted copies of the two images are used in turn as input to the
network, and the corresponding network outputs are then stitched
together in a single HR image, at the cost of only 16 forward
passes. The details of this shift-and-stitch approach <cit.>
can be found in the supplemental material.












 §.§ FDKN architecture



A more efficient alternative to DKN is to split the input images into
the same 16 subsampled and shifted parts as before, but this time stack them into new target and guidance images (Fig. <ref>), with 16 channels
for the former, and 16C channels for the latter, , C=3 when the RGB image is
used. The effective receptive field for FDKN is comparable to
that of DKN, but FDKN involves much fewer parameters because of the
reduced input image resolution and the shared weights across
channels. The individual channels are then recomposed into the final
upsampled image <cit.>, at the cost of only one forward pass. Specifically, we use a series of 6 convolutional layers of size 3 × 3 for feature extraction. For weight and offset regression, we apply a 1× 1 convolution on top of the feature extraction layers similar to DKN, but using more network parameters. For example, FDKN and DKN compute feature maps of size 16k^2 × 1 × 1 and k^2 × 1 × 1, respectively, for weight regression, from each feature of size 128 × 1 × 1. This allows FDKN to estimate kernel weights and offsets for all pixels simultaneously. The details of this shift-and-stack approach can be found in the supplemental material. In practice, FDKN gives a 17 times speed-up over DKN. Because it involves fewer parameters (0.6M vs. 1.1M for DKN), one might expect somewhat degraded results. Our experiments demonstrate that FDKN remains in the ballpark of that of DKN, still significantly better than competing approaches, and in one case even overperforming DKN.







§ EXPERIMENTS



In this section we present a detailed analysis and evaluation of our approach. More results and other applications of our model including saliency image upsampling, cross-modality image restoration, texture removal and semantic segmentation can be found in the supplement. 




 §.§ Implementation details




Following the experimental protocol of <cit.>, we train different models to upsample depth maps for scale factors 4 ×, 8 ×, 16 × with random initialization. We sample 1,000 RGB/D image pairs of size 640 × 480 from the NYU v2 dataset <cit.>. We use the same image pairs as in <cit.> to train the networks. The models are trained with a batch size of 1 for 40k iterations, giving roughly 20 epochs over the training data. We synthesize LR depth images (4 ×, 8 ×, 16 ×) from ground truth by bicubic downsampling. We use the Adam optimizer <cit.> with β_1 = 0.9 and β_2 = 0.999. As learning rate we use 0.001 and divide it by 5 every 10k iterations. Data augmentation and regularization techniques such as weight decay and dropout <cit.> are not used, since 1,000 RGB/D image pairs from the NYU dataset have proven to be sufficient to train our models (See the supplement). All networks are trained end-to-end using  <cit.>.








 §.§ Results



We test our models with the following four benchmark datasets. These feature aligned color and depth images. Note that we train our models with the NYU v2 dataset, and do not fine-tune them to other ones to evaluate its generalization ability.


	

	
  ∙ Middlebury dataset <cit.>: We use the 30 RGB/D image pairs from the 2001-2006 datasets provided by Lu <cit.>.
		

	
  ∙ Lu dataset <cit.>: This provides 6 RGB/D image pairs acquired by the ASUS Xtion Pro camera <cit.>. 
		

	
  ∙ NYU v2 dataset <cit.>: It consists of 1,449 RGB/D image pairs captured with the Microsoft Kinect <cit.>. We exclude the 1,000 pairs used for training, and use the rest (449 pairs) for evaluation
		

	
  ∙ Sintel dataset <cit.>: This dataset provides 1,064 RGB/D image pairs created from an animated 3D movie. It contains realistic scenes including fog and motion blur. We use 864 pairs from a final-pass dataset for testing.
		








We compare our method with the state of the art in Table <ref>. It shows the average RMSE between upsampling results and ground truth. All numbers except those for the Sintel dataset are taken from <cit.>. The results of DJF <cit.> and its residual version (DJFR <cit.>) are obtained by the provided models trained with the NYU v2 dataset. DMSG <cit.> uses the Middlebury and Sintel datasets for training the network. For fair comparison of DMSG with other CNN-based methods including ours, we retrain the DMSG model using the same image pairs from the NYU v2 dataset as in <cit.>. From this table, we observe four things: (1) Our models outperform the state of the art including CNN-based methods <cit.> by significant margins in terms of RMSE, even without the residual connection (DKN w/o Res. and FDKN w/o Res.). For example, DKN decreases the average RMSE by 52% (4 ×), 44% (8 ×) and 36% (16 ×) compared to DJFR <cit.>. (2) Our models trained without the guidance of HR color images (DKN^† and FDKN^†), using the depth map only, also outperform the state of the art. In particular, they give better results than DKN and FDKN for the Lu dataset <cit.>. A plausible expiation is that depth and color boundaries are less correlated, since the color images in the dataset are captured in a low-light condition. (3) We can clearly see that our models perform well on both synthetic and real datasets (, the Sintel and NYU v2 datasets), and generalize well to other images (, on the Middlebury dataset) outside the training dataset. (4) FDKN retains the superior performance of DKN, and even outperforms DKN for the Lu dataset. 


Qualitative results.
Figure <ref> shows a visual comparison of the upsampled depth maps (8×). The better ability to extract common structures from the color and depth images by our models here is clearly visible. Specifically, our results show a sharp depth transition without the texture-copying artifacts. In contrast, artifacts are clearly visible even in the results of DJFR <cit.>, which tends to over-smooth the results and does not recover fine details. This confirms once more the advantage of using the weighted average with spatially-variant kernels and an adaptive neighborhood system in depth map upsampling.





Runtime. Table <ref> shows runtime comparisons on the same machine. We report the runtime for DMSG <cit.>, DJFR <cit.> and our models with a Nvidia Titan XP and for other methods with an Intel i5 3.3 GHz CPU. Our current implementation for DKN takes on average 0.17 seconds for HR images of size 640 × 480. It is slower than DMSG <cit.> and DJFR <cit.>, but yields a significantly better RMSE (Fig. <ref> and Table <ref>). FDKN runs about 17 × faster than the DKN, as fast as DJFR, but with significantly higher accuracy.







 §.§ Discussion



We conduct an ablation analysis on different components in our models, and show the effects of different parameters for depth map upsampling (8 ×) on the NYU v2 dataset <cit.>. More discussion can be found in the supplement.

Network architecture.
We show the average RMSE for six variants of our models in Table <ref>. The baseline models learn kernel weights from HR color images only. The first row shows that this baseline already outperforms the state of the art (see Table <ref>). From the second row, we can see that our models trained using LR depth maps only give better results than the baseline, indicating that using the HR color images only is not enough to fully exploit common structures. The third row demonstrates that constructing kernels from both images boosts performance. For example, the average RMSE of DKN decreases from 5.92 to 5.03 for the 3× 3 kernel. The fourth and fifth rows show that learning the offsets significantly boosts the performance of our models. The average RMSE of DKN trained using the HR color or LR depth images only decreases from 5.92 to 5.37 and from 5.24 to 4.06, respectively, for the 3× 3 kernel. The last two rows demonstrate that the effect of learning kernel weights and offsets from both inputs is significant, and combining all components including the residual connection gives the best results. Note that learning to predict the spatial offsets is important because (1) learning spatially-variant kernels for individual pixels would be very hard otherwise, unless using much larger kernels to achieve the same neighborhood size, which would lead to an inefficient implementation, and (2) contrary to current architectures including DJF <cit.> and DMSG <cit.>, this allows sub-pixel information aggregation. 



Kernel size.
Table <ref> also compares the performances of networks with different size of kernels. We enlarge the kernel size gradually from 3 × 3 to 25 × 25 and compute the average RMSE. From the third row, we observe that the performance improves until size of 15 × 15. Increasing size further does not give additional performance gain. This indicates that aggregating pixels from a 15 × 15 window is enough for the task. For offset learning, we restrict the maximum range of the sampling position to 15× 15 for all experiments. That is, the results from the third to last rows are computed by aggregating 9, 25 or 49 samples sparsely chosen from a 15 × 15 window. The last row of Table <ref> suggests that our final models also benefit from using more samples. The RMSE for DKN decreases from 3.26 to 3.19 at the cost of additional runtime. For comparison, DKN with kernels of size 3 × 3, 5 × 5 and 7 × 7 take 0.17, 0.18 and 0.19 seconds, respectively, with a Nvidia Titan XP. A 3 × 3 size offers a good compromise in terms of RMSE and runtime and this is what we have used in all experiments.  

DownConv for DKN.
We empirically find that extracting features from large receptive fields is important to incorporate context for weight and offset learning. For example, reducing the size from 51 × 51 to 23 × 23 causes an increase of the average RMSE from 3.26 to 5.00 for the 3× 3 kernel. The DKN without DownConv layers can be implemented in a single forward pass, but requires more parameters (1.6M vs. 1.1M for DKN) to maintain the same receptive field size, with a total number of convolutions increasing from 0.6M to 1M at each pixel. We may use dilated convolutions <cit.> that support large receptive fields without loss of resolution. When using the same receptive field size as 51 × 51, the average RMSE for dilated convolutions increases from 3.26 to 4.30 for the 3× 3 kernel. The resampling technique (Fig. <ref>) thus appears to be the preferable alternative.  

 





§ CONCLUSION



We have presented a CNN architecture for depth map upsampling. Instead of regressing the upsampling results directly from the network, we use spatially-variant weighted averages where the set of neighbors and the corresponding kernel weights are learned end-to-end. A fast version achieves a 17 × speed-up compared to the plain DKN without much (if any) loss in performance. Finally, we have shown that the weighted averaging process, even using the LR depth image only without any guidance, with 9 samples sparsely chosen, is sufficient to set a new state of the art.



ieee





[pages=1]supp.pdf
[pages=2]supp.pdf
[pages=3]supp.pdf
[pages=4]supp.pdf
[pages=5]supp.pdf
[pages=6]supp.pdf
[pages=7]supp.pdf
[pages=8]supp.pdf
[pages=9]supp.pdf

