# An evaluation of Deep Learning based stereo dense matching dataset shift
from aerial images and a large scale stereo dataset 111The original paper is
published on
https://www.sciencedirect.com/science/article/pii/S1569843224000694. This is
the long version.

Teng Wu, Bruno Vallet, Marc Pierrot-Deseilligny, Ewelina Rupnik  Univ Gustave
Eiffel, ENSG, IGN, LASTIG, F-94160 Saint-Mandé, France

###### Abstract

Dense matching is crucial for 3D scene reconstruction since it enables the
recovery of scene 3D geometry from image acquisition. Deep Learning (DL)-based
methods have shown effectiveness in the special case of epipolar stereo
disparity estimation in the computer vision community. DL-based methods depend
heavily on the quality and quantity of training datasets. However, generating
ground-truth disparity maps for real scenes remains a challenging task in the
photogrammetry community. To address this challenge, we propose a method for
generating ground-truth disparity maps directly from Light Detection and
Ranging (LiDAR) and images to produce a large and diverse dataset for six
aerial datasets across four different areas and two areas with different
resolution images. We also introduce a LiDAR-to-image co-registration
refinement to the framework that takes special precautions regarding
occlusions and refrains from disparity interpolation to avoid precision loss.
Evaluating 11 dense matching methods across datasets with diverse scene types,
image resolutions, and geometric configurations, which are deeply investigated
in dataset shift, GANet performs best with identical training and testing
data, and PSMNet shows robustness across different datasets, and we proposed
the best strategy for training with a limit dataset. We will also provide the
dataset and training models; more information can be found at
https://github.com/whuwuteng/Aerial_Stereo_Dataset.

###### keywords:

Stereo dense matching, deep learning, benchmarking, dataset shift, LiDAR
processing, 3D reconstruction

##  1 Introduction

Dense image matching is a traditional topic in 3D reconstruction, and can be
performed in stereo-pair (with only two views) (Scharstein and Szeliski, 2002)
or multi-view stereo (MVS) (Jensen et al., 2014). In this paper, we focus on
stereo pair dense matching in the specific case of epipolar stereo pairs
(i.e., image correspondences have the same $y$ coordinate) because most of the
recent deep learning (DL) approaches are limited to this simple configuration.

While many traditional methods show good performance (Scharstein and Szeliski,
2002) and are widely used in the geospatial context, the recent successes of
DL-based dense matching methods in the computer vision community (Laga et al.,
2020) raise the question of their applicability in the geospatial context.
Considering the large differences between the general image datasets of
computer vision and the specific nature of geospatial photogrammetry, the
training dataset has an important impact on the learning thus on the
performance of DL-based methods. This paper investigates the relevance of the
training dataset on the 3D accuracy and the transferability of the learned
models in the realm of geospatial applications. Given the difficulty of
finding epipolar stereo datasets with ground truth disparity maps in aerial
photogrammetry, producing such datasets with good quality, diversity and
quantity is also an important work reported hereafter.

This paper is an extension of the work presented in (Wu et al., 2021) and the
contributions can be summarized as follows:

  1. 1.

We release a large aerial dataset consisting of epipolar stereo pairs along
with corresponding sparse ground truth disparity maps computed from a LiDAR
acquisitions

  2. 2.

We propose an efficient and accurate image-to-LiDAR registration method to
improve their geometric alignment; through the produced quality dataset we
ensure fair evaluation.

  3. 3.

We study and discuss the influence of the base to height ratio ($B/H$) on DL-
based dense matching methods.

  4. 4.

We use this dataset to explore the dataset shift between different areas,
resolutions and geometric configurations for DL-based methods.

  5. 5.

We evaluate 11 stereo matching methods, and include 7 additional recent DL-
based methods.

The paper is structured as follows: we start by a review of the state-of-the-
art stereo dense matching methods and benchmark datasets (Sections 2 and 3).
Then, we present our method to generate sparse but accurate ground truth
disparity maps from LiDAR point clouds and our six proposed datasets (Section
4). An in-depth evaluation of 11 methods is then proposed, that includes a
study on dataset shift for learning based methods (Section 5), the last part
is the conclusion.

##  2 State of the art for stereo dense matching

Depending on the strategies of proposed approaches, stereo dense matching
methods can be divided into two categories: traditional and learning based
methods. There has been many surveys in the past twenty years on traditional
methods (Scharstein and Szeliski, 2002) and recent surveys now focus on DL-
based methods (Laga et al., 2020; Hamid et al., 2020). In this paper, we
review and evaluate both categories.

###  2.1 Traditional methods

Traditional dense matching methods (Hirschmuller, 2005) usually can be
decomposed into four steps: hand-crafted features computation, feature
matching across images (cost volume computation), cost aggregation
(optimization) and disparity refinement. For the cost aggregation step,
traditional methods can be divided into local and global methods.

####  2.1.1 Local methods

Local methods mainly take into consideration the local features (hand-crafted
feature) (Hirschmuller and Scharstein, 2007) and use a local region in cost
aggregation (Tombari et al., 2008) or a cost volume filter (Hosni et al.,
2012). Support-weights can be used to improve the window based match (Yoon and
Kweon, 2006). Segmentation can also be used to select the local support region
(Tombari et al., 2007). Cross Based Cost Aggregation(CBCA) uses a local
support by a shape-adaptive region to make the cost aggregation (Zhang et al.,
2009). The limitation is that it is difficult to decide the local window size,
but local methods are very fast.

####  2.1.2 Global methods

Global methods mainly add an optimization in the cost aggregation step, based
on a Markov random field (MRF) model, dynamic programming (Van Meerbergen et
al., 2002), belief propagation (Sun et al., 2003), tree-reweighted message
passing (Kolmogorov, 2005) or Graphcut optimization (Boykov and Kolmogorov,
2004). Semi global matching(SGM) is a popular pyramid based method combining
dynamic programming optimization on several directions and using mutual
information (Hirschmuller, 2005). A GPU implementation of SGM (Hernandez-
Juarez et al., 2016) is also evaluated in this paper, as well as a Graphcut
based method using plane constraints (Taniai et al., 2017). Graphcut based
methods are slower than SGM which is often considered to offer the best
balance between efficiency and accuracy (Bethmann and Luhmann, 2015).

###  2.2 Learning based method

Recently, machine learning (ML) techniques have been applied to the stereo
dense matching problem and have demonstrated significant improvement over
traditional methods on some benchmark datasets (Batsos et al., 2018). With the
development of DL, the performance improvement is even more significant (Laga
et al., 2020). Learning based methods can be divided into traditional ML and
DL-based methods. For ML methods, only a small amount of training data is
sufficient, while DL-based methods need a much larger amount of training data.

####  2.2.1 Traditional machine learning

Traditional ML methods, for instance support vector machines (SVM) and random
forests(RF) have been widely applied to stereo dense matching. An approach
consists in learning the matching behaviors of local methods and combine them
probabilistically (Kong and Tao, 2006). Learning the hyper-parameters of the
stereo algorithm has also been proposed (Pal et al., 2012). RF can be applied
to select the ground control points (Spyropoulos et al., 2014). We also
analyse a method which combines man-crafted features with RF to produce
similarity, and then uses SGM optimization (Batsos et al., 2018). Finally in
(Li and Huttenlocher, 2008) SVM are used to learn a linear discriminant
function.

####  2.2.2 Deep learning

DL has proven successful in image recognition (Girshick, 2015), semantic
segmentation (Long et al., 2015), depth prediction (Laina et al., 2016) etc.
With the proliferation of DL-based methods, both hybrid methods and end-to-end
methods have been applied to stereo dense matching (Laga et al., 2020; Poggi
et al., 2021). While hybrid methods use DL to solve one step in the
traditional pipeline, end-to-end methods aim at replacing the full pipeline,
i.e. to produce a network that takes an epipolar image pair as input and
returns a disparity for each pixel of the left image or both images. Some
frameworks also aim at multi-task learning, which combines other tasks such as
semantic segmentation with stereo dense matching (Yang et al., 2018), which is
out of the scope of this paper.

##### Hybrid methods

The initial researches aimed at applying DL to improve a single step in the
traditional pipeline (Poggi et al., 2021). For the matching cost generation
step, 2D convolutional neural networks (CNN) proved effective in feature
extraction (Žbontar and LeCun, 2016). In order to make CNN efficient, feature
matching can be treated as a multi-class classification problem (Luo et al.,
2016). After feature extraction, traditional optimization is used to obtain
the final result. In the optimization step, SGM-Nets uses a CNN network to
provide learned penalties for SGM step (Seki and Pollefeys, 2017). In the
refinement step, variational networks can be used to refine the disparity
(Knöbelreiter and Pock, 2021).

##### End to end methods

There are two main types of architectures for end to end methods: 2D
convolution and 3D convolution based methods. 2D architectures use an encoder-
decoder structure with a 2D CNN while 3D architectures use 2D CNN for the
feature concatenation and then construct a 3D cost volume processed by a 3D
CNN to produce the final disparity.

For 2D architectures, DispNet was the first end-to-end method (Mayer et al.,
2016). Inspired by FlowNet (Dosovitskiy et al., 2015), DispNetS uses a 2D CNN
encoder-decoder structure, while DispNetC investigates correlation and a 2D
CNN is adopted for cost aggregation. DispNet was later combined to a residual
learning as a two stage training architecture (Pang et al., 2017). Residual
learning was also used in (Liang et al., 2018) to refine the disparity.
Finally, (Tonioni et al., 2019c) uses a multi-scale approach to refine the
disparity with 2D dilated convolutions networks.

For 3D architectures, the pioneer work GC-Net uses a 3D CNN based network as
cost aggregation in the cost volume(Kendall et al., 2017). Pyramid Stereo
Matching network uses spatial pyramid pooling and 3D CNN (Chang and Chen,
2018). High resolution stereo network structures use upscale in the 2D CNN
network, so the 3D cost volume can be downscaled (Yang et al., 2019a). For
high-resolution image matching, to handle the high memory consumption, prune
the 3D cost volume with a differential patch match method is proposed (Duggal
et al., 2019). From the penalty definition of SGM (Hirschmuller, 2005), a
Semi-Global Guided Aggregation(SGA) layer and a Local Guided Aggregation(LGA)
layer are integrated in a 3D CNN named GANet (Zhang et al., 2019a). More
recently, reinforce learning such as neural architecture search is applied to
the stereo dense matching problem (Cheng et al., 2020).

####  2.2.3 Dataset shift

Most of the works listed above are from computer vision or robotics community.
To evaluate the performance of the models, the training and evaluation data is
usually of the same type. The models are usually trained on general image
datasets such as KITTI or Middlebury which are very different from aerial
photogrammetry. The aerial images in these kinds of data are very diverse in
the type of scene, area or season. Because it is not always possible to obtain
ground truth data adapted to train for any specific collection, domain
adaptation or dataset shift ability is an important issue. Dataset shift
refers to the problem where a test image pictures a scene or an object never
seen in (or significantly different from) the training data (Quiñonero-Candela
et al., 2009).

Dataset shift raises three main issues: (1) differences in color and
brightness, that may come from different sensors and lighting conditions; (2)
differences in the range of cost values, range of disparity and cost volume
distribution, that come from different $B/H$ and scene types because of
different scene types or viewing conditions; (3) differences in image
resolution, scale.

To address dataset shift, based on an inaccurate disparity map produced by
traditional methods which are domain agnostic, then obtain the confidence map
of the disparity map, then combined the inaccurate disparity map with a
confidence loss to train the final model from a DL stereo (Tonioni et al.,
2019a). Considering that DL-based features depend on the learning dataset,
(Cai et al., 2020) provides hand-made feature as input, and then uses a 3D CNN
only for the cost aggregation. Based on GANet(Zhang et al., 2019a),
considering that batch normalization depends on the training data, a domain
norm has been proposed for the normalization step (Zhang et al., 2020). Song
et al. (2021) use the target data without ground truth and add a color
transfer pre-processing, a cost normalization layer and a self-supervised
occlusion-aware reconstruction to improve the domain adaption.

##  3 Benchmark dataset review

To evaluate stereo dense matching methods, many benchmark datasets have been
released, especially in computer vision community. There are two main
categories of stereo dense matching datasets: synthetic and real-world scenes.

###  3.1 Synthetic datasets

Some authors propose to generate virtual datasets based on advanced computer
graphics (Yang et al., 2019a; Mayer et al., 2016; Tonioni et al., 2019b).
CARLA is a synthetic dataset widely used in mobile mapping (Dosovitskiy et
al., 2017). In a synthetic dataset, the occlusion is easy to obtain (Butler et
al., 2012). Due to the large volume of synthetic datasets, SYNTHIA-Seqs(Ros et
al., 2016) or Scene Flow (Mayer et al., 2016) are usually used for pre-
training (Zhang et al., 2019b). Virtual KITTI 2 also provides stereo disparity
and is widely used in autonomous driving (Cabon et al., 2020). IRS is an
indoor dataset that also provides the surface normal (Wang et al., 2019).
Table 1 lists the most widely used synthetic dataset for stereo dense
matching.

Table 1: Synthetic datasets for disparity estimation.

Data | Year | Scene | Stereo | Citation
---|---|---|---|---
number
MPI-Sintel | 2012 | outdoor | 1150 | (Butler et al., 2012)
Scene Flow | 2016 | indoor + outdoor | 39824 | (Mayer et al., 2016)
SYNTHIA-Seqs | 2016 | outdoor(driving) | 10000 | (Ros et al., 2016)
Virtual KITTI 2 | 2020 | outdoor(driving) | 17000 | (Cabon et al., 2020)
IRS | 2021 | indoor | 100025 | (Wang et al., 2019)

###  3.2 Real-world datasets

For real-world datasets, image pairs are acquired in an indoor or outdoor
environments and a separate acquisition (in general with LiDAR) is performed
on the same scene to produce a ground truth depth for each image pixel.
Middlebury is a popular indoor dataset (Scharstein et al., 2014). The KITTI
dataset focuses on automobile driving, uses a stereo camera and LiDAR to
obtain the ground truth. It contains two versions, KITTI 2012 (Geiger et al.,
2012) and KITTI 2015 (Mayer et al., 2016). The ETH3D benchmark is another
example of high-resolution stereo pairs (Schops et al., 2017) contains both
indoor and outdoor scenes. The strong ongoing research activity on autonomous
driving has also resulted in several dedicated datasets (Yang et al., 2019b;
Huang et al., 2019). Meister et al. (2012) propose an outdoor dataset in a
challenging environment, but does not provide a ground truth depth. Table 2
lists some benchmarks for stereo dense matching on real scenes proposed by the
Robotics and computer vision communities.

Table 2: Benchmark datasets for disparity estimation. Stereo number is the
total number of stereo pairs used for both training and testing.

Data | Year | Scene | Stereo | Disparity | Citation
---|---|---|---|---|---
number | density
Middlebury V2 | 2003-2006 | indoor | 32 | dense | (Scharstein and Szeliski, 2003)
Middlebury V3 | 2014 | indoor | 33 | dense | (Scharstein et al., 2014)
ETH3D | 2017 | indoor + outdoor | 47 | dense | (Schops et al., 2017)
InStereo2K | 2020 | indoor | 2050 | dense | (Bao et al., 2020)
KITII2012 | 2012 | outdoor(driving) | 389 | sparse | (Geiger et al., 2012)
KITTI2015 | 2015 | outdoor(driving) | 1600 | sparse | (Mayer et al., 2016)
Cityscapes | 2016 | outdoor(driving) | 24998 | dense | (Cordts et al., 2016)
Drivingstereo | 2019 | outdoor(driving) | 182188 | sparse | (Yang et al., 2019b)
ApolloScape | 2019 | outdoor(driving) | 5165 | sparse | (Huang et al., 2019)
SatStereo | 2019††footnotemark:  | satellite | 72 | dense | (Patil et al., 2019)
DFC2019 | 2019 | satellite | 8634 | dense | (Bosch et al., 2019)

###  3.3 Photogrammetry datasets

In aerial photogrammetry, benchmark datasets focus mainly on MVS dense
matching (Cavegn et al., 2014). For satellite imagery, evaluation is usually
performed on the Digital Surface Model (DSM) or height map (Rupnik et al.,
2018; Bosch et al., 2019). The pipeline contains other steps, for example,
point cloud fusion and DSM generation (Cournet et al., 2020), and the data is
to evaluate the whole pipeline, not the sole stereo dense matching approach.
IARPA dataset (Bosch et al., 2016) is a MVS 3D reconstruction challenge for
satellite images and provides the aerial corresponding LiDAR. A recent
satellite image stereo benchmark from LiDAR DSM (Patil et al., 2019) uses the
IARPA dataset . With more and more DL-based methods applied in the geospatial
context, there are more satellite training data (Bosch et al., 2019).

##  4 Aerial Benchmark data generation

While ML, especially DL-based dense matching methods outperform traditional
ones, they usually need a lot of training data (including the ground truth) to
achieve good performance. Moreover, the fair evaluation requires a test set
separate from training data. Generating such training data from real scenes is
challenging. Moreover, the quality of the training data directly impacts the
performance of the trained model, so great care must be taken when producing
it. To avoid possible errors introduced by grid interpolation of point cloud,
we propose a method to generate sparse disparity ground truth from images and
LiDAR point cloud which can be registered if they are not well aligned, then
we produce epipolar pairs for training DL architectures, and finally, we
evaluate both traditional ML-based and DL-based dense epipolar matching
approaches.

In this section, we describe the production of these epipolar images and
corresponding ground truth disparity maps from the original aerial images and
LiDAR. Our workflow is mainly based on the open-source MicMac library
(Pierrot-Deseilligny et al., 2014). As far as we know, such training data are
rare in the photogrammetry community. Thus, we openly release our dataset
(including epipolar pairs and corresponding ground truth disparity maps).

###  4.1 Ground truth sparse disparity generation

Considering the configuration of the stereo dense matching methods
investigated in the experiments, we generate our ground truth as a sparse
disparity in epipolar geometry. Epipolar image generation is described in
Section 4.1.2. In real scenes, several factors influence the quality of a
LiDAR-based disparity ground truth :

  * 1.

Image orientation accuracy (inner consistence) which mainly depends on the
quality of the bundle adjustment and should be controlled in order to
guarantee sub-pixel accuracy.

  * 2.

Co-registration accuracy between image and LiDAR (external consistence): this
is addressed in Section4.1.1

  * 3.

Occlusions between the image and the LiDAR: Even if an image and a LiDAR point
cloud are acquired from the same platform, the scene is acquired from multiple
different viewpoints. Consequently, the whole LiDAR point cloud is probably
not seen in the image to which it projects. To address those scenarios we
developped a specific occlusion handling scheme described in Section 4.1.3.

  * 4.

Changes between the image and the LiDAR: if the images and LiDAR are not
acquired simultaneously, some disparity evaluations were poor not because of
the evaluated methods but due to changes (constructions, vegetation growth,…)
in the scene between the image and LiDAR acquisitions. In DL-based semantic
segmentation, such inconsistencies in the ground truth are called noise label
and several research works such as self-ensemble label filtering (Nguyen et
al., 2019) address this issue. Using a similar idea, we propose a two-step
based strategy to enhance the ground truth which is described in the section
4.1.4.

####  4.1.1 Image and LiDAR registration

In practice, it is not easy to find well-registered images and LiDAR due to
the different purpose of data collection, in particular if they are not
collected simultaneously. If the image and LiDAR is not well-registered, we
propose to refine the registration in order to improve their consistency.
Registration between aerial optical images and LiDAR is another difficult
topic (Zhou et al., 2016). In our case, given that the initial orientation is
usually quite accurate (close to the real orientation), a simple iterative
refinement based on ICP (Huang et al., 2018) is proposed. We start by
computing the 3D tie points corresponding to the feature points matched during
the bundle adjustment. As the initial orientation is close enough to the real
position, these tie points lie close to LiDAR points. As the noise in LiDAR is
small, the closest LiDAR point is regarded as a ground control point (GCP).
Using these GPCs, the orientation can be estimated again, and the whole
process can be repeated until convergence, iteratively improving the
orientation. It is difficult to know when to the terminate the loop, we
evaluated this approach by measuring the disparity error between an SGM result
and the LiDAR to indicate the accuracy of the orientation of images, the
analysis is refer to section 5.2.

####  4.1.2 Epipolar image generation

The learning based dense matching algorithms exploited assume an epipolar
rectification of the images where the counterpart of a pixel in one image lies
in the same line of the other image. Moreover some approaches are limited in
the maximum size of the input image. The first step of our pipeline is to
generate epipolar image pairs, and only the image pairs with sufficient
footprint overlap are considered. Coarse image footprints are obtained using
an approximate height of the scene and the Computational Geometry Algorithms
Library(CGAL) (Alliez et al., 2020) is used to compute their intersections. In
our experiment, only images pairs with an intersection area above half the
image footprint are considered for epipolar rectification, which was done
using a recent module of the MicMac library (Deseilligny and Rupnik, 2020).
The corresponding orientation parameter files are generated in order to allow
the direct projection of 3D point in the rectified images.

####  4.1.3 Occlusion aware point cloud projection

The main question of occlusion aware projection is to ensure that a LiDAR
point is only projected in an (epipolar) image if the corresponding 3D point
is actually seen in this image. As images can only capture the apparent
surface, we only keep the first LiDAR echo. Also, we filter out outliers based
on the point cloud density using a grid (Cho et al., 2004), if there is no
neighborhood points for one points in radius 3m, this point is noise point,
and removed.

The ground truth disparity maps used for both training and evaluation are
computed from the LiDAR point cloud. A perspective projection model is used to
project the 3D points to the image focal plane. The disparity $d$ is defined
in Equation 1.

$d=x_{l}-x_{r}$ (1)

where $x_{l}$ and $x_{r}$ are the projection coordinate on the $x$ axis in the
image plane after projecting the 3D point cloud to the left and right images.
By construction, they project at the same $y$ value.

Because the point cloud is sparse, it is difficult to predict the visibility
of each point in the image, which is a well known issue (Bevilacqua et al.,
2017). An example of occlusion (in the left image) is shown in Figure 1.
Points from the ground area should be removed as in reality, they are occluded
by the roof.

(a) Occlusion in stereo

(b) Points on image plane

Figure 1: Occlusion in projection.

In our previous work (Wu et al., 2021), an image space filter based on the
disparity was used. The idea is that if a point is surrounded by closer points
in image space, it is probably occluded. But we found that disparity
discontinuities (on vertical surfaces for instance) were not well handled by
this filter as the disparity varies rapidly there. To solve this issue, we
propose a ray tracing based occlusion detection before applying the filter.
The idea is to reconstruct a continuous 3D surface from the points, then use
ray tracing to detect the occlusions as intersections of the sight ray (from a
3D point to an image optical center) with this surface. However 3D surface
reconstruction from point clouds is a complex and time consuming problem.
Considering the globally 2.5D nature of aerial LiDAR, we propose a simple 2D
Delaunay triangulation (Shewchuk, 2020) in the horizontal plane to generate a
simple surface mesh from the point cloud. As illustrated in Equation 2, $z$
relates to the disparity $d$ directly. Because the height ($z$ value) is not
considered in 2D Delaunay triangulation, this can cause errors in areas where
the $z$ value can be multiple (overlaying roof and ground for instance). To
handle this issue we filter out triangle for which the difference in the $z$
coordinate is too large before ray tracing. In our experiment, the maximum $z$
difference for a triangle is 2 meters. As we generate ground truth disparity
maps for the right image, we trace 2 rays from each LiDAR point to the optical
center of the right and left images. We use the AABB Tree implemented in CGAL
(Alliez et al., 2020) to accelerate the intersection of these rays with the
valid triangles of the mesh. If both rays do not intersect the mesh, the point
is projected in both images and the disparity is computed as the difference
between the column indices in the right and left images. Because the pair is
epipolar, the projections have the same line index. Note that we could have
kept points seen only in the right image. In practice, adding this points
decreases the quality of the result as presented in section 5.3.

In epipolar geometry, the disparity at a pixel of the right image is related
to the depth of the 3D point seen by that pixel (Jain et al., 1995):

$z=\frac{b\cdot f}{d}$ (2)

where $z$ is the depth, $b$ is base line length, $f$ is the focal length, $d$
is the disparity. The disparity is negatively correlated to the depth. For
aerial images, the disparity is related to the height of objects, such that
ground points have larger disparities.

Finally, after ray tracing, we apply an image space disparity filter to handle
areas with large $z$ variance as proposed in (Biasutti et al., 2019): if the
difference between the disparity and its median in an image space neighborhood
is larger than a threshold, as shown in Equation 3, $d_{max}$ and $d_{min}$ is
the maximum and minimum disparity in the window, then the point is removed,
which filters occluded points quite effectively.

$\alpha=\frac{d-d_{min}}{d_{max}-d_{min}}$ (3)

####  4.1.4 Removing areas with scene changes

Because the wrong training data will deteriorates the performance, our aim is
not to detect changes precisely but to remove the epipolar pairs in which
significant changes occurred between image and LiDAR acquisitions from our
dataset. If both are acquired simultaneouly, this step can be skipped.

First, we use SGM(CUDA) (Hernandez-Juarez et al., 2016) to select high
accuracy (with high evaluation score) stereo pairs. Then, we fine-tune PSMNet
(Chang and Chen, 2018) on these pairs, then use the fine-tuned model on the
whole dataset. Finally the stereo pairs with large errors are removed.

In this paper, we evaluate the quality of dense matching using cumulative
histograms (proportion of pixels where the error between the estimated
disparity and the ground truth disparity map is smaller than $N$) which are
the exact opposite of the $N$-pixel error (proportion of pixels where the
error is larger than $N$). While 1-pixel error is a good indicator for the
precision of the disparity (we count only pixels where the disparity is
perfectly estimated), larger $N$s indicate the robustness of the method
(ability to give a correct, even if inaccurate, disparity in every situation).
In order to cope with possible scene changes, we decided to discard all stereo
pairs for which the 1-pixel error of SGM(CUDA) is over 60%. SGM(CUDA) was
chosen as it is our baseline, it is the fastest method and the quality of its
results is quite independent from the scene. A 1-pixel error over the 60%
threshold indicates that the matching problem is too hard (vegetation, water
or large untextured areas), or that there are significant changes in the scene
between the image and LiDAR acquisitions. As we do not want to filter out
”hard” cases (which would bias our evaluation), we train PSMNet on the ”easy”
dataset (with SGM 1-pixel error below 60%) and finally filter out the stereo
pairs for which the PSMNet 1-pixel error is over 40% which we consider is a
good indicator of change. More accurate change filtering can be obtained by
iterating this second step (learning PSMNet on the new ”easy/unchanged”
dataset to have a better model to filter out changes again). This strategy can
also be used to handle the asynchronous image and LiDAR acquisition.

###  4.2 Datasets presentation

Along with this paper, we publish 6 datasets, as listed in Table 3 produced
with the pipeline described above applied to images and LiDAR from different
countries. For certain areas, there are several datasets with different
resolutions and acquired at different times as summarized in Fig. 2.

Table 3: processed dataset for experiment.

dataset | color | GSD(cm) | LiDAR (pt/$m^{2}$) | train+test
---|---|---|---|---
ISPRS-Vaihingen | IR-R-G | 8 | 6.7 | 640 + 516
EuroSDR-Vaihingen | RGB | 20 | 6.7 | 421 + 353
Toulouse-UMBRA | RGB | 12.5 | 2-4 | 7409 + 21458
Toulouse-Metropole | RGB | 5 | 8 | 9813 + 14241
Enschede | RGB | 10 | 10 | 871 + 819
DublinCity | RGB | 3.4 | 250-348 | 35128+6799

(a) ISPRS Vaihingen

(b) EuroSDR Vahingen

(c) Toulouse UMBRA

(d) Toulouse Metropole

(e) Enschede

(f) DublinCity

Figure 2: All the dataset introduction.

####  4.2.1 ISPRS Vaihingen dataset

The Vaihingen dataset is from the ISPRS 3D reconstruction benchmark which
provides a good registration of images and LiDAR point cloud. The dataset is
composed of 20 images with a depth of 11bit (Rottensteiner et al., 2012). As
this dataset has an accurate geolocation, and image and LiDAR were collected
nearly at same time, we did not apply the image/LiDAR registration and change
filtering presented above, such that all the epipolar pairs are used and split
between train and test sets.

####  4.2.2 EuroSDR Vaihingen dataset

This data is an aerial image matching benchmark dataset (Haala, 2013), the
resolution is different from the ISPRS Vaihingen dataset, but the LiDAR is
same. Because the resolution is low, and the experiment area is not large, all
the possible pairs which have overlap are used. The image data was collected
nearly the same time as ISPRS-Vaihingen (Cramer, 2010). As the dataset size is
smaller than ISPRS Vaihingen, all the epipolar pairs are used and split into
training and testing set.

####  4.2.3 Toulouse UMBRA dataset

The dataset is published in (Adeline et al., 2013), even though there is a NIR
band in the origin dataset, only RGB bands are used. The image orientation is
not accurate so we applied our image/LiDAR registration. The image was
collected in 2012 and LiDAR was collected in 2013, so we also applied our
change filtering. As the dataset is very large, the training (1200 pairs) and
testing (870 pairs) subsets are randomly selected.

####  4.2.4 Toulouse Metropole dataset

The dataset was provided to the AI4GEO project (see Section 8) by the
metropole of Touloude. The image and LiDAR were collected at nearly same time
so we did not apply change filtering. The dataset has a higher resolution than
the Toulouse UMBRA dataset, and covers a larger area which contains the
Toulouse UMBRA area. Given its large size, the traning (1200 pairs) and
testing (1116 pairs) subsets are randomly selected.

####  4.2.5 Enschede dataset

This dataset is used and presented in (Zhang et al., 2018). The image was
collected in 2011 and the LiDAR is from AHN2 which was collected in 2012 in
the Enschede area (Nederland, 2021) so we applied our change filtering. The
original dataset has an oblique setting with 5 cameras. Considering that the
other datasets do not contain oblique images, we selected only the nadir
images to generate the epipolar pairs. The base to height ratio is large in
this dataset. The orientation is generated using Micmac with ground control
points and our image/LiDAR registration is applied. All the epipolar pairs are
used and split into training and testing subsets.

####  4.2.6 DublinCity dataset

The dataset is published in (Laefer et al., 2015; Zolanvari et al., 2019), the
image and LiDAR were collected nearly same time so we did not filter changes.
The original dataset contains very high density LiDAR data and both vertical
and oblique images. Once again, only the vertical images are used. Because the
dataset large size, we randomly sampled 1200 pairs for training and 257 pairs
for testing. For DublinCity dataset, the orientation files are provided, but
the orientation is not accurate, so the image/LiDAR registration refinement is
applied.

##  5 Evaluation and discussion

In our experiments, we evaluated traditional (providing a baseline), machine
learning (ML) and deep learning (DL) methods. For DL, we have tried to cover
the most significant works from the litterature but we had to discard:

  1. 1.

DispNet (first end-to-end DL-based stereo dense matching method), as it
requires a dense disparity ground truth and ours is sparse.

  2. 2.

GC-Net (first dense matching method using a 3D CNN to process the cost volume)
as it has no official code release.

As listed in Table 4, 11 methods were selected in the experiments, categorized
with traditional method, machine learning based method, DL-based method and
end-to-end method.

Table 4: A synthesis of the explored methods.

---
method |  feature extraction |  cost aggregation |  brief introduction |  citation
MICMAC |  NCC based feature |  dynamic programming |  variant of SGM implementation |  (Pierrot-Deseilligny and Paparoditis, 2006)
SGM (GPU) |  census based feature |  dynamic programming |  SGM implementation based on GPU |  (Hernandez-Juarez et al., 2016)
GraphCuts |  intensity and gradient consistency |  local expansion |  add plane surface assumption in GraphCuts framework |  (Taniai et al., 2017)
CBMV (SGM) |  man-crafted features with RF fusion |  dynamic programming |  Combine man-crafted features with RF to produce similarity, use SGM optimization |  (Batsos et al., 2018)
CBMV (GraphCuts) |  man-crafted features with RF fusion |  local expansion |  Combine man-crafted features with RF to produce similarity, use GraphCuts optimization |  (Batsos et al., 2018)
MC-CNN |  window-based 2D CNN feature |  dynamic programming |  crop the image into patchs, 2D CNN to extract feature similarity, then use SGM optimization |  (Zbontar et al., 2016)
DeepFeature |  window-based 2D CNN feature |  dynamic programming |  more efficient 2D CNN to extract feature similarity, then use SGM optimization |  (Luo et al., 2016)
PSM net |  2D CNN encoder-decoder feature |  3D CNN encoder-decoder |  a typical DL-based dense matching pipeline |  (Chang and Chen, 2018)
HRS net |  2D CNN encoder-decoder feature |  3D CNN encoder-decoder |  in the feature extraction step, unpooling to obtain high resolution feature |  (Yang et al., 2019a)
DeepPruner |  2D CNN encoder-decoder feature |  3D CNN encoder-decoder |  prune the search range using CNN based patch match |  (Duggal et al., 2019)
GANet |  2D CNN encoder-decoder feature |  3D CNN encoder-decoder |  guided aggregation network in 3D CNN |  (Zhang et al., 2019a)
LEAStereo |  2D CNN encoder-decoder feature |  3D CNN encoder-decoder |  Use Neural Architecture Search to search the network architecture |  (Cheng et al., 2020)

###  5.1 Experiment details

In order to evaluate the performance without bias, all the experimented
methods use the official code release. MICMAC is implemented in MicMac written
in C++ (micmacIGN, 2020). SGM (GPU) is based on C++ and CUDA (Hernandez-
Juarez, 2020). GraphCuts is written in C++ (Taniai, 2020). CBMV is written in
C++, CUDA (SGM implementation) and Python (Batsos, 2020). CBMV has two
pipelines in optimization step, dynamic programming optimization (CBMV (SGM))
and Graphcut (CBMV (GraphCuts)) which has the same implementation with
GraphCuts. MC-CNN is based on C++, CUDA (SGM implementation) and Lua torch
(Zbontar, 2020). DeepFeature (Luo, 2020) is similiar to MC-CNN. PSM net
(Chang, 2020), HRS net (Yang, 2020), DeepPruner (UberResearch, 2020), and
LEAStereo (Cheng, 2021) are based on Pytorch. GANet is based on CUDA and
Pytorch (Zhang, 2020).

For DL-based methods, the experimental setup and especially the
hyperparameters are important. In our experiments, we compare DL-based methods
pretrained on the KITTI dataset and fine-tuned on our new dataset. We found
that batch size has a strong impact on the memory requirements and on the
accuracy.

The training data size influences the performance of the training model: a
larger, more diverse training dataset usually gives a better result. For each
dataset, the number of available stereo pairs is different. In order to make
the comparison more impartial, we set the maximum number of training stereo
pairs used per dataset to 1200 (it can be less if the dataset has less than
1200 pairs).

The training strategy differs between methods: hybrid methods are pixel based,
while end-to-end based methods are stereo pair based, and some methods crop
the training images. The training time is also varies significantly between
different methods. Finally, we select the result with the best loss for each
method. In the experiment, we use the cluster of French National Centre for
Space Studies(CNES), the GPU card is NVIDIA Tesla V100, the memory size is
32GB. In order to make the statistics of the computation time of training, all
the experiments are training on Toulouse Metropole dataset, the detail is in
the Table 5. The training strategy of MC-CNN and DeepFeature is different from
end-to-end method, because learning similarity is pixel pair based, and all
the valid pixels which has LiDAR points are processed in the training, but in
the training of end-to-end methods randomly crop the input image, the cropped
size uses the default value, this means not all the data is used in the
training in the end-to-end method. Considering the total image number is same,
so the train time is training time of each epoch.

Table 5: Training configuration and time for each method. name | epoch | croped | batch size | train time/s | pairs
---|---|---|---|---|---
MC-CNN | 10 | $\times$ | 128 | 17591 | 1200
DeepFeature | 20 | $\times$ | 128 | 4121 | 1200
PSMNet | 500 | 256x512 | 12 | 216 | 1200
HRS net | 700 | 512x768 | 28 | 108 | 1200
DeepPruner | 900 | 256x512 | 16 | 108 | 1200
GANet | 500 | 256x256 | 4 | 1260 | 1200
LEAStereo | 500 | 288x576 | 4 | 451 | 1200

###  5.2 Orientation accuracy

In our experiments, we found that the accuracy of the image orientations
influences the evaluation of the stereo methods. According to our experience,
the 1-pixel error is a good indicator for orientation accuracy which is why we
used it for image/LiDAR registration. The results on the DublinCity dataset
Figure 3 show that our registration allows a significant decrease in this
error.

Figure 3: Cumulated histogram of the disparity error between the SGM and
PSMNet results and the LiDAR.

###  5.3 Occlusion analysis

Occlusion can be handled in two different ways in the ground truth generation:
a ground truth disparity map is produced at a pixel location of the left image
if a 3D LiDAR point projecting on that pixel:

  1. 1.

can be seen in both views

  2. 2.

can be seen at least in the left view

We led an experiment to evaluate which strategy is the best. We divided the
pixels into two categories: (seen) pixels which can be seen in two views, and
(occluded) pixel which can only be seen in the left view. Because the occluded
pixels form only a small part of the whole (seen + occluded) set, we propose
two train settings: training with the whole (seen + occluded) dataset and
training with only the seen pixels. For the evaluation, we propose three
settings: evaluating on all (seen + occluded) pixels, on seen pixels only and
on occ(luded) pixels only. Note that the DublinCity dataset has a much denser
LiDAR than other datasets and many tall buildings so there is a larger
proportion of occluded pixels. As shown in Table 6, adding occluded pixels to
the training set slightly improves the evaluation on occluded pixel, but there
is no significant difference on the seen pixels, because the occluded pixels
have a small percentage in the whole image, so for the evaluation of all
pixels, using occluded pixels decreases the accuracy on seen pixels. So in the
other experiments, only the non occluded pixels are analysed.

Table 6: Influence of training with and without occ(luded) pixels for PSM net. training | testing | <1-pixel | <2-pixel | <3-pixel | <5-pixel | <9-pixel
---|---|---|---|---|---|---
all | all | 84.62 | 92.83 | 95.31 | 97.37 | 98.99
seen | all | 84.69 | 92.84 | 95.31 | 97.37 | 99.00
all | seen | 84.62 | 92.83 | 95.31 | 97.37 | 98.99
seen | seen | 84.69 | 92.84 | 95.31 | 97.37 | 99.00
all | occ | 25.13 | 37.38 | 43.41 | 50.28 | 59.48
seen | occ | 24.45 | 36.71 | 43.12 | 49.97 | 59.38

###  5.4 ML performance on a single dataset

To make the comparison fair, all learning methods are trained and tested on
the same dataset. The CBMV model is completely learned from the dataset as it
only requires a small amount of training data. DL-based methods are finetuned
from the KITTI trained model provided officially. The results for the six
aerial dataset are detailed in Figure 4 displaying the $N$-pixel error. For
traditional method, due to the plane constraint, Graphcuts is better at
1-pixel error (more accurate), but worse than SGM method at large pixel error
(less robust). For CBMV, the results are similar between SGM and Graphcuts
optimization. DL-based methods finetuned on the aerial dataset improve
significantly the quality of the dense matching compared to the baseline SGM
method. Among DL-based methods, end to end methods perform better than hybrid
methods. Generally, in the plane area, CBMV(Graphcuts) mehtod is better than
traditional methods, which means that replacing handcrafted features by
learned features can improve traditional pipelines. For DL-based methods,
fine-tuning on (different blocks from) the same data area significantly
improves the performance. The rank of DL-based methods can vary between areas.
We believe that this is due to variations in the proportion of large base to
height ratio ($B/H$) stereo pairs as they are harder to match than small $B/H$
pairs, and some methods might be more or less sensitive to this $B/H$ induced
difficulty. This is why we investigate this sensitivity to the $B/H$ in the
next section.

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse UMBRA

(d) Result on Toulouse Metropolet

(e) Result on Enschede

(f) Result on DublinCity

Figure 4: All the method test on our dataset.

###  5.5 Influence of the base to height ratio

Comparing to KITTI dataset of which the $B/H$ is fixed, in our experiments, we
found that the $B/H$ of the epipolar stereo pairs has an important influence
on the evaluation result. Images along-strip are usually used for dense
matching because they have a large overlap and small $B/H$. Images across-
strip have a larger $B/H$ which can increase the intersection accuracy, but
leads to more errors because of a larger perspective distortion and more
occlusions (Tola et al., 2008). For different stereo pair and dataset, the
$B/H$ is different. The $B/H$ influences the disparity range and distribution.
As shown in Figure 5, $B/H$ is calculated from the orientation information.
EuroSDR Vaihingen and Enschede have a large $B/H$ compared with the others, as
shown in Figure 5(b) and Figure 5(e).

(a) ISPRS Vaihingen

(b) EuroSDR Vaihingen

(c) Toulouse UMBRA

(d) Toulouse Metropole

(e) Enschede

(f) DublinCity

Figure 5: The base to height ratio distribution of the dataset.

To evaluate the influence of the $B/H$, we decided to use the Toulouse
Metropole dataset which has a high quality and is large enough to split by
$B/H$. The original dataset consisting in 20 aerial images decomposed in 1349
large epipolar pairs is split into 26775 training cropped pairs and 39711
testing cropped pairs. The training set is further split into small
($B/H<0.4$, 129 pairs), middle ($B/H=$ 0.4 ~ 0.6, 511 pairs) and large
($B/H>0.6$, 65 pairs) subsets. We evaluated different settings for the
training as detailed in Table 7. For the selected setting, we have trained a
specialized network for each category (small, middle and large $B/H$) and
select the specialized model corresponding to the actual $B/H$ of the pair to
process. Because there are 200 pair are used as a validation set, so there are
600 pair in the full set.

Table 7: Training set for the $B/H$ experiment.
---
abbreviative name | small $B/H$ | middle $B/H$ | large $B/H$ | total
small | 1200 | 0 | 0 | 1200
middle | 0 | 1200 | 0 | 1200
large | 0 | 0 | 1200 | 1200
fusion | 600 | 0 | 600 | 1200
ave | 400 | 400 | 400 | 1200
random | 249 | 857 | 94 | 1200
all | 1200 | 1200 | 1200 | 3600
full | 6776 | 18069 | 1330 | 26175

We see in Figure 6 that the proportion of each $B/H$ category in the
evaluation dataset influences the evaluation of the method and that using more
training pairs improves the performance. For the small $B/H$ dataset, training
with small $B/H$ only gives an equivalent result as training with the whole
dataset, as shown in Figure 6(a), but training with large $B/H$ only gives
even worse results than SGM(CUDA). But for middle and large $B/H$, using more
training data always improve performance, as shown in Figure 6(b) and Figure
6(c). In Figure 6(c), we see that training with large $B/H$ improves the
performance. However, because using more data to train means more computation
time, we can look for the best strategy for an equivalent number of training
pairs. The experiments show that the optimal training strategy is then to
balance the 3 $B/H$ classes.

(a) Small $B/H$

(b) Middle $B/H$

(c) Large $B/H$

(d) All data

Figure 6: The base to height ratio experiment on Toulouse Metropole.

###  5.6 Dataset shift

In photogrammetric applications, it is hard and costly to obtain training data
for every scene, so we need to train on a limited diversity of areas and apply
the learned model to other areas, which is commonly referred to as
transferbility or dataset shift (Quiñonero-Candela et al., 2009). We propose
some experiments to study this dataset shift, considering that our 6 aerial
datasets span 4 different geographic areas. From our experience, several
factors influence the performance of DL-based dense matching methods : (1)
different scene type; (2) different image color and brightness, as shown in
Fig. 7, it varies from dataset; (3) different resolution of image; (4)
different disparity range; (5) different distribution in cost volume. These
factors influence the method transfer from one dataset to another. Considering
these factors, the transfer learning strategies and experiments are conducted
in three aspects :

  * 1.

Dataset shift between different types of scenes. While training datasets are
abundent in the computer vision community, training data is quite rare in
photogrammetry. Thus analysing the shift between computer vision and
photogrammetry datasets is interesting. To investigate this topic, we compare
training on our proposed datasets with training on the KITTI dataset. This is
made easy considering that all tested deep learing methods provide a model
trained on KITTI.

  * 2.

Dataset shift on scenes of the same type (remote sensing images) but between
different geographical areas.

  * 3.

Dataset shift on the same geographical area but with different image
resolutions. Among our datasets, two areas have data with different
resolutions: ISPRS-Vaihingen has 8 $cm$ GSD while EuroSDR-Vaihingen has 20
$cm$ GSD, and Toulouse UMBRA has 12.5 $cm$ GSD while Toulouse Metropole has 5
$cm$ GSD.

(a) ISPRS Vaihingen

(b) EuroSDR Vaihingen

(c) Toulouse UMBRA

(d) Toulouse Metropole

(e) Enschede

(f) DublinCity

Figure 7: The image histogram distribution of the dataset.

To discuss the performance of remote sensing images dataset shift, training on
a single area is more distinct, on the hand, the influence of more than one
training areas is also an interesting point, so we split the experiment into
two parts : (1) single data training, mainly discuss the above these factors
in dataset shift, refer to Section 5.6.1; and (2) multi-data training, mainly
discuss dataset shift with more than one training data, refer to Section
5.6.2.

####  5.6.1 Single data training in dataset shift

For the single data training dataset shift experiment we limited the number of
lines in Fig. 8 by only evaluating PSMnet and DeepPruner together. SGM(CUDA)
is chosen as the baseline and we used the pre-trained KITTI model to analyse
the dataset shift between different types of scenes. The name after each
method indicates the training data.

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse UMBRA

(d) Result on Toulouse Metropole

(e) Result on Enschede

(f) Result on DublinCity

Figure 8: Dataset shift on a single dataset.

Fig. 8 shows that the KITTI models perform globally worse than the SGM(CUDA)
baseline. DL-based methods perform better than the baseline but this
improvement is around twice less when the model is trained on a different
area. PSMNet has a better transferability than DeepPruner, which shows that
transferability depends on the network structure. In our experiments, the
model trained on Toulouse UMBRA performs badly. This is probably due to the
fact that it has quite different brightness from others as shown in Fig. 7(c).
This shows that brightness negatively influences transferability.

In Figure 8(a), the pink line shows that the model trained on the EuroSDR
Vaihingen is better than the others, which shows that quality degrades more
when training on a different area than when training at a different
resolution. EuroSDR Vaihingen is an exception: Fig. 8(b) shows that PSMNet
trained on DublinCity is slightly better than trained on ISPRS Vaihingen. All
models except the one trained on Toulouse UMBRA have a better performance than
SGM(CUDA). In Figure 8(d), all the trained models are better than SGM(CUDA),
because the contrast is low, which seems to have more impact on traditional
methods. Training on Toulouse Metropole and testing on Toulouse UMBRA is the
just a little worse than training from DublinCity which is our highest
resolution dataset, which shows the importance of the resolution of the
dataset used for training. In Figure 8(c), the base to height ratio is small
and SGM(CUDA) has a good performance while the model trained on Toulouse UMBRA
performs badly. For Enschede and DublinCity, there is only one dataset in the
area. Because the $B/H$ is large in Enschede (cf Fig. 5), the model trained on
EuroSDR Vaihingen has a better performance, as show in Fig. 8(e) (pink line
and pink dot line). Even though DublinCity has a very high resolution, the
PSMNet model trained on EuroSDR Vaihingen has the best performance as shown in
Fig. 8(f).

####  5.6.2 Multi-data training in dataset shift

Based on the conclusions of the single datatset transferability experiment, we
believe that training a model on multiple geographic areas can lead to a more
generic model that performs well on unseen areas (areas not in the training
set) which is a very important issue for the practical application to
photogrammetry, as we cannot expect to have ground truth learning data
everywhere. As the performance of a network is obviously very dependant on the
dataset used to train it, we propose an experiment with two settings : Aerial
all train the model on all available datasets; Aerial real train the model on
all datasets, except the ones that are on the same geographical area than the
test area. We believe that Aerial all is the best model that we can train
based on our datasets, but we cannot evaluate how well it transfers to unseen
areas. So Aerial real is a more realistic setting where the test area is
unseen in training. Considering that there are 4 different areas, the
experiment is testing on one dataset (from one area) and training on datasets
from the other 3 areas. We evaluated dataset shift for all learning based
methods but only present here the results for PSMNet, the results of the other
method are provided in the appendix.

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse UMBRA

(d) Result on Toulouse Metropole

(e) Result on Enschede

(f) Result on DublinCity

Figure 9: Training with multi-dataset.

As shown in Figure 9, training on a single dataset and testing on the same
dataset is better than the model from Aerial all. The model Aerial all is
always the second best. Models trained with datasets from different
geographical areas Aerial real transfer better to unseen areas than when the
learning is on a single (different)area. In Figure 9(a), training from EuroSDR
Vaihigen is better than the model from Aerial real. In Figures 9(b), 9(d) and
9(f), the model training from Aerial real is better than traing from a single
dataset. In Figure 9(c), for the Toulouse UMBRA dataset, training modle from
Aerial real is near to Dublin and is better than the other training from a
single dataset. In Figure 9(e), training from the model training from Aerial
real is only better than training from Toulouse UMBRA and Toulouse Metropole.

####  5.6.3 Quantitative evaluation

In order to make a quantitative evaluation of transferability, similar to the
”relative transfer gain” in (Mensink et al., 2021), we propose to estimate a
”relative shift gain”:

$R_{gain}=(\frac{p}{p_{base}}-1)*100$ (4)

In equation Equation 4, $p$ is the exact opposite of the $N$-pixel error ,
once again, we use SGM(CUDA) as the baseline $p_{base}$ to analyse the
”relative shift gain” for all the other methods.

While we displayed cumulative histograms in Figure 9, we need single metrics
here so we restrict to the 3-pixel error. To analysis the base to height
ratio, we add a new configuration Toulouse Metropole($B/H$), for the training,
there are 400 pairs from small ($B/H<0.4$), 400 pairs from middle ($B/H=$ 0.4
~ 0.6) and 400 pairs from large ($B/H>0.6$), for the testing, there are also
20 stereo pairs from small $B/H$, 20 stereo pairs from large $B/H$, and 20
stereo pairs from large $B/H$. For a more detailed analysis, 1-pixel and
average errors are included in the appendix. In Table 8, in order to show the
dataset shift, the results are highlighted by dataset. The color-coding in the
tables ranges from red to green, the entire column indicates the
transferability of a dataset, while the entire row indicates the
transferability of a method. From the experiment, we can make a summary, for
the hybrid methods:

  * 1.

Less sensitive to transfer.

  * 2.

Less improvement after fine-tuned compared to end-to-end methods.

For the end-to-end methods, the transferability depends on the methods:

  * 1.

PSMNet: the difference between the best case and the worst case is large.

  * 2.

GANet and LEAStereo: the difference is smaller, indicating that they are less
sensitive than PSMNet.

Concerning datasets:

  * 1.

Toulouse UMBRA: all the relative shift gains are large, which means that
training on any dataset can improve the performance.

  * 2.

Toulouse Metropole: training does not bring much improvement. But when the
testing data contain large base to height ratio pairs, training can improve
the performance.

In the Toulouse UMBRA and Toulouse Metropole columns, relative shift gains are
often negative, which means that these dataset are probably too different from
the others to hope for a good transfer. On the other hand, for ISPRS Vaihingen
and EuroSDR Vaihingen, relative shift gains are often positive. Even the input
bands are different, the means the bands dosen’t influence much in the DL-
based methods. Finally, we see that DublinCity performs best among the 6
dataset. As it is by far the dataset with the best resolution, a simple
conclusion is that a better resolution makes a dataset better for training.

Table 8: Relative shift gain on 3-pixel error based on SGM(CUDA)

---
Test Train | Method | KITTI | ISPRS | EuroSDR | Touluose | Touluose | Toulouse | Enschede | Dublin | Aerial | Aerial
Vaihingen | Vaihingen | UMBRA | Metropole | Metropole(B/H) | City | (real) | (all)
ISPRS | MC-CNN |  2.45 |  7.50 |  2.23 |  2.82 |  0.21 |  4.64 |  -0.75 |  3.41 |  1.80 |  4.23
EfficientDeep |  -0.84 |  4.58 |  0.92 |  0.47 |  -0.66 |  -0.02 |  -4.41 |  0.36 |  0.41 |  1.23
PSMNet |  -19.30 |  10.63 |  6.41 |  -0.61 |  2.75 |  4.46 |  2.47 |  3.93 |  5.14 |  9.10
Deeppruner |  -16.06 |  9.88 |  5.77 |  -1.79 |  2.39 |  3.11 |  -2.60 |  4.17 |  5.42 |  8.69
HRS Net |  -2.07 |  6.93 |  6.93 |  -4.87 |  -0.16 |  1.30 |  0.41 |  -1.23 |  1.72 |  5.23
GANet |  -4.93 |  10.70 |  6.95 |  -1.44 |  2.31 |  3.26 |  1.82 |  1.36 |  3.05 |  8.20
LEAStereo |  -7.67 |  7.70 |  5.58 |  0.55 |  4.45 |  4.79 |  6.03 |  4.46 |  5.08 |  5.96
EuroSDR | MC-CNN |  3.26 |  5.99 |  7.87 |  3.72 |  4.93 |  5.39 |  4.91 |  4.07 |  5.82 |  6.60
EfficientDeep |  -1.24 |  0.92 |  4.28 |  -1.83 |  0.62 |  0.51 |  3.78 |  0.45 |  0.32 |  0.32
PSMNet |  -15.70 |  3.97 |  10.31 |  -3.73 |  4.29 |  5.82 |  2.72 |  4.99 |  5.56 |  8.79
Deeppruner |  -7.82 |  2.63 |  10.32 |  -4.25 |  4.72 |  4.99 |  3.22 |  6.06 |  4.47 |  9.10
HRS Net |  -4.51 |  -3.50 |  7.14 |  -6.12 |  1.05 |  1.61 |  0.58 |  0.43 |  4.34 |  4.23
GANet |  -5.26 |  5.31 |  11.57 |  -2.43 |  5.42 |  6.88 |  3.97 |  5.15 |  6.24 |  7.92
LEAStereo |  -11.78 |  3.20 |  6.77 |  -1.79 |  3.21 |  4.39 |  5.38 |  4.07 |  4.53 |  5.21
UMBRA | MC-CNN |  5.62 |  7.19 |  4.66 |  10.17 |  5.60 |  6.87 |  3.21 |  5.92 |  6.30 |  8.80
EfficientDeep |  5.35 |  6.98 |  1.78 |  9.13 |  5.55 |  5.83 |  1.10 |  4.10 |  5.22 |  7.39
PSMNet |  -16.84 |  6.88 |  6.40 |  16.61 |  7.45 |  7.48 |  4.24 |  9.83 |  9.23 |  11.90
Deeppruner |  -12.20 |  4.67 |  6.78 |  15.18 |  9.11 |  5.88 |  3.81 |  8.82 |  8.98 |  11.91
HRS Net |  -0.20 |  2.06 |  0.77 |  12.33 |  2.06 |  3.03 |  3.27 |  4.29 |  7.01 |  10.42
GANet |  5.35 |  8.46 |  6.42 |  14.84 |  8.33 |  6.18 |  5.25 |  9.25 |  9.53 |  11.24
LEAStereo |  -11.05 |  7.56 |  5.83 |  13.84 |  6.24 |  6.12 |  9.73 |  9.43 |  8.47 |  9.55
Metropole | MC-CNN |  -2.44 |  -0.41 |  0.64 |  -0.51 |  1.32 |  1.73 |  -0.04 |  -1.39 |  0.92 |  1.94
EfficientDeep |  -1.23 |  -2.58 |  -0.13 |  -1.34 |  0.15 |  -0.02 |  -1.34 |  -0.63 |  -0.63 |  -0.35
PSMNet |  -19.59 |  0.48 |  1.77 |  -1.84 |  5.16 |  4.91 |  0.99 |  1.23 |  3.32 |  3.32
Deeppruner |  -10.21 |  -2.54 |  1.85 |  -3.36 |  5.02 |  4.86 |  -0.28 |  2.23 |  2.28 |  3.39
HRS Net |  -4.44 |  -2.06 |  -0.47 |  -5.38 |  1.66 |  2.31 |  -0.64 |  -0.83 |  0.52 |  1.02
GANet |  -1.23 |  1.54 |  2.81 |  -0.83 |  4.25 |  4.70 |  1.01 |  2.05 |  2.35 |  3.65
LEAStereo |  -11.44 |  1.01 |  1.63 |  -0.42 |  2.59 |  2.31 |  2.17 |  1.76 |  1.92 |  2.23
Metropole(B/H) | MC-CNN |  1.21 |  2.97 |  3.45 |  2.45 |  6.50 |  7.79 |  3.21 |  1.31 |  5.26 |  10.81
EfficientDeep |  1.94 |  0.42 |  3.64 |  1.81 |  4.76 |  7.77 |  8.04 |  3.98 |  4.73 |  4.51
PSMNet |  -17.62 |  5.46 |  6.85 |  1.44 |  12.20 |  12.67 |  7.09 |  7.59 |  9.64 |  10.95
Deeppruner |  -9.74 |  2.53 |  7.18 |  0.21 |  11.40 |  12.03 |  5.95 |  6.95 |  8.59 |  9.69
HRS Net |  -1.22 |  0.18 |  4.85 |  -1.68 |  5.70 |  8.15 |  3.99 |  2.70 |  5.85 |  6.66
GANet |  -0.94 |  4.96 |  7.77 |  3.25 |  11.40 |  11.80 |  6.11 |  6.26 |  8.73 |  9.55
LEAStereo |  -12.47 |  3.98 |  6.03 |  2.22 |  5.42 |  6.22 |  6.58 |  5.92 |  6.72 |  6.03
Enschede | MC-CNN |  2.03 |  2.02 |  5.20 |  -0.06 |  -0.12 |  1.76 |  6.00 |  4.00 |  5.04 |  5.62
EfficientDeep |  2.52 |  2.61 |  4.11 |  0.97 |  2.89 |  3.07 |  4.16 |  3.05 |  3.18 |  3.56
PSMNet |  -23.97 |  8.30 |  8.31 |  -1.60 |  8.10 |  8.21 |  11.13 |  6.85 |  5.98 |  10.51
Deeppruner |  -18.05 |  6.72 |  7.98 |  -1.19 |  7.71 |  7.07 |  10.88 |  7.33 |  10.49 |  9.87
HRS Net |  1.53 |  4.22 |  5.67 |  -3.06 |  6.14 |  5.98 |  8.32 |  3.48 |  5.46 |  7.08
GANet |  -8.26 |  7.68 |  9.26 |  -0.71 |  8.27 |  7.81 |  10.88 |  6.20 |  7.26 |  8.87
LEAStereo |  -15.35 |  8.26 |  7.94 |  2.03 |  6.82 |  6.98 |  7.76 |  6.72 |  7.76 |  8.08
DublinCity | MC-CNN |  1.60 |  -1.18 |  2.96 |  -1.12 |  0.55 |  2.02 |  3.16 |  5.25 |  2.04 |  4.90
EfficientDeep |  0.51 |  -4.33 |  2.76 |  -3.72 |  1.47 |  1.53 |  1.63 |  4.10 |  2.40 |  3.79
PSMNet |  -16.85 |  1.27 |  4.35 |  -12.84 |  -17.20 |  -0.48 |  3.70 |  9.81 |  7.60 |  9.01
Deeppruner |  -20.18 |  3.01 |  5.54 |  -10.40 |  -10.33 |  -10.14 |  3.49 |  9.86 |  6.13 |  9.13
HRS Net |  1.79 |  0.66 |  3.86 |  -5.73 |  -5.45 |  -0.32 |  4.29 |  7.88 |  5.22 |  7.28
GANet |  -11.00 |  -2.90 |  4.73 |  -1.15 |  0.32 |  -6.74 |  6.39 |  9.00 |  6.85 |  8.47
LEAStereo |  -13.36 |  3.84 |  5.71 |  2.24 |  3.11 |  4.26 |  7.12 |  7.45 |  5.17 |  7.16

##  6 Visual assessment

While $N$-pixel error is a statistical metric giving a global evaluation,
dense matching results are locally influenced by the scene. When the scene is
complex, errors can show different patterns on various scene types (buildings,
ground, vegetation,…). In order to give a visual assessment of the methods, we
show some results in a building scene and a vegetable scene of the ISPRS
Vaihingen dataset Figure 10. The results on the other 5 datasets are provided
in the appendix. Because the ground truth is sparse, the valid evaluated pixel
is sparse too but we have interpolated it with the nearest point in order to
facilitate the interpretation. The disparity maps are visualized using an
ambient occlusion shading implemented in MicMac. The result is shown in Figure
11 and Figure 12.

(a) Left image

(b) GT dispairty

(c) Left image

(d) GT dispairty

Figure 10: Tree and building extracts from the ISPRS Vaihingen dataset.

On buildings, the disparity is usually smooth on the roof, with a large
discontinuity on the boundary. The error map and disparity map are shown in
Figure 11 and the left image in Figure 10(a). We see that the GraphCuts-based
methods produce a smooth disparity which is good on the roofs but bad on the
discontinuity. Conversely, after fine-tuning, DL-based methods perform well on
the building’s boundary (disparity discontinuity). The pre-trained model is
not good, because the scene is quite different from KITTI, among them, the
DeepPruner gives the worst result, and the fine-tune methods are better than
the traditional methods.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 11: Error map and disparity visualization on building area for ISPRS
Vaihingen dataset.

The vegetable area is different from the man-made structure, the disparity of
vegetable changes frequently, but not large, the error map and disparity map
visualization is shown in Figure 12, and the left image is shown in Figure
10(c). The pre-trained result of DeepPruner is poor, which means that
DeepPruner is highly dependent on the training dataset. Because there is a
plane constraint in the GraphCuts method, the result are not satisfactory on
vegetable boundaries. The end-to-end methods give a smooth result, especially
for the bush area. The PSM net, DeepPruner, and GANet give a good result.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 12: Error map and disparity visualization on the vegetable area for
ISPRS Vaihingen dataset.

##  7 Conclusion

In this paper, we propose a method to generate training data for DL-based
stereo-dense matching methods, and produce a new big dataset from the aerial
dataset. The state-of-the-art methods are evaluated on this dataset,
containing both traditional and learning base methods. Dataset shift is an
important research point for the application because it is impossible to
obtain training data for every application. Several factors in dataset shift
are investigated, for example different resolutions, different scenes, image
color and brightness, and base to height ratio. We summary a conclusion from
the experiment :

  * 1.

ML depends on the training dataset, especially the end-to-end methods. Using
DL-based method can improve performance significantly.

  * 2.

For dataset shift, models from high-resolution datasets have a better
performance than low resolution.

  * 3.

Using the same area to train the model can improve performance, even with
different resolutions.

  * 4.

Hybrid methods generalize better than end-to-end methods, on the other handle,
end-to-end methods improve more when the training and testing data is same
type.

  * 5.

The transferability is not symmetric, and depends on the network structure.

  * 6.

Pre-trained methods work better in building areas than in tree areas. Using
aerial data to do the fine-tuning can improve performance in the building
areas with discontinuity.

In the training dataset generation, because the image and LiDAR are not
acquired at the same time, building changes can be removed easily, but
vegetable change is difficult to remove. Use the filtering to remove the low-
accuracy stereo pairs, unfortunately, some positive samples are removed. On
one handle, the difficult matching areas are removed, for example reflective
areas, on the other hand, it is difficult to know whether the vegetable is
changed or not, there is another possibility just due to matching difficulty.
Even training can improve the performance in the vegetable areas, but the
vegetable areas are removed because of the low evaluation. Semantic
information can be utilized as supplement information in future work. Dataset
shift is an important and difficult topic for dense matching in photogrammetry
applications, improving the performance of dataset shift is future work. It is
difficult to obtain the image and LiDAR at the same time, self-supervised or
unsupervised learning is also a promising solution.

##  8 Acknowledgments

This work was fully funded by the AI4GEO project.

##  Appendix A Introduction

Below we provide additional experiments, including complementary metrics such
as 1-pixel and the average errors, as well as the complete data-shift
comparison of all of the investigated DL-based dense image matching methods.
Visual assessments are also given for all of the studied dataset and methods.

##  Appendix B  Quantitative evaluation and discussion

###  B.1 1-pixel and average error metrics

We report the 1-pixel error (cf. Table 9) and the average error metrics (cf.
Table 10) because they reflect the image matching accuracy and the robustness
of a method, respectively. The color-coding in the tables ranges from red to
green, where red corresponds to worse and green to better performance in
relation to SGM (CUDA) which we consider as the baseline. The training
datasets and methods are in rows, and the testing datasets are in columns. In
other words, an entire column indicates the transferability of a dataset,
while an entire row indicates the transferability of a method. In particular,
for the 1-pixel error, we note that:

  * 1.

Base-to-height ratio ($B/H$) influences the performance. Compare Toulouse
Metropole ($B/H$) and Toulouse Metropole, the transferability of the model
trained on Toulouse Metropole ($B/H$) is sometimes worse than that of the
model trained on Toulouse Metropole.

  * 2.

For Toulouse Metropole, all rows are in red and orange color, which means
training methods can not improve compare to SMG(CUDA) on the small ($B/H$)
dataset.

  * 3.

HRS Net method underperforms in dataset shift, but GANet and LEAStereo
generalize better than the other end-to-end methods.

  * 4.

Except for the model KITTI, nearly all the rows of Enschede and Toulouse UMBRA
are in positive color, which means training on aerial data can improve
performance.

  * 5.

For the columns of Vaihingen EuroSDR and DublinCity, most values are positive
compared to the other columns, this means these two datasets are good for
training.

Contrary to the 1-pixel error, the average error (cf. Table 10) shows less
variability in the relative shift gain across the tested methods. In
particular, we note that:

  * 1.

Toulouse Metropole ($B/H$) gives a better result than Toulouse Metropole.

  * 2.

For training on the aerial dataset, most values are positive, which means
training on aerial data can improve the dense matching

  * 3.

End-to-end methods are better than hybrid methods

  * 4.

The large negative values are in DublinCity, which indicates training on low-
resolution data can not improve the performance when deployed on high-
resolution data.

  * 5.

Globally, PSMNet sometimes performs better than the other DL-based methods,
but GANet and LEAStereo are better at transferability to new scenes.

Table 9: Relative shift gain on 1-pixel error based on SGM(CUDA)

---
Test Train | Method | KITTI | ISPRS | EuroSDR | Touluose | Touluose | Toulouse | Enschede | Dublin | Aerial | Aerial
Vaihingen | Vaihingen | UMBRA | Metropole | Metropole(B/H) | City | (real) | (all)
ISPRS | MC-CNN |  -9.49 |  15.60 |  -9.41 |  -3.88 |  -16.54 |  -4.82 |  -18.08 |  -2.41 |  -10.77 |  -2.23
EfficientDeep |  -8.00 |  9.64 |  -5.86 |  -7.92 |  -10.25 |  -10.02 |  -12.30 |  -4.37 |  -7.04 |  -6.15
PSMNet |  -76.25 |  32.23 |  7.01 |  -8.78 |  -13.14 |  -2.03 |  -0.07 |  1.27 |  3.49 |  14.44
Deeppruner |  -45.46 |  29.05 |  1.72 |  -9.29 |  -4.37 |  -13.81 |  -5.22 |  1.76 |  3.79 |  15.88
HRS Net |  -15.37 |  17.12 |  17.12 |  -17.23 |  -10.88 |  -6.96 |  -9.20 |  -12.25 |  -6.00 |  3.59
GANet |  -23.00 |  32.82 |  8.72 |  -13.93 |  -3.27 |  -5.37 |  -5.18 |  -8.20 |  -5.56 |  14.68
LEAStereo |  -35.56 |  15.69 |  5.51 |  -2.86 |  1.53 |  2.40 |  6.10 |  3.30 |  2.33 |  5.36
EuroSDR | MC-CNN |  -3.43 |  9.85 |  4.62 |  2.37 |  6.56 |  -3.14 |  -0.74 |  0.58 |  5.26 |  4.22
EfficientDeep |  -10.20 |  -5.36 |  2.42 |  -10.51 |  -7.06 |  -8.25 |  2.56 |  -3.66 |  -6.71 |  -6.71
PSMNet |  -76.11 |  -1.48 |  18.30 |  -8.65 |  -12.87 |  0.98 |  3.89 |  7.50 |  8.14 |  14.11
Deeppruner |  -26.33 |  1.22 |  18.19 |  -19.02 |  2.50 |  -17.87 |  5.94 |  9.77 |  6.22 |  14.57
HRS Net |  -21.36 |  -22.30 |  5.01 |  -20.25 |  -10.71 |  -14.88 |  -9.69 |  -9.17 |  -0.89 |  -1.04
GANet |  -14.55 |  4.15 |  24.23 |  -9.23 |  6.65 |  0.97 |  7.80 |  7.14 |  8.01 |  13.80
LEAStereo |  -40.21 |  1.04 |  8.27 |  -7.07 |  0.39 |  1.71 |  6.22 |  4.39 |  3.64 |  5.67
UMBRA | MC-CNN |  4.60 |  3.61 |  1.83 |  13.31 |  6.92 |  2.94 |  -0.01 |  5.91 |  3.84 |  10.30
EfficientDeep |  3.50 |  7.54 |  0.22 |  13.28 |  5.44 |  3.83 |  -2.71 |  3.41 |  4.24 |  10.62
PSMNet |  -61.01 |  1.79 |  7.58 |  31.33 |  3.34 |  5.93 |  3.57 |  15.48 |  15.53 |  19.99
Deeppruner |  -25.98 |  -4.50 |  9.11 |  24.67 |  14.86 |  2.81 |  5.38 |  14.17 |  15.87 |  21.06
HRS Net |  -11.59 |  -8.22 |  -7.27 |  12.88 |  -8.22 |  -8.13 |  -6.39 |  -1.97 |  3.64 |  10.20
GANet |  3.50 |  14.32 |  12.88 |  27.96 |  11.43 |  2.66 |  8.97 |  15.95 |  17.52 |  19.73
LEAStereo |  -26.42 |  6.76 |  3.57 |  21.96 |  4.64 |  4.14 |  13.69 |  13.56 |  11.22 |  12.84
Metropole | MC-CNN |  -16.80 |  -7.96 |  -11.62 |  -10.88 |  -1.85 |  -5.81 |  -11.77 |  -11.30 |  -12.35 |  -5.60
EfficientDeep |  -11.50 |  -17.51 |  -7.13 |  -10.37 |  -2.77 |  -6.38 |  -11.18 |  -8.27 |  -8.64 |  -8.44
PSMNet |  -85.74 |  -6.81 |  -2.33 |  -10.50 |  3.26 |  3.24 |  -6.91 |  -6.23 |  -2.89 |  -2.89
Deeppruner |  -31.25 |  -12.77 |  -3.85 |  -11.82 |  5.87 |  4.01 |  -7.20 |  -4.25 |  -3.51 |  -0.95
HRS Net |  -20.95 |  -12.63 |  -9.73 |  -19.41 |  -7.87 |  -7.63 |  -13.44 |  -14.31 |  -10.29 |  -10.10
GANet |  -11.50 |  -4.44 |  0.98 |  -11.38 |  4.71 |  1.44 |  -6.48 |  -4.99 |  -2.85 |  1.56
LEAStereo |  -39.57 |  -2.37 |  -3.53 |  -7.38 |  -1.09 |  -3.40 |  -2.94 |  -3.43 |  -2.76 |  -2.91
Metropole(B/H) | MC-CNN |  -4.35 |  0.09 |  -0.31 |  0.92 |  10.29 |  10.50 |  0.19 |  0.59 |  2.77 |  23.03
EfficientDeep |  -2.34 |  -10.70 |  3.85 |  0.16 |  5.98 |  15.21 |  25.06 |  3.82 |  3.18 |  4.34
PSMNet |  -72.41 |  -3.14 |  7.88 |  -0.73 |  18.65 |  21.74 |  5.15 |  7.39 |  9.01 |  14.19
Deeppruner |  -24.31 |  -4.46 |  7.94 |  -4.30 |  13.79 |  21.28 |  5.05 |  7.52 |  8.36 |  12.55
HRS Net |  -10.04 |  -11.72 |  0.20 |  -10.43 |  1.54 |  4.90 |  -1.57 |  -3.76 |  0.27 |  2.21
GANet |  -5.13 |  -3.29 |  9.77 |  1.87 |  14.51 |  18.02 |  5.24 |  7.73 |  9.47 |  11.79
LEAStereo |  -34.58 |  2.56 |  5.37 |  0.45 |  5.94 |  5.27 |  5.98 |  5.80 |  6.54 |  5.37
Enschede | MC-CNN |  2.56 |  2.18 |  8.36 |  4.41 |  5.09 |  4.70 |  12.17 |  9.62 |  9.16 |  12.03
EfficientDeep |  6.74 |  3.11 |  9.05 |  2.67 |  8.53 |  5.31 |  9.99 |  8.75 |  6.95 |  7.66
PSMNet |  -80.23 |  11.96 |  17.79 |  2.59 |  2.90 |  8.00 |  23.89 |  16.75 |  9.68 |  22.32
Deeppruner |  -33.95 |  4.80 |  17.62 |  1.06 |  14.39 |  7.02 |  24.10 |  17.67 |  23.53 |  21.91
HRS Net |  -3.13 |  -10.32 |  6.76 |  -5.86 |  6.17 |  5.43 |  13.36 |  4.36 |  7.83 |  11.29
GANet |  -24.45 |  9.36 |  22.03 |  3.27 |  14.18 |  1.65 |  23.50 |  16.47 |  17.33 |  20.80
LEAStereo |  -49.49 |  13.64 |  14.56 |  8.35 |  11.96 |  11.62 |  16.96 |  16.63 |  16.96 |  17.71
DublinCity | MC-CNN |  2.42 |  -1.21 |  3.95 |  3.49 |  3.82 |  1.30 |  4.36 |  7.79 |  4.62 |  7.48
EfficientDeep |  -1.86 |  -10.44 |  7.35 |  -3.68 |  3.66 |  0.22 |  2.71 |  10.51 |  4.84 |  8.77
PSMNet |  -63.20 |  -17.19 |  12.50 |  -15.15 |  -45.03 |  -10.60 |  11.09 |  26.29 |  16.46 |  22.89
Deeppruner |  -35.15 |  -0.97 |  10.52 |  -15.63 |  -14.56 |  -39.53 |  13.27 |  28.00 |  8.31 |  25.14
HRS Net |  -2.36 |  -9.22 |  2.99 |  -12.40 |  -20.31 |  -26.36 |  4.53 |  16.17 |  0.81 |  14.14
GANet |  -28.69 |  -9.59 |  14.07 |  -2.01 |  -11.03 |  -45.30 |  17.89 |  24.52 |  14.59 |  21.88
LEAStereo |  -30.90 |  -2.91 |  8.02 |  3.13 |  4.30 |  5.55 |  14.68 |  17.06 |  11.23 |  14.77

Table 10: Relative shift gain on average pixel error based on SGM(CUDA)

---
Test Train | Method | KITTI | ISPRS | EuroSDR | Touluose | Touluose | Toulouse | Enschede | Dublin | Aerial | Aerial
Vaihingen | Vaihingen | UMBRA | Metropole | Metropole(B/H) | City | (real) | (all)
ISPRS | MC-CNN |  0.86 |  4.06 |  0.53 |  1.06 |  -0.86 |  2.05 |  -1.22 |  1.32 |  0.29 |  1.83
EfficientDeep |  -0.76 |  2.70 |  0.69 |  0.29 |  -0.58 |  -0.25 |  -2.39 |  -0.02 |  0.20 |  0.71
PSMNet |  -6.67 |  6.92 |  4.53 |  0.22 |  2.63 |  3.40 |  2.33 |  2.96 |  3.74 |  6.00
Deeppruner |  -8.80 |  6.65 |  4.25 |  -1.06 |  2.36 |  2.82 |  -1.05 |  3.28 |  4.06 |  5.91
HRS Net |  0.07 |  5.59 |  5.59 |  -1.93 |  1.84 |  2.65 |  2.06 |  0.77 |  2.79 |  4.59
GANet |  -3.58 |  6.88 |  4.77 |  -0.25 |  2.04 |  2.88 |  2.19 |  1.52 |  2.75 |  5.54
LEAStereo |  -3.73 |  5.35 |  4.13 |  0.99 |  3.54 |  3.79 |  4.37 |  3.30 |  3.76 |  4.35
EuroSDR | MC-CNN |  1.38 |  2.76 |  4.15 |  1.71 |  2.17 |  2.57 |  2.35 |  1.66 |  2.79 |  3.39
EfficientDeep |  -1.14 |  0.33 |  2.67 |  -1.07 |  0.20 |  0.30 |  2.22 |  0.02 |  0.17 |  0.17
PSMNet |  -6.95 |  2.70 |  6.53 |  -1.75 |  3.63 |  4.34 |  2.53 |  3.38 |  4.07 |  5.74
Deeppruner |  -3.97 |  1.59 |  6.58 |  -1.75 |  3.44 |  3.59 |  2.79 |  4.19 |  3.96 |  5.99
HRS Net |  -1.73 |  -0.71 |  5.52 |  -1.70 |  2.55 |  2.93 |  2.32 |  1.85 |  4.23 |  4.16
GANet |  -3.45 |  3.48 |  7.08 |  -0.11 |  3.85 |  4.95 |  3.62 |  3.59 |  4.40 |  5.20
LEAStereo |  -7.32 |  2.17 |  4.68 |  -0.34 |  2.48 |  3.47 |  3.83 |  2.97 |  3.34 |  3.75
UMBRA | MC-CNN |  2.34 |  3.73 |  1.30 |  5.08 |  1.71 |  3.08 |  0.91 |  2.37 |  2.53 |  4.07
EfficientDeep |  2.46 |  3.52 |  0.08 |  4.73 |  2.17 |  2.79 |  -0.12 |  1.70 |  2.54 |  3.63
PSMNet |  -6.88 |  4.36 |  4.34 |  10.19 |  5.03 |  5.05 |  3.25 |  6.52 |  6.02 |  7.56
Deeppruner |  -7.14 |  3.54 |  4.68 |  9.55 |  5.97 |  4.03 |  2.58 |  5.97 |  5.90 |  7.72
HRS Net |  1.55 |  2.33 |  1.97 |  8.44 |  2.33 |  3.56 |  3.86 |  4.33 |  5.61 |  7.40
GANet |  2.46 |  5.35 |  3.99 |  9.05 |  5.20 |  4.48 |  3.81 |  5.99 |  5.78 |  7.18
LEAStereo |  -6.44 |  5.19 |  4.22 |  8.69 |  4.07 |  4.14 |  6.14 |  5.88 |  5.44 |  6.06
Metropole | MC-CNN |  -2.93 |  -1.78 |  -1.01 |  -2.10 |  -0.49 |  -0.26 |  -1.44 |  -2.67 |  -0.47 |  0.07
EfficientDeep |  -2.00 |  -2.75 |  -1.05 |  -1.87 |  -1.00 |  -1.02 |  -2.03 |  -1.63 |  -1.48 |  -1.20
PSMNet |  -8.19 |  -0.24 |  0.83 |  -1.42 |  4.39 |  4.18 |  1.30 |  0.93 |  2.57 |  2.57
Deeppruner |  -5.19 |  -2.24 |  1.13 |  -2.47 |  4.38 |  4.25 |  -0.24 |  1.93 |  1.62 |  3.05
HRS Net |  -3.54 |  -1.01 |  -0.29 |  -2.91 |  2.61 |  3.16 |  0.70 |  0.65 |  1.30 |  2.02
GANet |  -2.00 |  0.89 |  1.94 |  -0.24 |  3.74 |  4.09 |  1.61 |  1.65 |  1.12 |  3.19
LEAStereo |  -6.01 |  0.23 |  0.92 |  -0.13 |  2.32 |  2.00 |  1.52 |  1.05 |  1.00 |  1.54
Metropole(B/H) | MC-CNN |  0.06 |  0.95 |  1.63 |  0.39 |  3.89 |  4.95 |  1.59 |  -0.20 |  2.98 |  7.55
EfficientDeep |  0.89 |  0.44 |  2.24 |  0.81 |  2.99 |  5.69 |  6.09 |  2.16 |  2.90 |  2.86
PSMNet |  -5.98 |  4.41 |  5.54 |  1.24 |  10.97 |  11.31 |  6.71 |  6.70 |  8.12 |  9.65
Deeppruner |  -5.21 |  2.15 |  6.00 |  0.77 |  10.24 |  10.85 |  5.42 |  6.23 |  7.11 |  8.80
HRS Net |  -0.52 |  1.54 |  5.02 |  1.20 |  7.20 |  9.15 |  5.20 |  4.07 |  6.37 |  7.65
GANet |  -0.62 |  4.30 |  6.67 |  3.64 |  9.95 |  10.58 |  6.25 |  5.47 |  6.58 |  8.62
LEAStereo |  -7.11 |  3.08 |  5.23 |  2.37 |  5.41 |  6.16 |  5.86 |  4.92 |  5.46 |  5.23
Enschede | MC-CNN |  0.32 |  -0.42 |  2.50 |  -1.86 |  -2.35 |  -0.69 |  3.16 |  1.59 |  2.37 |  2.91
EfficientDeep |  1.11 |  1.36 |  2.34 |  0.27 |  1.25 |  1.59 |  2.41 |  1.47 |  1.68 |  2.03
PSMNet |  -11.05 |  5.68 |  5.77 |  -0.62 |  6.28 |  6.21 |  8.44 |  4.89 |  4.70 |  7.61
Deeppruner |  -10.67 |  4.66 |  5.76 |  -0.56 |  5.88 |  5.30 |  8.30 |  5.37 |  8.07 |  7.39
HRS Net |  1.62 |  3.60 |  4.87 |  -1.08 |  5.61 |  5.59 |  7.06 |  3.55 |  5.04 |  6.06
GANet |  -6.65 |  5.59 |  6.54 |  0.04 |  6.11 |  5.96 |  8.10 |  4.40 |  5.33 |  6.47
LEAStereo |  -8.56 |  5.97 |  5.72 |  1.63 |  5.18 |  5.30 |  5.55 |  4.76 |  5.55 |  5.76
DublinCity | MC-CNN |  -0.15 |  -3.31 |  0.90 |  -3.25 |  -1.51 |  -0.09 |  1.07 |  2.76 |  -0.25 |  2.46
EfficientDeep |  -0.59 |  -4.67 |  0.89 |  -4.57 |  -0.13 |  -0.02 |  0.32 |  2.02 |  0.73 |  1.78
PSMNet |  -8.89 |  0.90 |  2.61 |  -10.92 |  -15.30 |  -1.23 |  2.42 |  6.66 |  5.24 |  6.12
Deeppruner |  -16.47 |  1.78 |  3.79 |  -9.14 |  -9.41 |  -8.99 |  2.26 |  6.70 |  4.15 |  6.22
HRS Net |  1.66 |  0.77 |  3.00 |  -4.45 |  -3.35 |  0.80 |  3.44 |  5.66 |  3.96 |  5.26
GANet |  -8.63 |  -2.28 |  2.96 |  -0.49 |  0.84 |  -1.01 |  4.38 |  6.15 |  4.80 |  5.77
LEAStereo |  -8.07 |  2.84 |  4.12 |  1.81 |  2.25 |  3.13 |  4.97 |  5.17 |  3.51 |  5.00

###  B.2 Dataset shift comparisons across tested methods

##### MC-CNN

The performance of the MC-CNN method varies slightly across datasets, even
when training on the mobile mapping data KITTI and testing on aerial datasets.
We attribute this to the fact that it is a hybrid method, where the DL-
extracted features are followed by the traditional SGM step. All in all, we
can draw the following general conclusions:

  * 1.

The higher resolution of the images, the better the models trained on them
(for EuroSDR Vaihingen in Figure 13(b) the best is the model trained on ISPRS
Vaihingen)

  * 2.

Augmenting the training set with other different datasets increases the error
variance (see Aerial real for ISPRS Vaihingen in Figure 13(a));

  * 3.

MC-CNN is on par with SGM(CUDA) on the small $B/H$ images of the Toulouse
Metropole in Figure 13(c).

  * 4.

MC-CNN performance on large $B/H$ outperforms the classical SGM(CUDA) (see the
Enschede dataset in Figure 13(e)).

  * 5.

The performance of pretrained model(KITTI) varies, for example, in dataset
Toulouse Metropole, pretained model performs worst, but in dataset DublinCity
shown in Figure 13(f), pretrained model is better than SGM(CUDA).

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse Metropole

(d) Result on Toulouse UMBRA

(e) Result on Enschede

(f) Result on DublinCity

Figure 13: Training from multi-dataset using MC-CNN.

##### EfficientDeep

Because EfficientDeep is also a hybrid method, the results are similar to that
of MC-CNN (see Figure 14). We draw the following generic conclusion from our
experiments:

  * 1.

The variance of training data influences the performance of the model, as
shown in Figure 14(a), the performance of the model of Aerial(all) is far
lower than that of the model of ISPRS Vaihingen, the model of EuroSDR
Vaihingen is a little better than SGM(CUDA).

  * 2.

Even though EfficientDeep is similar to MC-CNN, different network performs
differently, for example, for dataset EuroSDR Vaihingen, compared to
SGM(CUDA), fine-trained models of MC-CNN improve the performance, but fine-
trained models of EfficientDeep don’t improve (see Figure 13(b) and Figure
14(b)).

  * 3.

The higher image resolution has the more precise result. In Toulouse UMBRA
(cf. Figure 14(c)) both ISPRS Vaihingen and DublinCity perform best.

  * 4.

For dataset Toulouse UMBRA and Enschede shown in Figures 14(d) and 14(e),
using EfficientDeep to learning the feature can improve the result compared to
SGM(CUDA).

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse Metropole

(d) Result on Toulouse UMBRA

(e) Result on Enschede

(f) Result on DublinCity

Figure 14: Training from multi-dataset using Efficient DL feature.

##### DeepPruner

Unlike MC-CNN or EfficientDeep, DeepPruner is an end-to-end method and the
results show different tendencies (cf. Figure 15). We conclude that:

  * 1.

The KITTI-trained models perform worst of all, while the models trained on
Aerial(real) produce good results overall across all datasets, slightly worse
than training on the same dataset.

  * 2.

Augmenting the training with different datasets slightly deteriorates the
results compared to a model trained and tested on images coming from the same
acquisition. However, a universal model trained on heterogeneous scenes still
remains a good compromise and an interesting solution toward a one-for-all
model. Compare the Aerial(all) (which includes ISPRS Vaihingen) with the
training using exclusively the ISPRS Vaihingen in Figure 15(a).

  * 3.

DeepPruner produces smoother results than the traditional SGM (CUDA). Note
that the better performance compare to SGM (CUDA) on large pixel error in
Figure 15(c).

  * 4.

Transferring the DeepPruner models to datasets different from their training
samples deteriorate the matching performance. For example, the model from
Toulouse Metropole works better on Toulouse UMBRA than on the others.

  * 5.

For difficult dataset where SGM(CUDA) performs badly, fine-trained models of
textitDeepPruner give great improvement, for example Toulouse UMBRA and
Enschede.

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse Metropole

(d) Result on Toulouse UMBRA

(e) Result on Enschede

(f) Result on DublinCity

Figure 15: Training from multi-dataset using DeepPruner.

##### HRS net

This architecture and performance is similar to that of DeepPruner, the main
difference is that HRS net uses hierarchical feature volume decoder to
simulate a coarse to fine strategy. Based on Figure 16 we conclude the
following:

  * 1.

For HRS net, the model of KITTI is as good as the other aerial models,
indicating that HRS net is less dependent on the type of the training data.

  * 2.

HRS net produces a smooth resul (see Figure 16(c)) with good performance on
large pixel errors.

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse Metropole

(d) Result on Toulouse UMBRA

(e) Result on Enschede

(f) Result on DublinCity

Figure 16: Training from multi-dataset using HRS net.

##### GANet

GANet focuses on 3D cost volume processing by adding guided aggregation layer.
Based on Figure 17 we conclude the following:

  * 1.

The KITTI model performs worst than the other models, meaning GANet is
sensitive to the choice of training data.

  * 2.

It produces smooth results, with low scores on 1-pixel error and high scores
on large pixel errors (see Figure 17(a))

  * 3.

Overall GANet performs well and has good transferability, most models are
better than SGM(CUDA).

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse Metropole

(d) Result on Toulouse UMBRA

(e) Result on Enschede

(f) Result on DublinCity

Figure 17: Training from multi-dataset using GANet.

##### LEAStereo

LEAStereo is an end-to-end method based on Neural Architecture Search, which
tries to learn neural network architectures automatically. We present the
results in Figure 18 and conclude the following:

  * 1.

It highly depends on the training data, models from KITTI perform worst, and
training on aerial data leads to a significant improvement.

  * 2.

It has good transferability among the aerial dataset, these lines are quite
close.

  * 3.

It produces smooth results. As shown in Figure 18(c), for the 1-pixel error,
the fine-trained models do not have any advantage, but at large pixel error,
the training models on aerial data outperform the SGM(CUDA).

(a) Result on ISPRS Vaihingen

(b) Result on EuroSDR Vaihingen

(c) Result on Toulouse Metropole

(d) Result on Toulouse UMBRA

(e) Result on Enschede

(f) Result on DublinCity

Figure 18: Training with multi-dataset using LEAStereo net.

###  B.3 Discussion of Dataset shift

In general, we can make a conclusion:

  * 1.

The transferability of hybrid methods is better than that of end-to-end
methods on different types of datasets. Training on KITTI is not far from
training from aerial data (see Figure 13 and Figure 14). But for the same type
of dataset, in the paper for the aerial scene, the end-to-end methods have
better transferability.

  * 2.

End-to-end methods highly depend on the training dataset, training on KITTI
performs poorly when tested on aerial data.

  * 3.

DL-base methods usually give better results on high pixel error.

  * 4.

The transferability is not symmetric, DublinCity works well on Toulouse data,
but Toulouse data performs less well on DublinCity(cf Table 10).

  * 5.

The transferability depends on the network structure, the scene, the
resolution, and the base to height ratio, generally, DeepPruner performs
better than  HRS net, but sometimes it gives bad results, for example,
Toulouse data on DublinCity in Figure 15(f).

##  Appendix C Visual assessment

The pixel error metric describes the precision and robustness of a method
through a global score. Visual assessment, on the other hand, gives insights
into the quality of depth prediction performance across different surface
types, e.g. in built-up areas or in natural environments. In the following, we
reveal the performance of the other 5 datasets (the 1st being already covered
in the main paper) over buildings (cf. Figure 19) and vegetation (cf. Figure
20). Table 11 sums up the methods’ performance at selected scene’s objects
such as small buildings, shadows, discontinuities, leaves and branches.

(a)EuroSDR Vaihingen

(b)Toulouse Metropole

(c)Toulouse UMBRA

(d)Enschede

(e)DublinCity

Figure 19: Examples for the building area for each dataset. The left image is
shown in the first row, and the disparity is shown in the second row in jet
color, from blue to red indicating the disparity from small to large.

(a)EuroSDR Vaihingen

(b)Toulouse Metropole

(c)Toulouse UMBRA

(d)Enschede

(e)DublinCity

Figure 20: Examples for the tree area for each dataset. The left image is
shown in the first row, and the disparity is shown in the second row in jet
color, from blue to red indicating the disparity from small to large. Table
11: Summary of performance of reconstruction in visual assessment

---
Method | small building | shadow | discontinuity | leaf | branch | pre-trained | trained
SGM(CUDA) | ✓ | ✕ | ✕ | ✕ | ✕ | – | –
MICMAC | ✓ | ✕ | ✕ | ✓ | ✕ | – | –
GraphCuts | ✕ | ✕ | ✕ | ✕ | ✕ | – | –
CBMV(SGM) | ✓ | ✓ | ✕ | ✓ | ✕ | – | ✓
CBMV(GraphCuts) | ✓ | ✓ | ✕ | ✕ | ✕ | – | ✓
MC-CNN(KITTI) | ✕ | ✓ | ✕ | ✕ | ✕ | ✓ | –
DeepFeature(KITTI) | ✕ | ✓ | ✕ | ✕ | ✕ | ✓ | –
MC-CNN | ✓ | ✓ | ✕ | ✕ | ✕ | – | ✓
DeepFeature | ✓ | ✓ | ✕ | ✕ | ✕ | – | ✓
PSM Net(KITTI) | ✕ | ✕ | ✕ | ✕ | ✕ | ✓ | –
HRS Net(KITTI) | ✕ | ✕ | ✕ | ✕ | ✕ | ✓ | –
DeepPruner(KITTI) | ✕ | ✕ | ✕ | ✕ | ✕ | ✓ | –
GANet(KITTI) | ✕ | ✕ | ✕ | ✕ | ✕ | ✓ | –
LEAStereo(KITTI) | ✕ | ✕ | ✕ | ✕ | ✕ | ✓ | –
PSM Net | ✓ | ✕ | ✓ | ✓ | ✕ | – | ✓
HRS Net | ✓ | ✓ | ✕ | ✓ | ✕ | – | ✓
DeepPruner | ✓ | ✓ | ✕ | ✓ | ✕ | – | ✓
GANet | ✓ | ✓ | ✓ | ✓ | ✕ | – | ✓
LEAStereo | ✓ | ✓ | ✕ | ✓ | ✕ | – | ✓

##### EuroSDR Vaihingen

Reconstruction of building contours (cf. Figure 21) or trees (cf. Figure 22)
from models pre-trained on KITTI performs poorly. We believe this is due to
the large discontinuities present in this dataset. Fine-tuning significantly
improves the results.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM Net(KITTI)

HRS Net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM Net

HRS Net

DeepPruner

GANet

LEAStereo

Figure 21: Error map and disparity visualization on building area for EuroSDR
Vaihingen.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM Net(KITTI)

HRS Net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM Net

HRS Net

DeepPruner

GANet

LEAStereo

Figure 22: Error map and disparity visualization on tree area for EuroSDR
Vaihingen.

##### Toulouse Metropole

Over man-made objects all the methods perform well, even on pre-trained
models. Fine-tuning further enhances the results, especially at
discontinuities (cd. Figure 23). As far as the trees go, both the traditional
SGM and DL-based methods on pre-trained models produce mediocre outcomes.
Fine-tuning helps to recover the tree canopies (cf. Figure 24). PSMNet
performs best of all.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 23: Error map and disparity visualization on building area for Toulouse
Metropole.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 24: Error map and disparity visualization on tree area for Toulouse
Metropole.

##### Toulouse UMBRA

On large man-made objects (with the exception of discontinuities) all the
methods perform relatively well (cf. Figure 25). We also observe the following
tendencies:

  * 1.

GraphCuts-based methods outrun the SGM-based methods.

  * 2.

PSM net performs poorly in dark shadow areas.

  * 3.

LEAStereo performs best among the DL-based methods.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GAnet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GAnet

LEAStereo

Figure 25: Error map and disparity visualization on building area for Toulouse
UMBRA.

Another example from the tree area is shown in Figure 26, all the trees are
flourishing, so the large disparity discontinuity is not frequent in tree
areas, but there are big error pixels for all the methods, SGM based methods
have big errors on the disparity discontinuity areas, even with DL-based
features. For the end-to-end methods, pre-trained models work badly, even
after fine-tuning, PSM net and DeepPruner also have a lot of errors. After
fine-tuning, GAnet obtains the best result, and the other end-to-end methods
produce a smooth result.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GAnet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GAnet

LEAStereo

Figure 26: Error map and disparity visualization on tree area for Toulouse
UMBRA.

##### Enschede

The traditional SGM-based methods perform well on man-made objects, and so
does the CBMV(GraphCuts) after re-learning the features. The fine-tuned end-
to-end methods are equally good, in particular the PSM net and GAnet (cf.
Figure 27). On vegetated areas, contrary to the end-to-end methods, the hybrid
methods perform well even on pre-trained models (cf. Figure 28). We observe
that disparities predicted from pre-trained models of PSM net and HRS net
differ in quality which points to their sensitivity to the training data.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 27: Error map and disparity visualization on building area for Enschede
dataset.

For the result, on the bottom left, we observe an inconsistency between the
scene represented in ground truth data and the images. This is more likely due
to the temporal difference between the two acquisitions, i.e. LiDAR and
images.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

DeepFeature(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

DeepFeature

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 28: Error map and disparity visualization on tree area for Enschede
dataset.

##### DublinCity

All in all in build-up areas the traditional methods show good results, while
among the DL-based methods, the pertained model of DeepPruner is the worst,
after fine-tuning, GANet is the best (cf. Figure 29). DublinCity images are
high resolution and thus the buildings’ discontinuities span large
disparities. This in turn affects the performance of GANet. The performance in
the vegetated area differs depending on whether the trees are leafy and
leafless (cf. Figure 19). Neither of the methods is capable of reconstructing
the leafless trees, even after fine-tuning, for the tree leafy, the training
model improves much(cf. Figure 30).

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

EfficientDeep(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

EfficientDeep

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 29: Error map and disparity visualization on building area for
DublinCity dataset.

MICMAC

SGM(CUDA)

GraphCuts

CBMV(SGM)

CBMV(GraphCuts)

MC-CNN(KITTI)

EfficientDeep(KITTI)

PSM net(KITTI)

HRS net(KITTI)

DeepPruner(KITTI)

GANet(KITTI)

LEAStereo(KITTI)

MC-CNN

EfficientDeep

PSM net

HRS net

DeepPruner

GANet

LEAStereo

Figure 30: Error map and disparity visualization on tree area for DublinCity
dataset.

###  C.1 Discussion of Visual assessment

From the visual assessment, generally, we can make a conclusion:

  * 1.

Using aerial data to do the fine-tuning can improve performance in the
building areas with discontinuity.

  * 2.

Pre-trained methods work better in building areas than in tree areas.

  * 3.

For the tree area, using the pre-trained model on KITTI of DL methods usually
produces a bad result. Using fine-tuning can improve the result, but the
performance of the building area is better than the tree area.

  * 4.

For the building area, the traditional methods give good results, but after
training, DL-based methods also improve the results much.

  * 5.

For the tree area without leaves, because the discontinuity is too large, DL-
based methods do not improve much.

##  Appendix D Reference

## References

  * Adeline et al. (2013) Adeline, K., Le Bris, A., Coubard, F., Briottet, X., Paparoditis, N., Viallefont, F., Riviere, N., Papelard, J.P., David, N., Déliot, P., et al., 2013\.  Description de la campagne aéroportée umbra: étude de l’impact anthropique sur les écosystèmes urbains et naturels avec des images thr multispectrales et hyperspectrales.  Revue française de photogrammétrie et de télédétection 202, 79–92.
  * Alliez et al. (2020) Alliez, P., Tayeb, S., Wormser, C., 2020.  3D fast intersection and distance computation, in: CGAL User and Reference Manual. 5.2 ed.. CGAL Editorial Board.  URL: https://doc.cgal.org/5.2/Manual/packages.html#PkgAABBTree.
  * Bao et al. (2020) Bao, W., Wang, W., Xu, Y., Guo, Y., Hong, S., Zhang, X., 2020.  Instereo2k: a large real dataset for stereo matching in indoor scenes.  Science China Information Sciences 63, 1–11.
  * Batsos (2020) Batsos, K., 2020.  Cbmv.  https://github.com/kbatsos/CBMV.
  * Batsos et al. (2018) Batsos, K., Cai, C., Mordohai, P., 2018.  Cbmv: A coalesced bidirectional matching volume for disparity estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2060–2069.
  * Bethmann and Luhmann (2015) Bethmann, F., Luhmann, T., 2015\.  Semi-global matching in object space.  International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences .
  * Bevilacqua et al. (2017) Bevilacqua, M., Aujol, J.F., Biasutti, P., Brédif, M., Bugeau, A., 2017.  Joint inpainting of depth and reflectance with visibility estimation.  ISPRS Journal of Photogrammetry and Remote Sensing 125, 16–32.
  * Biasutti et al. (2019) Biasutti, P., Bugeau, A., Aujol, J.F., Brédif, M., 2019\.  Visibility estimation in point clouds with variable density, in: International Conference on Computer Vision Theory and Applications (VISAPP).
  * Bosch et al. (2019) Bosch, M., Foster, K., Christie, G., Wang, S., Hager, G.D., Brown, M., 2019\.  Semantic stereo for incidental satellite images, in: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE. pp. 1524–1532.
  * Bosch et al. (2016) Bosch, M., Kurtz, Z., Hagstrom, S., Brown, M., 2016\.  A multiple view stereo benchmark for satellite imagery, in: 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), IEEE. pp. 1–9.
  * Boykov and Kolmogorov (2004) Boykov, Y., Kolmogorov, V., 2004\.  An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision.  IEEE transactions on pattern analysis and machine intelligence 26, 1124–1137.
  * Butler et al. (2012) Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J., 2012\.  A naturalistic open source movie for optical flow evaluation, in: A. Fitzgibbon et al. (Eds.) (Ed.), European Conf. on Computer Vision (ECCV), Springer-Verlag. pp. 611–625.
  * Cabon et al. (2020) Cabon, Y., Murray, N., Humenberger, M., 2020.  Virtual kitti 2.  arXiv preprint arXiv:2001.10773 .
  * Cai et al. (2020) Cai, C., Poggi, M., Mattoccia, S., Mordohai, P., 2020\.  Matching-space stereo networks for cross-domain generalization.  arXiv preprint arXiv:2010.07347 .
  * Cavegn et al. (2014) Cavegn, S., Haala, N., Nebiker, S., Rothermel, M., Tutzauer, P., 2014.  Benchmarking high density image matching for oblique airborne imagery.  The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 40, 45\.
  * Chang (2020) Chang, J.R., 2020.  Psmnet.  https://github.com/JiaRenChang/PSMNet.
  * Chang and Chen (2018) Chang, J.R., Chen, Y.S., 2018\.  Pyramid stereo matching network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410–5418.
  * Cheng (2021) Cheng, X., 2021.  Leastereo.  https://github.com/XuelianCheng/LEAStereo.
  * Cheng et al. (2020) Cheng, X., Zhong, Y., Harandi, M., Dai, Y., Chang, X., Li, H., Drummond, T., Ge, Z., 2020\.  Hierarchical neural architecture search for deep stereo matching.  Advances in Neural Information Processing Systems 33, 22158–22169.
  * Cho et al. (2004) Cho, W., Jwa, Y.S., Chang, H.J., Lee, S.H., 2004.  Pseudo-grid based building extraction using airborne lidar data.  International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences 35, 378–381.
  * Cordts et al. (2016) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B., 2016.  The cityscapes dataset for semantic urban scene understanding, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223.
  * Cournet et al. (2020) Cournet, M., Sarrazin, E., Dumas, L., Michel, J., Guinet, J., Youssefi, D., Defonte, V., Fardet, Q., 2020\.  Ground truth generation and disparity estimation for optical satellite imagery.  The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 43, 127–134.
  * Cramer (2010) Cramer, M., 2010.  The dgpf-test on digital airborne camera evaluation overview and test design.  PFG Photogrammetrie, Fernerkundung, Geoinformation , 73–82.
  * Deseilligny and Rupnik (2020) Deseilligny, M., Rupnik, E., 2020\.  Epipolar rectification of a generic camera .
  * Dosovitskiy et al. (2015) Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T., 2015.  Flownet: Learning optical flow with convolutional networks, in: Proceedings of the IEEE international conference on computer vision, pp. 2758–2766.
  * Dosovitskiy et al. (2017) Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V., 2017.  Carla: An open urban driving simulator, in: Conference on robot learning, PMLR. pp. 1–16.
  * Duggal et al. (2019) Duggal, S., Wang, S., Ma, W.C., Hu, R., Urtasun, R., 2019\.  Deeppruner: Learning efficient stereo matching via differentiable patchmatch, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 4384–4393.
  * Geiger et al. (2012) Geiger, A., Lenz, P., Urtasun, R., 2012.  Are we ready for autonomous driving? the kitti vision benchmark suite, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 3354–3361.
  * Girshick (2015) Girshick, R., 2015.  Fast r-cnn, in: Proceedings of the IEEE international conference on computer vision, pp. 1440–1448.
  * Haala (2013) Haala, N., 2013.  The landscape of dense image matching algorithms, pp. 271–284.
  * Hamid et al. (2020) Hamid, M.S., Abd Manap, N., Hamzah, R.A., Kadmin, A.F., 2020\.  Stereo matching algorithm based on deep learning: A survey.  Journal of King Saud University-Computer and Information Sciences .
  * Hernandez-Juarez (2020) Hernandez-Juarez, D., 2020.  sgm.  https://github.com/dhernandez0/sgm.
  * Hernandez-Juarez et al. (2016) Hernandez-Juarez, D., Chacón, A., Espinosa, A., Vázquez, D., Moure, J.C., López, A.M., 2016\.  Embedded real-time stereo estimation via semi-global matching on the gpu.  Procedia Computer Science 80, 143–153.
  * Hirschmuller (2005) Hirschmuller, H., 2005.  Accurate and efficient stereo processing by semi-global matching and mutual information, in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE. pp. 807–814.
  * Hirschmuller and Scharstein (2007) Hirschmuller, H., Scharstein, D., 2007\.  Evaluation of cost functions for stereo matching, in: 2007 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 1–8.
  * Hosni et al. (2012) Hosni, A., Rhemann, C., Bleyer, M., Rother, C., Gelautz, M., 2012.  Fast cost-volume filtering for visual correspondence and beyond.  IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 504–511.
  * Huang et al. (2018) Huang, R., Zheng, S., Hu, K., 2018.  Registration of aerial optical images with lidar data using the closest point principle and collinearity equations.  Sensors 18, 1770\.
  * Huang et al. (2019) Huang, X., Wang, P., Cheng, X., Zhou, D., Geng, Q., Yang, R., 2019.  The apolloscape open dataset for autonomous driving and its application.  IEEE transactions on pattern analysis and machine intelligence 42, 2702–2719.
  * Jain et al. (1995) Jain, R., Kasturi, R., Schunck, B.G., 1995.  Machine vision. volume 5.  McGraw-hill New York.
  * Jensen et al. (2014) Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H., 2014.  Large scale multi-view stereopsis evaluation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 406–413.
  * Kendall et al. (2017) Kendall, A., Martirosyan, H., Dasgupta, S., Henry, P., Kennedy, R., Bachrach, A., Bry, A., 2017.  End-to-end learning of geometry and context for deep stereo regression, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 66–75.
  * Knöbelreiter and Pock (2021) Knöbelreiter, P., Pock, T., 2021\.  Learned collaborative stereo refinement.  International Journal of Computer Vision , 1–18.
  * Kolmogorov (2005) Kolmogorov, V., 2005.  Convergent tree-reweighted message passing for energy minimization, in: International Workshop on Artificial Intelligence and Statistics, PMLR. pp. 182–189.
  * Kong and Tao (2006) Kong, D., Tao, H., 2006.  Stereo matching via learning multiple experts behaviors., in: BMVC, Citeseer. p. 2.
  * Laefer et al. (2015) Laefer, D.F., Abuwarda, S., Vo, A.V., Truong-Hong, L., Gharibi, H., 2015.  2015 aerial laser and photogrammetry survey of dublin city collection record.  URL: https://geo.nyu.edu/catalog/nyu-2451-38684.
  * Laga et al. (2020) Laga, H., Jospin, L.V., Boussaid, F., Bennamoun, M., 2020\.  A survey on deep learning techniques for stereo-based depth estimation.  arXiv preprint arXiv:2006.02535 .
  * Laina et al. (2016) Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N., 2016.  Deeper depth prediction with fully convolutional residual networks, in: 2016 Fourth international conference on 3D vision (3DV), IEEE. pp. 239–248.
  * Li and Huttenlocher (2008) Li, Y., Huttenlocher, D.P., 2008\.  Learning for stereo vision using the structured support vector machine, in: 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 1–8.
  * Liang et al. (2018) Liang, Z., Feng, Y., Guo, Y., Liu, H., Chen, W., Qiao, L., Zhou, L., Zhang, J., 2018.  Learning for disparity estimation through feature constancy, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2811–2820.
  * Long et al. (2015) Long, J., Shelhamer, E., Darrell, T., 2015.  Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440.
  * Luo (2020) Luo, W., 2020.  cvpr16_stereo_public.  https://bitbucket.org/saakuraa/cvpr16{_}stereo{_}public/src/master/.
  * Luo et al. (2016) Luo, W., Schwing, A.G., Urtasun, R., 2016.  Efficient deep learning for stereo matching, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5695–5703.
  * Mayer et al. (2016) Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., Brox, T., 2016.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4040–4048.
  * Meister et al. (2012) Meister, S., Jähne, B., Kondermann, D., 2012.  Outdoor stereo camera system for the generation of real-world benchmark data sets.  Optical Engineering 51, 021107\.
  * Mensink et al. (2021) Mensink, T., Uijlings, J., Kuznetsova, A., Gygli, M., Ferrari, V., 2021.  Factors of influence for transfer learning across diverse appearance domains and task types.  arXiv preprint arXiv:2103.13318 .
  * micmacIGN (2020) micmacIGN, 2020.  micmac.  https://github.com/micmacIGN/micmac.
  * Nederland (2021) Nederland, A.H., 2021.  Ahn.  URL: https://www.ahn.nl/.
  * Nguyen et al. (2019) Nguyen, D.T., Mummadi, C.K., Ngo, T.P.N., Nguyen, T.H.P., Beggel, L., Brox, T., 2019\.  Self: Learning to filter noisy labels with self-ensembling.  arXiv preprint arXiv:1910.01842 .
  * Pal et al. (2012) Pal, C.J., Weinman, J.J., Tran, L.C., Scharstein, D., 2012\.  On learning conditional random fields for stereo.  International Journal of Computer Vision 99, 319–337.
  * Pang et al. (2017) Pang, J., Sun, W., Ren, J.S., Yang, C., Yan, Q., 2017\.  Cascade residual learning: A two-stage convolutional neural network for stereo matching, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 887–895.
  * Patil et al. (2019) Patil, S., Comandur, B., Prakash, T., Kak, A.C., 2019\.  A new stereo benchmarking dataset for satellite images.  arXiv preprint arXiv:1907.04404 .
  * Pierrot-Deseilligny et al. (2014) Pierrot-Deseilligny, M., Jouin, D., Belvaux, J., Maillet, G., Girod, L., Rupnik, E., Muller, J., Daakir, M., Choqueux, G., Deveau, M., 2014\.  Micmac, apero, pastis and other beverages in a nutshell.  Institut Géographique National .
  * Pierrot-Deseilligny and Paparoditis (2006) Pierrot-Deseilligny, M., Paparoditis, N., 2006\.  A multiresolution and optimization-based image matching approach: An application to surface reconstruction from spot5-hrs stereo imagery., in: ISPRS Workshop On Topographic Mapping From Space, Ankara, Turkey.
  * Poggi et al. (2021) Poggi, M., Tosi, F., Batsos, K., Mordohai, P., Mattoccia, S., 2021.  On the synergies between machine learning and binocular stereo for depth estimation from images: a survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence .
  * Quiñonero-Candela et al. (2009) Quiñonero-Candela, J., Sugiyama, M., Lawrence, N.D., Schwaighofer, A., 2009\.  Dataset shift in machine learning.  Mit Press.
  * Ros et al. (2016) Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M., 2016.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234–3243.
  * Rottensteiner et al. (2012) Rottensteiner, F., Sohn, G., Jung, J., Gerke, M., Baillard, C., Benitez, S., Breitkopf, U., 2012.  The isprs benchmark on urban object classification and 3d building reconstruction.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences I-3 (2012), Nr. 1 1, 293–298.
  * Rupnik et al. (2018) Rupnik, E., Pierrot-Deseilligny, M., Delorme, A., 2018.  3d reconstruction from multi-view vhr-satellite images in micmac.  ISPRS Journal of Photogrammetry and Remote Sensing 139, 201–211.
  * Scharstein et al. (2014) Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., Westling, P., 2014.  High-resolution stereo datasets with subpixel-accurate ground truth, in: German conference on pattern recognition, Springer. pp. 31–42.
  * Scharstein and Szeliski (2002) Scharstein, D., Szeliski, R., 2002\.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  International journal of computer vision 47, 7–42.
  * Scharstein and Szeliski (2003) Scharstein, D., Szeliski, R., 2003\.  High-accuracy stereo depth maps using structured light, in: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., IEEE. pp. I–I.
  * Schops et al. (2017) Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A., 2017.  A multi-view stereo benchmark with high-resolution images and multi-camera videos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3260–3269.
  * Seki and Pollefeys (2017) Seki, A., Pollefeys, M., 2017\.  Sgm-nets: Semi-global matching with neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 231–240.
  * Shewchuk (2020) Shewchuk, J.R., 2020.  A two-dimensional quality mesh generator and delaunay triangulator, Computer Science Division University of California at Berkeley, Berkeley, California.  URL: https://www.cs.cmu.edu/~quake/triangle.html.
  * Song et al. (2021) Song, X., Yang, G., Zhu, X., Zhou, H., Wang, Z., Shi, J., 2021.  Adastereo: a simple and efficient approach for adaptive stereo matching, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10328–10337.
  * Spyropoulos et al. (2014) Spyropoulos, A., Komodakis, N., Mordohai, P., 2014.  Learning to detect ground control points for improving the accuracy of stereo matching, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1621–1628.
  * Sun et al. (2003) Sun, J., Zheng, N.N., Shum, H.Y., 2003.  Stereo matching using belief propagation.  IEEE Transactions on pattern analysis and machine intelligence 25, 787–800.
  * Taniai (2020) Taniai, T., 2020.  Localexpstereo.  https://github.com/t-taniai/LocalExpStereo.
  * Taniai et al. (2017) Taniai, T., Matsushita, Y., Sato, Y., Naemura, T., 2017\.  Continuous 3d label stereo matching using local expansion moves.  IEEE transactions on pattern analysis and machine intelligence 40, 2725–2739.
  * Tola et al. (2008) Tola, E., Lepetit, V., Fua, P., 2008.  A fast local descriptor for dense matching, in: 2008 IEEE conference on computer vision and pattern recognition, IEEE. pp. 1–8.
  * Tombari et al. (2007) Tombari, F., Mattoccia, S., Di Stefano, L., 2007.  Segmentation-based adaptive support for accurate stereo correspondence, in: Pacific-Rim Symposium on Image and Video Technology, Springer. pp. 427–438.
  * Tombari et al. (2008) Tombari, F., Mattoccia, S., Di Stefano, L., Addimanda, E., 2008\.  Classification and evaluation of cost aggregation methods for stereo correspondence, in: 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 1–8.
  * Tonioni et al. (2019a) Tonioni, A., Poggi, M., Mattoccia, S., Di Stefano, L., 2019a.  Unsupervised domain adaptation for depth prediction from images.  IEEE transactions on pattern analysis and machine intelligence 42, 2396–2409.
  * Tonioni et al. (2019b) Tonioni, A., Rahnama, O., Joy, T., Stefano, L.D., Ajanthan, T., Torr, P.H., 2019b.  Learning to adapt for stereo, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9661–9670.
  * Tonioni et al. (2019c) Tonioni, A., Tosi, F., Poggi, M., Mattoccia, S., Stefano, L.D., 2019c.  Real-time self-adaptive deep stereo, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 195–204.
  * UberResearch (2020) UberResearch, 2020.  Deeppruner.  https://github.com/uber-research/DeepPruner.
  * Van Meerbergen et al. (2002) Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L., 2002\.  A hierarchical symmetric stereo algorithm using dynamic programming.  International Journal of Computer Vision 47, 275–285.
  * Wang et al. (2019) Wang, Q., Zheng, S., Yan, Q., Deng, F., Zhao, K., Chu, X., 2019.  Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation.  arXiv preprint arXiv:1912.09678 .
  * Wu et al. (2021) Wu, T., Vallet, B., Pierrot-Deseilligny, M., Rupnik, E., 2021\.  A new stereo dense matching benchmark dataset for deep learning.  ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 43, 405–412.
  * Yang (2020) Yang, G., 2020.  high-res-stereo.  https://github.com/gengshan-y/high-res-stereo.
  * Yang et al. (2019a) Yang, G., Manela, J., Happold, M., Ramanan, D., 2019a.  Hierarchical deep stereo matching on high-resolution images, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5515–5524.
  * Yang et al. (2019b) Yang, G., Song, X., Huang, C., Deng, Z., Shi, J., Zhou, B., 2019b.  Drivingstereo: A large-scale dataset for stereo matching in autonomous driving scenarios, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 899–908.
  * Yang et al. (2018) Yang, G., Zhao, H., Shi, J., Deng, Z., Jia, J., 2018\.  Segstereo: Exploiting semantic information for disparity estimation, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 636–651.
  * Yoon and Kweon (2006) Yoon, K.J., Kweon, I.S., 2006\.  Adaptive support-weight approach for correspondence search.  IEEE transactions on pattern analysis and machine intelligence 28, 650–656.
  * Zbontar (2020) Zbontar, J., 2020.  mc-cnn.  https://github.com/jzbontar/mc-cnn.
  * Žbontar and LeCun (2016) Žbontar, J., LeCun, Y., 2016\.  Stereo matching by training a convolutional neural network to compare image patches.  The journal of machine learning research 17, 2287–2318.
  * Zbontar et al. (2016) Zbontar, J., LeCun, Y., et al., 2016\.  Stereo matching by training a convolutional neural network to compare image patches.  J. Mach. Learn. Res. 17, 2287–2318.
  * Zhang (2020) Zhang, F., 2020.  Ganet.  https://github.com/feihuzhang/GANet.
  * Zhang et al. (2019a) Zhang, F., Prisacariu, V., Yang, R., Torr, P.H., 2019a.  Ga-net: Guided aggregation net for end-to-end stereo matching, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 185–194.
  * Zhang et al. (2020) Zhang, F., Qi, X., Yang, R., Prisacariu, V., Wah, B., Torr, P., 2020.  Domain-invariant stereo matching networks, in: European Conference on Computer Vision, Springer. pp. 420–439.
  * Zhang et al. (2009) Zhang, K., Lu, J., Lafruit, G., 2009.  Cross-based local stereo matching using orthogonal integral images.  IEEE transactions on circuits and systems for video technology 19, 1073–1079.
  * Zhang et al. (2018) Zhang, Z., Gerke, M., Vosselman, G., Yang, M.Y., 2018\.  A patch-based method for the evaluation of dense image matching quality.  International journal of applied earth observation and geoinformation 70, 25–34.
  * Zhang et al. (2019b) Zhang, Z., Lathuilière, S., Pilzer, A., Sebe, N., Ricci, E., Yang, J., 2019b.  Online adaptation through meta-learning for stereo depth estimation.  arXiv preprint arXiv:1904.08462 .
  * Zhou et al. (2016) Zhou, Q.Y., Park, J., Koltun, V., 2016.  Fast global registration, in: European conference on computer vision, Springer. pp. 766–782.
  * Zolanvari et al. (2019) Zolanvari, S., Ruano, S., Rana, A., Cummins, A., da Silva, R.E., Rahbar, M., Smolic, A., 2019.  Dublincity: Annotated lidar point cloud and its applications.  arXiv preprint arXiv:1909.03613 .