\subfloat[
\textbf{Flow to depth transfer: performance comparison}.
\label{tab:transfer_flow_to_depth}
]{
\begin{minipage}{0.45\linewidth}{\begin{center}
\begin{tabular}{lccccccccccccc}
    \toprule
    
    
    Model & Abs Rel & Sq Rel & RMSE & RMSE log \\
    
    \midrule
    
    rand init, w/o ft & 0.536 & 1.309 & 1.300 & 0.584 \\
    flow init, w/o ft & 0.198 & 0.364 & 0.599 & 0.235 \\
    
    \midrule
    
    rand init, ft (50K) & 0.074 & 0.028 & 0.225 & 0.103 \\
    flow init, ft (50K) & 0.066 & 0.023 & 0.203 & 0.092 \\
    
    rand init, ft (100K) & 0.069 & 0.025 & 0.211 & 0.097 \\
    flow init, ft (100K) & \textbf{0.063} & \textbf{0.021} & \textbf{0.193} & \textbf{0.088} \\

    \bottomrule
\end{tabular}
\end{center}}\end{minipage}
}
Our unified model naturally enables cross-task transfer since each task uses exactly the same learnable parameters for feature extraction. For example, without any finetuning, a pretrained optical flow model can be directly used for the task of rectified stereo matching and unrectified stereo depth estimation. Moreover, when finetuning with the pretrained flow model as initialization, we not only enjoy faster training speed for stereo and depth, but also achieve better performance, as evidenced by our experiments (Table~\ref{tab:cross_task_transfer}).

{Our unified model with only one task-agnostic hierarchical matching refinement outperforms RAFT~\cite{teed2020raft} with 31 refinement steps on the challenging Sintel~\cite{butler2012naturalistic} dataset while running faster (Fig.~\ref{fig:flow_iter_vs_epe} and Table~\ref{tab:flow_raft_vs_gmflow}), demonstrating the effectiveness and efficiency of our method. Our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow/stereo/depth datasets (KITTI Flow~\cite{menze2015object}, Sintel~\cite{butler2012naturalistic}, Middlebury~\cite{scharstein2014high}, KITTI Stereo~\cite{menze2015object}, ETH3D Stereo~\cite{schops2017multi}, Argoverse Stereo~\cite{chang2019argoverse}, ScanNet~\cite{dai2017scannet}, SUN3D~\cite{xiao2013sun3d}, RGBD-SLAM~\cite{sturm2012benchmark} and Scenes11~\cite{ummenhofer2017demon}), while being simpler and more efficient in terms of model design and inference speed.}

For unrectified stereo depth estimation, we assume the camera intrinsic and extrinsic parameters (${\bm K}_1, {\bm E}_1, {\bm K}_2, {\bm E}_2$) for image ${\bm I}_1$ and ${\bm I}_2$ are known (\emph{i.e}\onedot, posed images). They can be obtained via additional sensors like IMU and GPS, or reliably estimated using Structure-from-Motion software like COLMAP~\cite{schonberger2016structure}. To estimate depth, we take an approach similar to the classic plane-sweep stereo method~\cite{collins1996space}. More specifically, we first discretize a predefined depth range $[d_{\mathrm{min}}, d_{\mathrm{max}}]$ as $[d_1, d_2, \cdots, d_N]$ (in our implementation, we discretize the inverse depth domain, while we use depth here for ease of notation). Then for each depth candidate $d_i (i=1, 2, \cdots, N)$, we compute the 2D correspondences $\hat{{\bm G}}_{\mathrm{2D}} \in \mathbb{R}^{H \times W \times 2}$ in ${\bm F}_2$ given the current depth value:

Our unified model thus far with only one task-agnostic hierarchical matching refinement is able to outperform RAFT~\cite{teed2020raft} with 31 refinements while running faster (Fig.~\ref{fig:flow_iter_vs_epe} and Table~\ref{tab:flow_raft_vs_gmflow}), which demonstrates the effectiveness and efficiency of our global matching-based formulation. However, our matching-based method is also complementary to previous cost volume and convolution-based local regression approach. We observe the strength of our unified matching method mainly in the presence of large motion (Table~\ref{tab:flow_conv_vs_softmax} and \ref{tab:flow_raft_vs_gmflow}). For small motions, it might not be necessary to perform global matching (Table~\ref{tab:global_local_match}) and in this case, local regression is advantageous. To achieve the best system-level performance, one straightforward way is to combine the strengths of these two kinds of flow estimation approaches. That is, the local regression method is used as a post-processing step to our unified model. This further improves fine-grained details and regions that are hard to match. 

\noindent {\bf Sintel.} Table~\ref{tab:flow_raft_vs_gmflow} shows the results on Things validation set and Sintel (clean and final) training sets after training on Chairs and Things training sets. Without using any refinement, our method achieves better performance on Things and Sintel (clean) than RAFT with 11 refinements. By using an additional task-agnostic hierarchical matching refinement at $1/4$ feature resolution (Sec.~\ref{sec:hierarchical_match}), our method outperforms RAFT with 31 refinements, especially on large motion ($s_{40+}$). Fig.~\ref{fig:flow_iter_vs_epe} visualizes the results. Furthermore, our model enjoys faster inference speed compared to RAFT and also does not require a large number of sequential processing. On the high-end A100 GPU, our model gains more speedup compared to RAFT's sequential architecture ($2.29\times$ \emph{vs}\onedot. $1.87 \times$, \emph{i.e}\onedot, ours: $151 \to 66$, RAFT: $170 \to 91$), reflecting that our method can benefit more from advanced hardware acceleration and demonstrating its potential for further speed optimization.

\noindent {\bf Stereo cross-attention: 1D \emph{vs}\onedot \ 2D.} Unlike 2D optical flow, rectified stereo matching is a 1D correspondence task that corresponding pixels lie on the same horizontal scanline. Thus, it's not necessary to perform 2D cross-attention in the Transformer to model cross-view interactions and 1D horizontal cross-attention is sufficient. As shown in Table~\ref{tab:stereo_1d_2d_attn}, using 1D cross-attention is not only more efficient in terms of inference time (measured for KITTI resolution ($384\times 1248$) on a single V100 GPU), but also leads to better performance since unnecessary matching information is avoided. We note that the parameter-free cross-attention operation (2D, 1D or any other forms) doesn't affect the learnable parameters (\emph{i.e}\onedot, the linear projection layers) of the Transformer, and thus the pretrained model for optical flow and stereo matching tasks can still be shared.

\noindent \textbf{Training schedule.} For ablation experiments, we train our model on the ScanNet for 50K iterations, with a batch size of 80 and a learning rate of 4e-4. The depth range for training and testing is set to $[0.5, 10]$ meters, and the number of depth candidates in the matching layer (Eq.~\eqref{eq:depth_match}) is set to $64$. The training and testing process on SUN3D~\cite{xiao2013sun3d}, RGBD-SLAM~\cite{sturm2012benchmark} and Scenes11~\cite{ummenhofer2017demon} datasets will be elaborated in Sec.~\ref{sec:depth_benchmark}.

\subsection{Cross-Task Transfer}
One unique benefit of our unified model is that it naturally enables cross-task transfer since all the learnable parameters are exactly the same. More specifically, we can directly use a pretrained optical flow model and apply it to both rectified stereo matching and unrectified stereo depth estimation tasks. As shown in Table~\ref{tab:transfer_flow_to_stereo} and Table~\ref{tab:transfer_flow_to_depth}, our pretrained optical flow model performs significantly better than a random initialized model. Notably, no previous optical flow model can be directly used for unrectified stereo depth estimation, while our model achieves promising results as visualized in Fig.~\ref{fig:vis_flow2depth}. The pretrained flow model can be further finetuned for stereo and depth tasks, which not only leads to faster training speed, but also achieves better performance than random initialization (Table~\ref{tab:cross_task_transfer}). 
{"value": "0.536", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, w/o ft"}} {"value": "1.309", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, w/o ft"}} {"value": "1.300", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, w/o ft"}} {"value": "0.584", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, w/o ft"}} {"value": "0.198", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, w/o ft"}} {"value": "0.364", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, w/o ft"}} {"value": "0.599", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, w/o ft"}} {"value": "0.235", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, w/o ft"}} {"value": "0.074", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (50K)"}} {"value": "0.028", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (50K)"}} {"value": "0.225", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (50K)"}} {"value": "0.103", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (50K)"}} {"value": "0.066", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (50K)"}} {"value": "0.023", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (50K)"}} {"value": "0.203", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (50K)"}} {"value": "0.092", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (50K)"}} {"value": "0.069", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (100K)"}} {"value": "0.025", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (100K)"}} {"value": "0.211", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (100K)"}} {"value": "0.097", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "rand init, ft (100K)"}} {"value": "0.063", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (100K)"}} {"value": "0.021", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (100K)"}} {"value": "0.193", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (100K)"}} {"value": "0.088", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"depth range": "[0.5, 10]", "depth candidates": "64"}, "model": "GMFlow", "model settings": {"configuration": "flow init, ft (100K)"}}