\subfloat[
\textbf{Unrectified stereo depth estimation task}.
\label{tab:depth_ablation}
]{
\begin{minipage}{0.5\linewidth}{
\begin{center}
\setlength{\tabcolsep}{2pt} 
\begin{tabular}{lccccccccccccc}
    \toprule
    
    
    setup & Abs Rel & Sq Rel & RMSE & RMSE log & Param (M) \\
    
    \midrule
    
    full & \textbf{0.074} & \textbf{0.028} & \textbf{0.225} & \textbf{0.103} & 4.7 \\
    w/o cross attn. & 0.095 & 0.043 & 0.284 & 0.132 & 4.3 \\
    w/o position & 0.078 & 0.031 & 0.237 & 0.109 & 4.7 \\
    w/o FFN & 0.089 & 0.041 & 0.276 & 0.127 & 2.3 \\
    w/o self attn. & 0.081 & 0.034 & 0.248 & 0.114 & 4.3 \\
    w/o propagation & 0.091 & 0.045 & 0.293 & 0.144 & 4.6 \\

    \bottomrule
    \end{tabular}
\end{center}}\end{minipage}
}
{Our unified model with only one task-agnostic hierarchical matching refinement outperforms RAFT~\cite{teed2020raft} with 31 refinement steps on the challenging Sintel~\cite{butler2012naturalistic} dataset while running faster (Fig.~\ref{fig:flow_iter_vs_epe} and Table~\ref{tab:flow_raft_vs_gmflow}), demonstrating the effectiveness and efficiency of our method. Our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow/stereo/depth datasets (KITTI Flow~\cite{menze2015object}, Sintel~\cite{butler2012naturalistic}, Middlebury~\cite{scharstein2014high}, KITTI Stereo~\cite{menze2015object}, ETH3D Stereo~\cite{schops2017multi}, Argoverse Stereo~\cite{chang2019argoverse}, ScanNet~\cite{dai2017scannet}, SUN3D~\cite{xiao2013sun3d}, RGBD-SLAM~\cite{sturm2012benchmark} and Scenes11~\cite{ummenhofer2017demon}), while being simpler and more efficient in terms of model design and inference speed.}

For unrectified stereo depth estimation, we assume the camera intrinsic and extrinsic parameters (${\bm K}_1, {\bm E}_1, {\bm K}_2, {\bm E}_2$) for image ${\bm I}_1$ and ${\bm I}_2$ are known (\emph{i.e}\onedot, posed images). They can be obtained via additional sensors like IMU and GPS, or reliably estimated using Structure-from-Motion software like COLMAP~\cite{schonberger2016structure}. To estimate depth, we take an approach similar to the classic plane-sweep stereo method~\cite{collins1996space}. More specifically, we first discretize a predefined depth range $[d_{\mathrm{min}}, d_{\mathrm{max}}]$ as $[d_1, d_2, \cdots, d_N]$ (in our implementation, we discretize the inverse depth domain, while we use depth here for ease of notation). Then for each depth candidate $d_i (i=1, 2, \cdots, N)$, we compute the 2D correspondences $\hat{{\bm G}}_{\mathrm{2D}} \in \mathbb{R}^{H \times W \times 2}$ in ${\bm F}_2$ given the current depth value:

\begin{equation}
\label{eq:depth_match}
    {\bm M}_{\mathrm{depth}} = \mathrm{softmax} ({\bm C}_{\mathrm{depth}}) \in \mathbb{R}^{H \times W \times N}.
\end{equation}

\noindent \textbf{Training schedule.} For methodology comparison (Sec.~\ref{sec:method_comp}) and ablation experiments (Sec.~\ref{sec:ablation}), we first train our default $1/8$ feature resolution model on Chairs dataset for 100K iterations, with a batch size of 16 and a learning rate of 4e-4. We then finetune the model on Things dataset for 200K iterations, with a batch size of 8 and a learning rate of 2e-4. The models thus far all use bilinear upsamling to upsample to the full resolution flow prediction for simplicity. For later experiments, we use RAFT's convex upsampling~\cite{teed2020raft} and our models are trained on Things dataset for 800K iterations, which leads to improved performance~\cite{xu2022gmflow}. For the final fine-tuning process on Sintel and KITTI datasets for benchmark comparisons, we report details in Sec.~\ref{sec:flow_benchmark}.

\noindent {\bf Model components.} We ablate different components of our full model in Table~\ref{tab:stereo_ablation}. The results are consistent with those for the optical flow task in Table~\ref{tab:transformer} and Table~\ref{tab:prop}. That is, the cross-attention contributes most, but the other components also contribute to the performance gains.

\noindent \textbf{Metrics.} Following previous methods~\cite{Tang2019BANetDB, im2019dpsnet}, we use 4 error metrics for evaluation of the depth quality, including Absolute Relative difference (Abs Rel), Squared Relative difference (Sq Rel), Root Mean Squared Error (RMSE) and RMSE in log scale (RMSE log).

\noindent \textbf{Training schedule.} For ablation experiments, we train our model on the ScanNet for 50K iterations, with a batch size of 80 and a learning rate of 4e-4. The depth range for training and testing is set to $[0.5, 10]$ meters, and the number of depth candidates in the matching layer (Eq.~\eqref{eq:depth_match}) is set to $64$. The training and testing process on SUN3D~\cite{xiao2013sun3d}, RGBD-SLAM~\cite{sturm2012benchmark} and Scenes11~\cite{ummenhofer2017demon} datasets will be elaborated in Sec.~\ref{sec:depth_benchmark}.

\noindent {\bf Model components.} We ablate different components of our full model in Table~\ref{tab:depth_ablation}. The results are consistent with those for optical flow and stereo matching tasks in Table~\ref{tab:transformer}, Table~\ref{tab:prop} and Table~\ref{tab:stereo_ablation}. That is, the cross-attention contributes most, but the other components also contribute to the performance gains.
{"value": "0.074", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.7"}} {"value": "0.028", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.7"}} {"value": "0.225", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.7"}} {"value": "0.103", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.7"}} {"value": "4.7", "type": "Hyper-parameter/Architecture", "model": "GMFlow", "parameter/architecture name": ["number of parameters", "Param (M)"], "dataset": "ScanNet"} {"value": "0.095", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "4.3"}} {"value": "0.043", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "4.3"}} {"value": "0.284", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "4.3"}} {"value": "0.132", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "4.3"}} {"value": "4.3", "type": "Hyper-parameter/Architecture", "model": "GMFlow", "parameter/architecture name": ["number of parameters", "Param (M)"], "dataset": "ScanNet"} {"value": "0.078", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.7"}} {"value": "0.031", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.7"}} {"value": "0.237", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.7"}} {"value": "0.109", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.7"}} {"value": "4.7", "type": "Hyper-parameter/Architecture", "model": "GMFlow", "parameter/architecture name": ["number of parameters", "Param (M)"], "dataset": "ScanNet"} {"value": "0.089", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "2.3"}} {"value": "0.041", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "2.3"}} {"value": "0.276", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "2.3"}} {"value": "0.127", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "2.3"}} {"value": "2.3", "type": "Hyper-parameter/Architecture", "model": "GMFlow", "parameter/architecture name": ["number of parameters", "Param (M)"], "dataset": "ScanNet"} {"value": "0.081", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "4.3"}} {"value": "0.034", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "4.3"}} {"value": "0.248", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "4.3"}} {"value": "0.114", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "4.3"}} {"value": "4.3", "type": "Hyper-parameter/Architecture", "model": "GMFlow", "parameter/architecture name": ["number of parameters", "Param (M)"], "dataset": "ScanNet"} {"value": "0.091", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Abs Rel", "Absolute Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o propagation", "Param (M)": "4.6"}} {"value": "0.045", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["Sq Rel", "Squared Relative difference"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o propagation", "Param (M)": "4.6"}} {"value": "0.293", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE", "Root Mean Squared Error"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o propagation", "Param (M)": "4.6"}} {"value": "0.144", "type": "Result", "training data/set": "ScanNet", "test data/set": "ScanNet", "task": ["Depth Prediction", "Depth Estimation"], "metric": ["RMSE log", "RMSE in log scale"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o propagation", "Param (M)": "4.6"}} {"value": "4.6", "type": "Hyper-parameter/Architecture", "model": "GMFlow", "parameter/architecture name": ["number of parameters", "Param (M)"], "dataset": "ScanNet"}