\begin{table}[t]
    \centering
    \setlength{\tabcolsep}{2pt} 
    \begin{tabular}{llcccccccccccccc}
    \toprule
    
    Training data & Method & EPE & F1-all & $s_{0-10}$ & $s_{10-40}$ & $s_{40+}$ &   \\

    
    \midrule
    
    \multirow{2}{*}[-2pt]{C + T} & RAFT & 5.32 & 17.46 & 0.67 & 1.58 & 13.68 \\
    & GMFlow & 7.77 & 23.40 & 0.74 & 2.19 & 20.34 \\
    & GMFlow+ & 5.74 & 17.63 & 0.64 & 1.69 & 14.86 \\
    
    \midrule
    
    \multirow{2}{*}[-2pt]{C + T + VK} & RAFT & 2.45 & 7.90 & \textbf{0.43} & 1.18 & 5.70 \\
    & GMFlow & 2.85 & 10.77 & 0.49 & 1.16 & 6.87 \\
    & GMFlow+ & \textbf{2.25} & \textbf{7.20} & 0.48 & \textbf{1.10} & \textbf{5.12} \\

    \bottomrule
    \end{tabular}
    \caption{\textbf{Generalization on KITTI 2015 optical flow dataset} after training on synthetic Chairs (C), Things (T) and Virtual KITTI 2 (VK) datasets. 
    }
    \label{tab:flow_gen_kitti}
    
    
\end{table}
{Our unified model with only one task-agnostic hierarchical matching refinement outperforms RAFT~\cite{teed2020raft} with 31 refinement steps on the challenging Sintel~\cite{butler2012naturalistic} dataset while running faster (Fig.~\ref{fig:flow_iter_vs_epe} and Table~\ref{tab:flow_raft_vs_gmflow}), demonstrating the effectiveness and efficiency of our method. Our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow/stereo/depth datasets (KITTI Flow~\cite{menze2015object}, Sintel~\cite{butler2012naturalistic}, Middlebury~\cite{scharstein2014high}, KITTI Stereo~\cite{menze2015object}, ETH3D Stereo~\cite{schops2017multi}, Argoverse Stereo~\cite{chang2019argoverse}, ScanNet~\cite{dai2017scannet}, SUN3D~\cite{xiao2013sun3d}, RGBD-SLAM~\cite{sturm2012benchmark} and Scenes11~\cite{ummenhofer2017demon}), while being simpler and more efficient in terms of model design and inference speed.}

\subsection{Optical Flow}
\noindent \textbf{Datasets and evaluation setup.} Following previous optical flow methods~\cite{ilg2017flownet,sun2018pwc,teed2020raft}, we first train on the FlyingChairs (Chairs)~\cite{dosovitskiy2015flownet} and FlyingThings3D (Things)~\cite{mayer2016large} datasets, and then evaluate on Sintel~\cite{butler2012naturalistic} and KITTI~\cite{menze2015object} training sets for cross-dataset generalization. We also evaluate on the Things validation set to see how the model performs on the same-domain data. Finally, we perform additional fine-tuning on Sintel and KITTI training sets and report the performance on the online benchmarks.

\noindent \textbf{Training schedule.} For methodology comparison (Sec.~\ref{sec:method_comp}) and ablation experiments (Sec.~\ref{sec:ablation}), we first train our default $1/8$ feature resolution model on Chairs dataset for 100K iterations, with a batch size of 16 and a learning rate of 4e-4. We then finetune the model on Things dataset for 200K iterations, with a batch size of 8 and a learning rate of 2e-4. The models thus far all use bilinear upsamling to upsample to the full resolution flow prediction for simplicity. For later experiments, we use RAFT's convex upsampling~\cite{teed2020raft} and our models are trained on Things dataset for 800K iterations, which leads to improved performance~\cite{xu2022gmflow}. For the final fine-tuning process on Sintel and KITTI datasets for benchmark comparisons, we report details in Sec.~\ref{sec:flow_benchmark}.

\noindent {\bf Sintel.} Table~\ref{tab:flow_raft_vs_gmflow} shows the results on Things validation set and Sintel (clean and final) training sets after training on Chairs and Things training sets. Without using any refinement, our method achieves better performance on Things and Sintel (clean) than RAFT with 11 refinements. By using an additional task-agnostic hierarchical matching refinement at $1/4$ feature resolution (Sec.~\ref{sec:hierarchical_match}), our method outperforms RAFT with 31 refinements, especially on large motion ($s_{40+}$). Fig.~\ref{fig:flow_iter_vs_epe} visualizes the results. Furthermore, our model enjoys faster inference speed compared to RAFT and also does not require a large number of sequential processing. On the high-end A100 GPU, our model gains more speedup compared to RAFT's sequential architecture ($2.29\times$ \emph{vs}\onedot. $1.87 \times$, \emph{i.e}\onedot, ours: $151 \to 66$, RAFT: $170 \to 91$), reflecting that our method can benefit more from advanced hardware acceleration and demonstrating its potential for further speed optimization.

\noindent {\bf KITTI.} Table~\ref{tab:flow_gen_kitti} shows the generalization results on KITTI training set after training on Chairs and Things training sets. In this evaluation setting, our method doesn't outperform RAFT, which is mainly caused by the gap between the synthetic training sets and the real-world testing dataset. One key reason behind our inferior performance is that RAFT, relying on fully convolutional neural networks, benefits from the inductive biases in convolution layers, which requires a relatively smaller size training data to generalize to a new dataset in comparison with Transformers~\cite{dosovitskiy2020image, d2021convit,xu2021vitae,zhang2022vitaev2}. To substantiate this claim, we finetune both RAFT and our GMFlow on the additional Virtual KITTI 2~\cite{cabon2020virtual} dataset. The results in Table~\ref{tab:flow_gen_kitti} verify that the performance gap becomes smaller when more data is available. We also train another version GMFlow+ that uses 6 additional local regression refinements (Sec.~\ref{sec:local_reg_refine}), we can observe from Table~\ref{tab:flow_gen_kitti} that GMFlow+ outperforms RAFT on KITTI dataset.

\noindent \textbf{Training schedule.} For ablation experiments, we train our model on the ScanNet for 50K iterations, with a batch size of 80 and a learning rate of 4e-4. The depth range for training and testing is set to $[0.5, 10]$ meters, and the number of depth candidates in the matching layer (Eq.~\eqref{eq:depth_match}) is set to $64$. The training and testing process on SUN3D~\cite{xiao2013sun3d}, RGBD-SLAM~\cite{sturm2012benchmark} and Scenes11~\cite{ummenhofer2017demon} datasets will be elaborated in Sec.~\ref{sec:depth_benchmark}.

\noindent \textbf{Sintel.} Following RAFT~\cite{teed2020raft}, we further finetune our Things trained model on several mixed datasets that consist of KITTI 2015~\cite{menze2015object}, HD1K~\cite{kondermann2016hci}, FlyingThings3D~\cite{mayer2016large} and Sintel~\cite{butler2012naturalistic} training sets. We first finetune on the mixed dataset for 200K iterations with a batch size of 8 and a learning rate of 2e-4. Then we finetune on the Sintel training sets only with a larger crop size $416 \times 1024$ for 5K iterations. The batch size is 8 and the learning rate is 1e-4. To generate the flow prediction results on the Sintel test sets, we first resize the original images to $416 \times 1024$ and then resize the prediction back to the original image resolution for submission. The results on Sintel test set are shown in Table~\ref{tab:flow_sintel_test}. We achieve state-of-the-art results on the highly competitive Sintel (clean) dataset. On Sintel (final) dataset, our performance is only second to the recent FlowFormer~\cite{huang2022flowformer} model, which uses a Transformer model that is pretrained on the large scale ImageNet dataset and is more computationally expensive due to the large number of sequential refinements like RAFT. The visual comparisons with RAFT are shown in Fig.~\ref{fig:vis_sintel_compare}, our method can better capture the motion of fast-moving objects like the moving hand.

\noindent \textbf{KITTI.} We further finetune our VKITTI2 trained GMFlow+ model (Table~\ref{tab:flow_gen_kitti}) on the mixed KITTI 2012 and KITTI 2015 training sets for 30K iterations. The batch size is 8 and the learning rate is 2e-4. The comparison results with previous methods are shown in Table~\ref{tab:flow_kitti_test}. Again, we achieve better performance than RAFT.
{"value": "5.32", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "17.46", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": "F1-all", "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "0.67", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{0-10}", "EPE motion magnitude 0-10"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "1.58", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{10-40}", "EPE motion magnitude 10-40"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "13.68", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{40+}", "EPE motion magnitude 40+"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "7.77", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "23.40", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": "F1-all", "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "0.74", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{0-10}", "EPE motion magnitude 0-10"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "2.19", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{10-40}", "EPE motion magnitude 10-40"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "20.34", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{40+}", "EPE motion magnitude 40+"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "5.74", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "17.63", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": "F1-all", "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "0.64", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{0-10}", "EPE motion magnitude 0-10"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "1.69", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{10-40}", "EPE motion magnitude 10-40"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "14.86", "type": "Result", "training data/set": ["C + T", "Chair Things"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{40+}", "EPE motion magnitude 40+"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "2.45", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "7.90", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": "F1-all", "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "0.43", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{0-10}", "EPE motion magnitude 0-10"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "1.18", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{10-40}", "EPE motion magnitude 10-40"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "5.70", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{40+}", "EPE motion magnitude 40+"], "experimental settings": {"xx": "yy"}, "model": "RAFT", "model settings": {"xx": "yy"}} {"value": "2.85", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "10.77", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": "F1-all", "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "0.49", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{0-10}", "EPE motion magnitude 0-10"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "1.16", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{10-40}", "EPE motion magnitude 10-40"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "6.87", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{40+}", "EPE motion magnitude 40+"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"xx": "yy"}} {"value": "2.25", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "7.20", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": "F1-all", "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "0.48", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{0-10}", "EPE motion magnitude 0-10"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "1.10", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{10-40}", "EPE motion magnitude 10-40"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}} {"value": "5.12", "type": "Result", "training data/set": ["C + T + VK", "Chair Things Virtual KITTI 2"], "test data/set": ["KITTI", "KITTI train"], "task": "Optical Flow", "metric": ["EPE s_{40+}", "EPE motion magnitude 40+"], "experimental settings": {"xx": "yy"}, "model": "GMFlow+", "model settings": {"xx": "yy"}}