\subfloat[
\textbf{Transformer components}. Cross-attention contributes most.
\label{tab:transformer}
]{
\centering
\begin{minipage}{0.45\linewidth}
{\begin{center}
\begin{tabular}{lccccccccccccccc}
    \toprule
    
    \multirow{2}{*}[-2pt]{setup} & \multicolumn{1}{c}{Things (val)} & \multicolumn{2}{c}{Sintel (train)} &  \multirow{2}{*}[-2pt]{\begin{tabular}[x]{@{}c@{}}Param\\(M) \end{tabular}} \\
    \addlinespace[-10pt] \\
    \cmidrule(lr){2-2} \cmidrule(lr){3-4}
    \addlinespace[-10pt] \\
    & clean & clean & final & \\
    \midrule

    full & \textbf{6.67} & \textbf{2.28} & \textbf{3.44} & 4.2 \\
    
    w/o cross attn. & 10.84 & 4.48 & 6.32 & 3.8  \\
    w/o position & 8.38 & 2.85 & 4.28 & 4.2 \\
    w/o FFN & 8.71 & 3.10 & 4.43 & 1.8 & \\
    w/o self attn. & 7.04 & 2.49 & 3.69 & 3.8 \\
    
    \bottomrule
    \end{tabular}
\end{center}}
\end{minipage}
}
Recent iterative 2D methods like RAFT-Stereo~\cite{lipson2021raft} and CREStereo~\cite{li2022practical} mostly follow the high-level design of the RAFT~\cite{teed2020raft} architecture for optical flow, while introducing several task-specific components (\emph{e.g}\onedot, 1D correlation) to make such a method suitable for the stereo matching task. In contrast, we show that our matching-based perspective enables to use the \textit{same} model for both optical flow and stereo matching, with exactly the same learnable parameters. Besides, our model is also more efficient since we don't rely on any 3D convolutions or a large number of sequential refinements. On the other hand, although MC-CNN~\cite{Zbontar2016Stereo} also tries to learn strong features for matching, the features in MC-CNN are extracted \textit{independently} with a convolutional network, without considering cross-view interactions. However, as evidenced by our results, cross-view interactions are crucial for strong and discriminative features (see Table~\ref{tab:flow_conv_vs_softmax} and Table~\ref{tab:transformer}). Perhaps the most related stereo work to ours is STTR~\cite{li2021revisiting}, which also uses a Transformer and matching-based disparity computation. However, STTR relies on a complex optimal transport matching layer and doesn’t produce predictions for occluded pixels, while we use a much simpler softmax operation and a simple flow propagation layer to handle occlusions. Moreover, STTR is designed to solve the stereo matching task, while we are seeking a unified model applicable to three different dense correspondence estimation tasks.

\subsection{Depth Estimation}
Learning-based depth estimation methods can be broadly categorized into monocular and multi-view approaches. Monocular methods~\cite{garg2016unsupervised,godard2017unsupervised,zhou2017unsupervised,xu2019rdn4depth,godard2019digging,ranftl2020towards} take a single image as input and use generic network architectures like ResNet~\cite{he2016deep} to predict the dense depth map, while multi-view methods~\cite{ummenhofer2017demon,im2019dpsnet,Tang2019BANetDB,Teed2020DeepV2D,yao2018mvsnet} usually focus on how to encode the geometric inductive bias (cost volume, warping, \emph{etc}\onedot) into the network architecture. Compared with monocular methods, multi-view depth estimation can better leverage the information from additional viewpoints and usually lead to improved performance~\cite{watson2021temporal}. Since multi-view information (\emph{e.g}\onedot, video sequences) are usually readily available for many applications, we consider multi-view depth estimation in this paper. A popular multi-view depth pipeline is using the plane-sweep stereo~\cite{im2019dpsnet,gu2020cascade} approach, where different depth planes are tested for correctness. However, like rectified stereo matching, the state-of-the-art methods are usually dominated by 3D convolution-based approaches~\cite{im2019dpsnet,Teed2020DeepV2D}, which accordingly introduces cubic computational complexity.  In this paper, we approach this task from an explicit matching-based perspective and use a Transformer to obtain strong features for matching, achieving highly competitive performance without relying on any 3D convolutions. This is different from the recent work TransMVSNet~\cite{ding2022transmvsnet}, which still relies on 3D convolutions for cost volume post-processing and where the Transformer is used before the cost volume construction stage. Thus, our method is simpler and more lightweight. We also note that we only consider the two-view depth estimation setting in this paper, but our method can be easily extended to the multi-view case. 

Key to our formulation lies in obtaining high-quality discriminative features for matching. To achieve this, we combine a common convolutional network (CNN) with a Transformer~\cite{vaswani2017attention} as the feature extractor. More specifically, we first use a weight-sharing ResNet~\cite{he2016deep} to extract $8\times$ downsampled features to keep computation tractable, similar to previous flow methods~\cite{sun2018pwc,teed2020raft}. However, the two features from the CNN are extracted independently, without considering their mutual relations yet. Integrating knowledge from the potential matching candidates in another image can intuitively enhance the feature's distinctiveness and surpass ambiguities, as demonstrated by sparse matching methods~\cite{sarlin2020superglue}. This can be naturally implemented with the cross-attention mechanism, which is able to selectively aggregate information from another image by measuring cross-view feature similarities. We also use a self-attention layer to further improve the feature's quality by considering larger context than the convolutional layer, and a two-layer feed-forward network (FFN, \emph{i.e}\onedot, MLP) to further increase the capacity of the network following the original Transformer~\cite{vaswani2017attention}'s design. The self-attention, cross-attention and FFN constitute a Transformer block, and our final Transformer architecture is a stack of six Transformer blocks which gradually improve the performance (Table.~\ref{tab:flow_conv_vs_softmax}).

Specifically, for the extracted convolutional features $\Tilde{\bm F}_1$ and $\Tilde{\bm F}_2$, we first add fixed 2D sine and cosine positional encodings (following DETR~\cite{carion2020end}) to the features since they lack spatial information. Adding the position information also makes the matching process consider not only the feature similarity but also their spatial distance, which can help resolve ambiguities and improve performance (Table~\ref{tab:transformer}). Then the features are fed into the Transformer for feature enhancement. More specifically, for self-attention, the query, key and value in the attention mechanism~\cite{vaswani2017attention} are the same feature. For cross-attention, the key and value are same but different from the query to model cross-view interactions. This process is performed for both $\Tilde{\bm F}_1$ and $\Tilde{\bm F}_2$ symmetrically:

One issue with the standard Transformer architecture~\cite{vaswani2017attention} is the quadratic computational complexity due to the pair-wise attention operation. To improve efficiency, we adopt the shifted local window attention strategy from Swin Transformer~\cite{liu2021Swin}. However, unlike Swin that uses a \emph{fixed window size}, we split the feature to \emph{fixed number of local windows} to make the window size adaptive with the feature's spatial size. In this way, the attention mechanism can model long-range dependencies on high-resolution feature maps and accordingly better performance for large displacements can be achieved.  Specifically, we split the input feature of size $H \times W$ to $K \times K$ windows (each with size $\frac{H}{K} \times \frac{W}{K}$, better for large displacements flow if $K$ is smaller, see Table~\ref{tab:split_attn}), and perform self- and cross-attentions within each local window independently. For every two consecutive local windows, we shift the window partition by $(\frac{H}{2K}, \frac{W}{2K})$ to introduce cross-window connections. In our method, we split into $2 \times 2$ windows (each with size $\frac{H}{2} \times \frac{W}{2}$), which leads to a good speed-accuracy trade-off (Table~\ref{tab:split_attn}).

\noindent {\bf Flow estimation approach.} We compare our Transformer and softmax-based flow estimation method with cost volume and convolution-based approaches. Specifically, we adopt the state-of-the-art cost volume construction method in RAFT~\cite{teed2020raft} that concatenates 4 local cost volumes at 4 scales, where each cost volume has a dimension of $H \times W \times (2R+1)^2$. Here $H$ and $W$ denote the feature's spatial size, and the search range $R$ is set to 4 following RAFT. To regress flow, we stack different numbers of convolutional residual blocks~\cite{he2016deep} to see how the performance varies. The final optical flow is obtained with a $3 \times 3$ convolution with 2 output channels. For our method, we stack different numbers of Transformer blocks for feature enhancement and the final optical flow is obtained with a global correlation and softmax layer. Table~\ref{tab:flow_conv_vs_softmax} shows that the performance improvement of our method is more significant compared to the cost volume and convolution-based approach. For instance, our method with 2 Transformer blocks is already able to outperform 8 convolution blocks, especially in the presence of large motions ($s_{40+}$). The performance can be further improved by stacking more layers, surpassing the cost volume and convolution-based approach by a large margin. We also replace the Transformer in our model with a convolutional network for feature enhancement, which leads to a large drop in performance. This is largely due to the unique advantage of the cross-attention mechanism for modeling cross-view interactions (see Table~\ref{tab:transformer} for detailed evaluations of the Transformer components), which enables aggregation the information from the other frame by considering cross-view similarities and thus greatly improves the quality of the extracted features. This is not achievable with convolutions~\cite{Zbontar2016Stereo}.

\noindent {\bf Transformer components.} We ablate different Transformer components in Table~\ref{tab:transformer}. The cross-attention contributes most, since it models the cross-view interactions between two features, which integrates the knowledge from another image and greatly improves the quality of the extracted features. Also, the position information makes the matching process position-dependent, which can help alleviate the ambiguities in pure feature similarity-based matching. Removing the feed-forward network (FFN) reduces a large number of parameters, while also leading to a moderate performance drop. The self-attention aggregates contextual cues within the same feature, leading to additional performance gains.

\noindent {\bf Model components.} We ablate different components of our full model in Table~\ref{tab:stereo_ablation}. The results are consistent with those for the optical flow task in Table~\ref{tab:transformer} and Table~\ref{tab:prop}. That is, the cross-attention contributes most, but the other components also contribute to the performance gains.
{"value": "6.67", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Things (val) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.2"}} {"value": "2.28", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.2"}} {"value": "3.44", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) final", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "full", "Param (M)": "4.2"}} {"value": "4.2", "type": "Hyper-parameter/Architecture", "model": "GMFlow full", "parameter/architecture name": ["Param (M)", "number of parameters (M)"], "dataset": ["FlyingChairs FlyingThings3D", "Chairs Things"]} {"value": "10.84", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Things (val) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "3.8"}} {"value": "4.48", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "3.8"}} {"value": "6.32", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) final", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o cross attn.", "Param (M)": "3.8"}} {"value": "3.8", "type": "Hyper-parameter/Architecture", "model": "GMFlow w/o cross attn.", "parameter/architecture name": ["Param (M)", "number of parameters (M)"], "dataset": ["FlyingChairs FlyingThings3D", "Chairs Things"]} {"value": "8.38", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Things (val) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.2"}} {"value": "2.85", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.2"}} {"value": "4.28", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) final", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o position", "Param (M)": "4.2"}} {"value": "4.2", "type": "Hyper-parameter/Architecture", "model": "GMFlow w/o position", "parameter/architecture name": ["Param (M)", "number of parameters (M)"], "dataset": ["FlyingChairs FlyingThings3D", "Chairs Things"]} {"value": "8.71", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Things (val) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "1.8"}} {"value": "3.10", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "1.8"}} {"value": "4.43", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) final", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o FFN", "Param (M)": "1.8"}} {"value": "1.8", "type": "Hyper-parameter/Architecture", "model": "GMFlow w/o FFN", "parameter/architecture name": ["Param (M)", "number of parameters (M)"], "dataset": ["FlyingChairs FlyingThings3D", "Chairs Things"]} {"value": "7.04", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Things (val) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "3.8"}} {"value": "2.49", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) clean", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "3.8"}} {"value": "3.69", "type": "Result", "training data/set": ["FlyingChairs FlyingThings3D", "Chairs Things"], "test data/set": "Sintel (train) final", "task": "Optical Flow", "metric": ["end-point-error", "EPE"], "experimental settings": {"xx": "yy"}, "model": "GMFlow", "model settings": {"configuration": "w/o self attn.", "Param (M)": "3.8"}} {"value": "3.8", "type": "Hyper-parameter/Architecture", "model": "GMFlow w/o self attn.", "parameter/architecture name": ["Param (M)", "number of parameters (M)"], "dataset": ["FlyingChairs FlyingThings3D", "Chairs Things"]}