mechanism on it, and reshapes the result to $h \times w \times d_v$. If $d_v \ne d$, it further applies a 1x1 convolution to restore the dimensionality to $d$. Finally, it adds the resultant features to the input features to form a residual structure.

Equivalence between Dot-Product and Efficient Attention {#sec:proof}
-------------------------------------------------------

Following is a formal proof of the equivalence between dot-product and efficient attention when using scaling normalization. Substituting the scaling normalization formula in Equation into Equation gives $$\label{eq:ca-scaling}
\bm{D}(\bm{Q}, \bm{K}, \bm{V}) = \frac{\bm{Q}\bm{K}^\mathrm{T}}{n}\bm{V}.$$ Similarly, plugging the scaling normalization formulae in Equation into Equation results in $$\label{eq:ea-scaling}
\bm{E}(\bm{Q}, \bm{K}, \bm{V}) = \frac{\bm{Q}}{\sqrt{n}}\left(\frac{\bm{K}^\mathrm{T}}{\sqrt{n}}\bm{V}\right).$$

Since scalar multiplication is commutative with matrix multiplication and matrix multiplication is associative, we have $$\begin{aligned}
\begin{split}
\label{eq:proof}
    \bm{E}(\bm{Q}, \bm{K}, \bm{V}) &= \frac{\bm{Q}}{\sqrt{n}}\left(\frac{\bm{K}^\mathrm{T}}{\sqrt{n}}\bm{V}\right) \\
    &= \frac{1}{n}\bm{Q}\left(\bm{K}^\mathrm{T}\bm{V}\right) \\
    &= \frac{1}{n}\left(\bm{Q}\bm{K}^\mathrm{T}\right)\bm{V} \\
    &= \frac{\bm{Q}\bm{K}^\mathrm{T}}{n}\bm{V}.
\end{split}\end{aligned}$$ Comparing Equations and , we get $$\label{eq:qed}
    \bm{E}(\bm{Q}, \bm{K}, \bm{V}) = \bm{D}(\bm{Q}, \bm{K}, \bm{V}).$$ Thus, the proof is complete.

The above proof works for the softmax normalization variant with one caveat. The two softmax operations on $\bm{Q}, \bm{K}$ are not exactly equivalent to the single softmax on $\bm{Q}\bm{K}^\mathrm{T}$. However, they closely approximate the effect of the original softmax function. The critical property of $\sigma_\text{row}\left(\bm{Q}\bm{K}^\mathrm{T}\right)$ 