of interpreting the key features as $n$ feature vectors in $\mathbb{R}^{d_k}$, the module regards them as $d_k$ single-channel feature maps. Efficient attention uses each of these feature maps as a weighting over all positions and aggregates the value features from all positions through weighted summation to form a global context vector. The name reflects the fact that the vector does not correspond to a specific position, but is a global description of the input features.

The following equation characterizes the efficient attention mechanism: $$\label{eq:ea}
    \bm{E}(\bm{Q}, \bm{K}, \bm{V}) = \rho_q(\bm{Q})\left(\rho_k(\bm{K})^\mathrm{T}\bm{V}\right),$$ where $\rho_q$ and $\rho_k$ are normalization functions for the query and key features, respectively. The implementation of the same two normalization methods as for dot-production attention are $$\begin{aligned}
\begin{split}
\label{eq:ea-norm}
    \text{Scaling: } & \rho_q(\bm{Y}) = \rho_k(\bm{Y}) = \frac{\bm{Y}}{\sqrt{n}}, \\
    \text{Softmax: } & \rho_q(\bm{Y}) = \sigma_\text{row}(\bm{Y}), \\
    & \rho_k(\bm{Y}) = \sigma_\text{col}(\bm{Y}),
\end{split}\end{aligned}$$ where $\sigma_\text{row}, \sigma_\text{col}$ denote applying the softmax function along each row or column of matrix $\bm{Y}$, respectively.

The efficient attention module is a concrete implementation of the mechanism for computer vision data. For an input feature map $x \in \mathbb{R}^{h \times w \times d}$, the module flattens it to a matrix $\bm{X} \in \mathbb{R}^{hw \times d}$, applies the efficient attention 