vision. [@attention] initially proposed dot-product attention for machine translation. Subsequently, the Transformer [@transformer] adopted the mechanism to model long-range temporal dependencies between words. [@nonlocal] introduce dot-product attention for the modeling of long-range dependencies between pixels in image and video understanding.

For each input feature vector $\bm{x}_i \in \mathbb{R}^d$ that corresponds to the $i$-th position, dot-product attention first uses three linear layers to convert $\bm{x}_i$ into three feature vectors, , the query feature $\bm{q}_i \in \mathbb{R}^{d_k}$, the key feature $\bm{k}_i \in \mathbb{R}^{d_k}$, and the value feature $\bm{v}_i \in \mathbb{R}^{d_v}$. The query and key features must have the same feature dimension $d_k$. One can measure the similarity between the $i$-th query and the $j$-th key as $\rho(\bm{q}_i^\mathrm{T}\bm{k}_j)$, where $\rho$ is a normalization function. In general, the similarities are asymmetric, since the query and key features are the outputs of two separate layers. The dot-product attention module calculates the similarities between all pairs of positions. Using the similarities as weights, position $i$ aggregates the value features from all positions via weighted summation to obtain its output feature.

If one represents all $n$ positions’ query, key, and value features in matrix forms as $\bm{Q} \in \mathbb{R}^{n \times d_k}$, $\bm{K} \in \mathbb{R}^{n \times d_k}, \bm{V} 