is that each row of it sums up to $1$ and represents a normalized attention distribution over all positions. The matrix $\sigma_\text{row}(\bm{Q})\sigma_\text{col}
(\bm{K})^\mathrm{T}$ shares this property. Therefore, the softmax variant of efficient attention is a close approximate of that variant of dot-product attention. Section \[sec:vs\] demonstrates this claim empirically.

Interpretation of Efficient Attention {#sec:interpretation}
-------------------------------------

Efficient attention brings a new interpretation of the attention mechanism. In dot-product attention, selecting position $i$ as the reference position, one can collect the similarities of all positions to position $i$ and form an attention map $\bm{s}_i$ for that position. The attention map $\bm{s}_i$ represents the degree to which position $i$ attends to each position $j$ in the input. A higher value for position $j$ on $\bm{s}_i$ means position $i$ attends more to position $j$. In dot-product attention, every position $i$ has such an attention map $\bm{s}_i$, which the mechanism uses to aggregate the value features $\bm{V}$ to produce the output at position $i$.

In contrast, efficient attention does not generate an attention map for each position. Instead, it interprets the key features $\bm{K} \in \mathbb{R}^{n \times d_k}$ as $d_k$ attention maps $\bm{k}^\mathrm{T}_j$. Each $\bm{k}^\mathrm{T}_j$ is a global attention map that does not correspond to 