on sequence tasks. They hold the state-of-the-art records on virtually all tasks in natural language processing [@bert; @gpt2; @xlnet] and is highly competitive on end-to-end speech recognition [@speech-tfm; @deep-san]. [@nonlocal] first adapted dot-product attention for computer vision and proposed the non-local module. They achieved state-of-the-art performance on video classification and demonstrated significant improvements on object detection, instance segmentation, and pose estimation. Subsequent works applied it to various fields in computer vision, including image restoration [@nlrn], video person re-identification [@video-reid], and notably generative adversarial image modeling, where SAGAN [@sagan] and BigGAN [@biggan] substantially advanced the state-of-the-art using the non-local module.

Efficient attention mainly builds upon the version of dot-product attention in the non-local module. Following [@nonlocal], the team conducted most experiments on object detection and instance segmentation. The paper compares the resource efficiency of the efficient attention module against the non-local module under the same performance and their performance under the same resource constraints.

Scaling Attention
-----------------

Besides dot-product attention, there are a separate set of techniques the literature refers to as attention. This section refers to them as scaling attention. While dot-product attention is effective for global dependency modeling, scaling attention focuses on emphasizing important features and suppressing uninformative ones. 