computes a sum of the global context vectors with the query feature as the weights to produce the output feature at the position. This algorithm avoids the generation of the pairwise attention matrix, whose size is quadratic in the spatiotemporal size of the input. Therefore, it achieves linear complexities with respect to input size and obtains significant efficiency improvements.

The principal contribution of this paper is the efficient attention mechanism, which:

1.  has linear memory and computational complexities with respect to the spatiotemporal size of the input;

2.  possesses the same representational power as the widely adopted dot-product attention mechanism;

3.  allows the incorporation of significantly more attention modules into a neural network, which brings substantial performance boosts to tasks such as object detection and instance segmentation (on MS-COCO 2017) and image classification (on ImageNet); and

4.  facilitates the application of attention on resource-hungry tasks, such as stereo depth estimation (on the Scene Flow dataset).

Related Works
=============

Dot-Product Attention
---------------------

[@attention] proposed the initial formulation of the dot-product attention mechanism to improve word alignment in machine translation. Successively, [@transformer] proposed to completely replace recurrence with attention and named the resultant architecture the Transformer. The Transformer architecture is highly successful 