across scales and network depths to improve feature expressiveness. It is lightweight but highly effective. Therefore, this work incorporated it into the baseline model.

MegDet [@megdet], the winning entry of the MS-COCO Detection Challenge 2017, reported that large batch sizes and cross-GPU synchronous batch normalization is significantly beneficial to the performance of a deep detector. The baseline incorporated these two techniques.

Method {#sec:method}
======

This section introduces the efficient attention mechanism. It is mathematically equivalent with the widely adopted dot-product attention mechanism in computer vision (, the attention mechanism in the Transformer [@transformer] and the non-local module [@nonlocal]). However, efficient attention has linear memory and computational complexities with respect to the number of pixels or words (hereafter referred to as positions).

Section \[sec:ca\] reviews the dot-product attention mechanism and identifies its critical drawback on large inputs to motivate efficient attention. The introduction of the efficient attention mechanism is in Section \[sec:ea\]. Section \[sec:proof\] shows the equivalence between dot-product and efficient attention. Section \[sec:interpretation\] discusses the interpretation of the mechanism. Section \[sec:efficiency\] analyzes its efficiency advantage over dot-product attention.

![image](figures/ca-ea.png){width="\linewidth"}

A Revisit of Dot-Product Attention {#sec:ca}
----------------------------------

Modeling long-range dependencies has been a central challenge for natural language processing and computer 