St, Seattle, WA 98103, United States\
    [zhuorans@google.com]{}
- |
    Zhang Mingyuan[^2], Zhao Haiyu, Yi Shuai\
    SenseTime International\
    71 Robinson Road \#14-128, Singapore\
    [zhangmingyuan,zhaohaiyu,yishuai@sensetime.com]{}
- |
    Li Hongsheng\
    The Chinese University of Hong Kong\
    Sha Tin, Hong Kong\
    [hsli@ee.cuhk.edu.hk]{}
bibliography:
- 'egbib.bib'
title: 'Efficient Attention: Attention with Linear Complexities'
---

Introduction {#sec:intro}
============

Attention is a mechanism in neural networks that focuses on long-range dependency modeling, a key challenge to deep learning that convolution and recurrence struggle to solve. A recent series of works developed the highly successful dot-product attention mechanism, which facilitates easy integration into a deep neural network. The mechanism computes the response at every position as a weighted sum of features at all positions in the previous layer. In contrast to the limited spatial and temporal receptive fields of convolution and recurrence, dot-product attention expands the receptive field to the entire input in one pass. Using dot-product attention to efficiently model long-range dependencies allows convolution and recurrence to focus on local dependency modeling, in which they specialize. Dot-product attention-based models now hold state-of-the-art records on nearly all tasks in natural language processing [@transformer; @gpt; @bert; @gpt2]. The non-local module [@nonlocal], an adaptation of dot-product attention for computer vision, 