Transformers are awesome, but they have this quadratic complexity problem in the attention matrix. The time and memory scale with the square of the input size. This is a problem for really big inputs. Lots of people have tried to get around this problem. Here's one example paper:
https://arxiv.org/abs/2203.08913
Memorizing Transformers
2022-03-16
Wu; Yuhuai; Rabe; Markus N; Hutchins; DeLesley; Szegedy; Christian
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2203.08913">Download PDF</a></p><blockquote>
 Abstract: Language models typically need to be trained or finetuned in order to acquire
new knowledge, which involves updating their weights. We instead envision
language models that can simply read and memorize new data at inference time,
thus acquiring new knowledge immediately. In this work, we extend language
models with the ability to memorize the internal representations of past
inputs. We demonstrate that an approximate kNN lookup into a non-differentiable
memory of recent (key, value) pairs improves language modeling across various
benchmarks and tasks, including generic webtext (C4), math papers (arXiv),
books (PG-19), code (Github), as well as formal theorems (Isabelle). We show
that the performance steadily improves when we increase the size of memory up
to 262K tokens. On benchmarks including code and mathematics, we find that the
model is capable of making use of newly defined functions and theorems during
test time.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Yuhuai(Tony) Wu [<a href="/show-email/6bcd4bbb/2203.08913">view email</a>]
 <br /><strong>[v1]</strong>
Wed, 16 Mar 2022 19:54:35 UTC (1,168 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2203.08913

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:2203.08913 (cs)
 [Submitted on 16 Mar 2022]

 Title: Memorizing Transformers

 Authors: Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy
 Download PDF
 Abstract: Language models typically need to be trained or finetuned in order to acquire new knowledge, which involves updating their weights. We instead envision language models that can simply read and memorize new data at inference time, thus acquiring new knowledge immediately. In this work, we extend language models with the ability to memorize the internal representations of past inputs. We demonstrate that an approximate kNN lookup into a non-differentiable memory of recent (key, value) pairs improves language modeling across various benchmarks and tasks, including generic webtext (C4), math papers (arXiv), books (PG-19), code (Github), as well as formal theorems (Isabelle). We show that the performance steadily improves when we increase the size of memory up to 262K tokens. On benchmarks including code and mathematics, we find that the model is capable of making use of newly defined functions and theorems during test time.
 Comments: Published as a conference paper at ICLR 2022 (spotlight) 
 Subjects: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
 Cite as: arXiv:2203.08913 [cs.LG]  
 (or arXiv:2203.08913v1 [cs.LG] for this version) 
 https://doi.org/10.48550/arXiv.2203.08913 
 Focus to learn more  
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Yuhuai(Tony) Wu [view email]
 [v1] Wed, 16 Mar 2022 19:54:35 UTC (1,168 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 2203
 Change to browse by:
 cs
 cs.AI
 cs.CL

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Date Published: 2022-03-16
Authors:
Yuhuai  Wu, yuhuai@google.com
Markus N Rabe, mrabe@google.com
Delesley  Hutchins, delesley@google.com
Christian  Szegedy, szegedy@google.com
Abstract
Language models typically need to be trained or finetuned in order to acquire new knowledge, which involves updating their weights. We instead envision language models that can simply read and memorize new data at inference time, thus acquiring new knowledge immediately. In this work, we extend language models with the ability to memorize the internal representations of past inputs. We demonstrate that an approximate kNN lookup into a non-differentiable memory of recent (key, value) pairs improves language modeling across various benchmarks and tasks, including generic webtext (C4), math papers (arXiv), books (PG-19), code (Github), as well as formal theorems (Isabelle). We show that the performance steadily improves when we increase the size of memory up to 262K tokens. On benchmarks including code and mathematics, we find that the model is capable of making use of newly defined functions and theorems during test time.
1 INTRODUCTION
Transformers (Vaswani et al., 2017) have led to remarkable progress in natural language processing (Devlin et al., 2019; Brown et al., 2020) , mathematical reasoning (Polu & Sutskever, 2020; Wang et al., 2020a; Rabe et al., 2021; Li et al., 2021; Hahn et al., 2021; Cobbe et al., 2021) , and program synthesis (Austin et al., 2021; Chen et al., 2021; Li et al., 2022) . However, transformer performance on many of these tasks is limited by the context length of attention, which is typically short. The ability to attend to far-away tokens is important in many situations. In novels, characters and events are referenced across multiple chapters. In source code, references to classes and functions may occur quite far from the places in which they are defined. In theorem proving, proofs make use of previously defined lemmas.
Attention over long sequences is also useful as a form of rapid learning. Facts and information which are stored in the form of weight matrices must be slowly trained over hundreds of thousands of training steps. By using attention, however, a model can simply memorize facts (e.g. function definitions) by storing them as (key, value) pairs in long-term memory, and then retrieve those facts later by creating a query that attends to them. In this case, attention acts as a form of information retrieval, allowing the model to look up facts that it has seen previously.
We demonstrate that a simple and effective way to increase the size of the attention context is to use approximate k-nearest-neighbor (kNN) lookup, which is widely used in information retrieval. A number of extremely scalable implementations of kNN lookup are available, such as ScaNN (Guo et al., 2020) and Faiss (Johnson et al., 2021) .
There are two things which distinguish our approach from previous work on long-range attention (c.f. Section 2). First, unlike some other approaches, kNN lookup does not do averaging or summarization of tokens at long distances, but retrieves exact values even from the distant context. Second, gradients are not backpropagated into the external memory, which is critical to the scalability of our technique. The keys and values are a function of model parameters, so attempting to backpropagate gradients into external memory would necessarily involve computing all of the keys and values with the current model parameters on every training step. However, if the external memory is not differentiable, then we can instead instead reuse keys and values that were previously computed on prior training steps, which drastically reduces the amount of computation for large memories. With our technique, we are easily able to scale external memory up to sequence lengths of 131k or 262k tokens on a single TPU device, while maintaining a reasonable step time.
We show that model perplexity steadily improves with the size of external memory on a variety of language modelling tasks, including C4 (long documents only), Github code repositories, PG-19 books, formal proofs in Isabelle, and arXiv math papers. We further show that models can generalize to larger memory sizes than they were trained on: models trained with a small memory show gains from using a much larger memory at inference time. Finally, we show that our models are actually using memory in the way that we had hoped, e.g. by looking up the definitions of lemmas in a theorem proving corpus.
The simplicity of the changes to the Transformer architecture allows us to easily integrate this approach into existing code bases, including extremely large language models. We further show that the improvements to quality are maintained across models of increasing size, and that the model improvements gained from adding memory are even larger than increasing the size of the model by 5X or more as shown in Figure 1 .
2 RELATED WORK
A great deal of work has been done on efficient long-range attention mechanisms; see Tay et al. (2020; 2021) recent surveys. Sliding windows (Beltagy et al., 2020 ) use a long sequence, but attend within a smaller window, thus reducing complexity to the window size, rather than total sequence length. Approximate mechanisms such as Linformer (Wang et al., 2020b), and Performer (Choromanski et al., 2021) refactor the attention matrix by using a different kernel than softmax to obtain O(N ) complexity. Pooling strategies such as Hierarchical 1D attention (Zhu & Soricut, 2021), and Combiner (Ren et al., 2021) apply pooling or averaging over tokens at longer distances. Sparse strategies such as Big Bird (Zaheer et al., 2020) select only a subset of tokens to attend to; Routing Transformers (Roy et al., 2021) use clustering to select the subset, while Reformer (Kitaev et al., 2020) relies on hashing. Hierarchical mechanisms (Ainslie et al., 2020) combine multiple tokens into phrases or sentences to reduce sequence length. Expire-span (Sukhbaatar et al., 2021) prunes far-away tokens that it learns are "unimportant". (Zemlyanskiy et al., 2021) process long sequences in two passes with different encoders. The second pass is given a lot of context by accessing summaries of the first pass.
Feedback transformers (Fan et al., 2020 ) use a recurrent architecture in which each token attends to the output of the final layer instead of the previous layer. Recurrence does not increase the size of the attention context itself, but it expands the receptive field at the cost of parallelism and training speed.
Truncated backpropagation through time (Williams & Peng, 1990) was originally introduced as a way of training recurrent neural networks (RNN) over very long sequences, when the entire sequence does not fit in memory. The sequence is chopped into segments, and after each training step, the final RNN state for the segment is saved in a non-differentiable cache, and used as the initial state on the next training step. Neural caches (Grave et al., 2017) extend the cache to contain a record of many prior hidden states, and attend over them. Transformer-XL (Dai et al., 2019) cache. In contrast, we use a very large cache without compression, combined with an approximate kNN attention mechanism over it. Sukhbaatar et al. (2019) make the observation that the feed-forward portion of a transformer layer functions very much like attention if one replaces the ReLU activation with softmax. They implement a combined attention over both tokens from the input sequence and a learned (and differentiable) "memory". Lample et al. (2019) exploit this observation to replace the feed-forward layers (FFNs) with a fast kNN lookup over a much larger "memory", and achieve large gains in model accuracy without significant computation overhead. (We use kNN lookup to approximate attention to previous tokens, not to replace the FFN.) Non-differentiable external memory has been used in different ways by Khandelwal et al. (2020) , who run a pre-trained model over an entire corpus, and construct a large table of (key, token) pairs. They then use that table to replace the final softmax layer for token selection in the model, which results in significant improvements in language modeling. Yogatama et al. (2021) extend this approach by a gating mechanism and a process to compress the context into keys for retrieval.
There are several works that combine retrieval with transformers. REALM (Guu et al., 2020) , MARGE (Lewis et al., 2020a) , RAG (Lewis et al., 2020b) , and composite memory for dialog (Fan et al., 2021) retrieve documents from a knowledge base to improve question answering or dialogue. The knowledge base consists of text snippets and is static and typically separate from the inputs and outputs of the models. Instead, we focus on language modeling using a decoder-only model, and propose a simple model that unifies attention and retrieval.
k-nearest-neighbor lookup is a general-purpose technique that is used for a wide variety of machine learning and retrieval tasks, and high-performance implementations are available for various architectures (Johnson et al., 2021; Guo et al., 2020) . Memory-efficient Transformers (Gupta et al., 2021) replace dense attention with a kNN lookup to increase speed and reduce memory usage.
3 METHOD
The architecture of our kNN-augmented transformer is shown in Figure 2 . The bulk of the model is a vanilla, decoder-only transformer (Vaswani et al., 2017) . The input text is tokenized, and the tokens are embedded into vector space. The embedding vectors are passed through a series of transformer layers, each of which does dense self-attention, followed by a feed-forward network (FFN). Since this is a decoder-only language model, we use a causal attention mask and the token embeddings of the last layer are used to predict the next token.
Long documents are split into subsequences of 512 tokens, and each subsequence is used as the input for one training step. In contrast to standard practice, we do not shuffle the subsequences; instead, each long document is fed into the transformer sequentially, from beginning to end, as is done with Transformer-XL (Dai et al., 2019) . We also use a Transformer-XL style cache, which holds the keys and values from the previous training step. When doing self-attention, the cached keys and values are prepended to the current keys and values, and we use a sliding-window causal mask (Beltagy et al., 2020) so that each token has a local context that includes the previous 512 tokens.
3.1 kNN-AUGMENTED ATTENTION LAYER
One of the transformer layers near the top of the stack is a kNN-augmented attention layer, which combines two forms of attention. Like all of the other layers, it uses standard dense self-attention on the local context, which is the input subsequence for the current training step. Unlike the other layers, however, it also does an approximate k-nearest-neighbor search into the external memory.
The same queries are used for both the local context, and for the external memory. The keys and values also belong to the same distribution; after each training step, the (key, value) pairs in the local context are appended to the end of the external memory. If the document is very long, old (key, value) pairs will be dropped from the memory to make room for new ones. Thus, for each head, the external memory keeps a cache of the prior M (key, value) pairs, where M is the memory size.
The kNN lookup will return a set of retrieved memories, which consist of the top-k (key, value) pairs that kNN search returns for each query (i.e. each token) in the input subsequence. As with standard dense attention, we first construct an attention matrix by computing the dot product of each query against the retrieved keys, then apply softmax, and finally return a weighted sum of the retrieved values. Unlike standard dense attention, the retrieved memories contain a different set of (key, value) pairs for each query.
Attention over the local context is performed in the usual way. The results of kNN-attention and local attention are then combined using a learned gate:
g = σ(b g ) (1) V a = V m g + V c (1 -g) (2)
where σ is the sigmoid function, and is element-wise multiplication. V a is the combined result of attention, V m is the result of attending to external memory, and V c is the result of attending to the local context. The bias b g is a learned per-head scalar parameter, which allows each head to choose between local and long-range attention. In our experiments, the value of the gate g does not depend on the content of the token at each position, although that would be a trivial extension to implement. We did observe that over time, most heads learned to attend almost exclusively to external memory.
Position bias. For dense attention within the local context, we use the T5 relative position bias (Raffel et al., 2020) . As noted by Dai et al. (2019) , adding a global position encoding to each token does not work well when processing long documents. We don't use a position bias for the retrieved memories.
Experiments on the PG19 dataset (Sun et al., 2021) have shown that relative position does not appear to matter at long range, and the T5 relative bias puts all long-range tokens in the same bucket anyway.
Batching. Figure 3 illustrates how multiple long documents of different lengths are packed into a batch, and split into subsequences. Each subsequence in the batch comes from a different document, and thus requires a separate external memory, which is cleared at the start of each new document.
3.2 DISTRIBUTIONAL SHIFT
Because each long document is processed over multiple training steps, there is a distributional shift in the keys and values that are stored in external memory. The model parameters that produce the queries change over time, and will thus have shifted since the keys and values were stored. For very large memories, older records may become "stale." Similar observations have been made for CrossBatch memory (Wang et al., 2020c) in the vision domain.
To reduce the effects of staleness, we normalize keys and queries (Henry et al., 2020) . Normalization does not eliminate staleness, but it at least ensures that older keys and newer keys do not differ in magnitude. We also found that normalization helps stabilize training with the Transformer-XL cache.
In some of our experiments, we observed that training models from scratch with a large memory sometimes resulted in worse performance than pretraining the model with a small memory of size 8192, and then finetuning it on a larger memory. This training instability could be due to staleness. However, models seem to be able to cope with a limited degree of staleness (with the small memory) by adjusting their queries accordingly.
3.3 APPROXIMATE kNN
We employ approximate kNN search rather than exact kNN search because it significantly improves the computational speed of our model. We use a simple approximation of kNN for TPUs, which has a recall of about 90%, i.e. 90% of the true top k are returned in the approximate top k. There are various other efficient approximate kNN algorithms available for CPU and GPU/TPU, for example through Faiss (Johnson et al., 2021) or ScaNN (Guo et al., 2020) , which can scale into the billions.
4 EXPERIMENTS
We evaluate the effect of adding external memory on five language modeling tasks, all of which involve long-form text: English language books (PG-19), long web articles (C4), technical math papers (arXiv Math), source code (Github), and formal theorems (Isabelle). The results show significant improvements in the perplexity of the model with the addition of external memory. We experimented with various sizes of external memory, from 1536 to as high as 262K. On most of the datasets, there was an initial sharp gain from adding a small external memory, followed by smaller but steadily increasing gains as the size of the memory was increased.
4.1 DATASETS
arXiv Math For the arXiv dataset, we collected a corpus of papers by downloading them via the arXiv Bulk Data Access1 . We filtered papers to include only articles labeled as "Mathematics" and whose L A T E X source was available. The number of tokens per paper in this dataset is roughly comparable to the number of tokens per book in PG19, because L A T E X source has many special characters and the tokenizer tends to output small subwords.
Github We used BigQuery2 to obtain a large corpus of Github repositories that are published with open-source licenses. We used file endings to filter for files in the languages C, C++, Java, Python (including Jupyter notebooks), Go, and TypeScript. Individual source code files are often fairly short, and there are many dependencies and cross-references between files in the repository. To capture these dependencies, we created one long document for each Github repository by traversing the directory tree, and concatenating all of the files within it. The order in which files are traversed within the repository is random, but each subdirectory is processed as a unit, so that all the files within the subdirectory are close to each other in the resulting document. Source code is usually structured so that related files are all grouped together in the same subdirectory; this traversal preserves that structure, while still shuffling files and subdirectories in random order.
Formal Math -Isabelle The Isabelle corpus consists of formal mathematical proofs of theories. We collected all 627 theories available on The Archive of Formal Proofs3 (as of October 6, 2021) and an additional 57 theories from the Isabelle standard library4 to create a corpus of 684 theories. All theories have open-source licenses. Each theory is a self-contained mathematical object, on topics such as foundational logic, advanced analysis, algebra, or cryptography, and consists of multiple files containing proofs. As with the Github corpus, all files that make up a theory are concatenated together into one long document. Unlike the Github corpus, we order the files according to their import dependencies, so that later files use sub-theorems that are proved in earlier files.
C4(4K+) C4, the colossal cleaned common crawl, is a very large collection of documents that have been scraped from the internet (Raffel et al., 2020) . We filtered out all documents that have less than 4096 tokens to focus on documents where memory can have an impact.
PG-19 PG-19 is a large dataset of English-language books, published prior to 1919, which were retrieved from the Project Gutenberg archive (Rae et al., 2020; Sun et al., 2021) . PG-19 is one of the few public datasets that only contains full-length books, and has become a benchmark for long-range natural language text modeling.
4.2 EXPERIMENTAL METHOD
We used a 12-layer decoder-only transformer (with and without Transformer-XL cache) with an embedding size of 1024, 8 attention heads of dimension 128, and an FFN hidden layer of size 4096. For all of our experiments, we used k = 32. Unless specified otherwise, we use the 9th layer as the kNN augmented attention layer. We used a sentence-piece (Kudo & Richardson, 2018) tokenizer with a vocabulary size of 32K.
We used the Adafactor optimizer (Shazeer & Stern, 2018) . In preliminary experiments, we conducted a hyperparameter search to determine the optimal learning rate among three choices ({3.0, 1.0, 3 • 10 -1 }), and found that 1.0 works best. We used a linear warmup schedule for the first 1000 steps, followed by square root decay. We trained the models from scratch for 500K steps on all the datasets, except for the Isabelle dataset. Isabelle is small, so we stopped training after 100K steps when the model began to overfit. We ran all of our experiments on 32 TPU cores. Our models were implemented in JAX (Bradbury et al., 2018) and Flax (Heek et al., 2020) .
When comparing models with different context lengths, we adjusted the batch size (the number of documents in a batch) so that there are always 2 17 tokens in a batch. E.g., a model with a context length of 512 has a batch size of 256, while the 2048 model has a batch size of 64.
We experimented with multiple implementations of approximate kNN lookup with different tradeoffs between quality and computational cost. We did not observe a significant degradation of the model quality when switching to lower quality approximations of kNN, so the model appears to be quite robust with respect to the quality of kNN retrieval. For a model with around 200M trainable parameters the step time increased from 0.2s to 0.25s when we added a memory of size 8K, and to 0.6s when we added a memory of size 65K (measured on TPUv3).
4.3 EFFECT OF EXTERNAL MEMORY
Adding external memory results in substantial gains across datasets and architectures, as shown in Table 4 . Across all five datasets, adding external memory to either the vanilla Transformer or the Transformer-XL architecture improves perplexity by a substantial amount. For example, on Increasing the size of the memory increases the benefit of the memory. The best perplexities for all datasets and architectures were obtained with a memory size of 65K.
Note that Transformer-XL with context size 2048 already has a theoretical receptive field that is quite large. Each token in a higher layer can attend up to 2048 tokens away in the layer below, so the total receptive field is 2048 • 12 (layers) ∼ 25K. Nevertheless, we still saw a substantial gain when adding an external memory of size 8192 to this model. kNN attention into memory would appear to be a more effective way to retrieve information from the distant past than the Transformer-XL cache.
On the other hand, we also saw improvements by adding XL cache to the large-memory (65K) models.
In a vanilla (non-XL) Transformer, the first few tokens in a sequence have very little context, and thus have higher perplexity. The XL cache provides additional local short-range context at the start of a sequence, which complements the long-range context provided by external memory.
Interestingly, in a vanilla Transformer, using even a small external memory of size 1536 provides a gain in perplexity which is almost as good as using a local context of size 2048 but no memory (e.g. Table 4 ). This is surprising, because the external memory is not differentiable, and is added only to one layer of the Transformer, whereas increasing the context size is differentiable and affects all layers. We conclude that the lower layers of a Transformer don't necessarily need long-range context, and having a differentiable memory is not as important as one might suspect.
4.4 SCALING TO LARGER MODELS
We scaled up the Transformer model to sizes of 1 and 8 billion parameters. For the 1 billion parameter model, we use 8 layers, 32 heads with head dimension 128, d_model 2048, and d_ff 16384. For the 8 billion parameter model, we use 64 heads, 16 layers, d_model 4096, and d_ff 32768. We used a context size of 2048, memory size of 8192, and no XL cache. We ran the comparisons to the vanilla Transformer on the arXiv math dataset. Scaling plots are shown in Figure 1 .
External memory provides a consistent improvement to the model as it is scaled up. Remarkably, we found that the smaller Memorizing Transformer with just 8k tokens in memory can match the perplexity of a larger vanilla Transformer which has 5X more trainable parameters.
4.5 FINETUNING ON LARGER MEMORIES
Finetuning on a larger memory. In some cases, training was unstable when using large memories, possibly due to distributional shift early in the training (See Section 3.2). Thus, for memories of 131K or more tokens, we first pretrain the model with a memory size of 8192 or 65K for 500K steps, and then finetune it with the larger memory for an additional 20K steps. The results of finetuning on the arXiv Math data set are shown in Finetuning a non-memory model to use memory Pretraining can be very costly both in time and computational resources. Thus, a natural question to ask is: can one fine-tune a pretrained Transformer to use external memory? The answer is yes! We took a pre-trained 1B vanilla Transformer model, and fine-tuned it to use external memory (the 1B models used in Section 4.4). The fine-tuning result is shown in Figure 6 . Notice that the model quickly learns to use external memory. Within 20K steps (4% of the pre-training time) the fine-tuned model has already closed 85% of the gap between it and the 1B Memorizing Transformer, and after 100k steps it has closed the gap entirely.
4.6 INFORMATION RETRIEVAL PATTERNS
We conducted a qualitative study of what the model was actually retrieving from external memory, by finding which tokens showed the biggest improvements in cross-entropy loss when the size of the memory was increased, and then examining the top-k retrieved memories for those tokens. We found that the model gained the most when looking up rare words, such as proper names, references, citations, and function names, where the first use of a name is too far away from subsequent uses to fit in the local context. This result is in keeping with the prior analysis of long-context Transformers on PG19 (Sun et al., 2021) , which found similar lookup patterns. For this experiment, we used a slightly older version of the architecture without the gating mechanism.
Which tokens show a benefit from memory? Figure 7 shows a visualization of which tokens show an improvement when the size of the external memory is increased. We selected a math paper at random, and plotted the difference in cross entropy loss for each token x i in the paper, comparing two models with the same parameters, but with memories of different sizes. ∆ i = cross-entropy 8192 (x i ) cross-entropy 32K (x i ). Positive values show an improvement in loss.
The x-axis on the chart is the token number i, while the y-axis is ∆ i . For the first 8192 tokens, the difference between the two models is zero, since the larger capacity of the 32K memory isn't being used yet. However, after token 8193, we can see that the larger memory helps, on average, over the smaller memory. The benefit is not universal, since the predictions for some tokens become worse, possibly due to the fact that a relevant retrieved memory no longer makes it into the top-k when the size of the external memory is increased. This figure also shows that the benefit of external memory is somewhat sparse. The improvement in perplexity seems to be mainly driven by a small percentage of tokens that obtain a large improvement in cross-entropy loss when using the larger memory.
What information is being looked up? Given that only a subset of tokens shows improvement from external memory, we did a further investigation into what, exactly, those tokens are using the memory for. We took those tokens which showed the largest improvement in cross-entropy loss, and for each of them tokens, we examined the top-k retrieved memories. We studied arXiv math, Github and Isabelle corpus. For arXiv math and Github, we found the model retrieved function and variable names. See more details with examples in Appendix B. Retrieving mathematical definitions. Our case study on the Isabelle corpus provides one of the clearest illustrations of how a model can make good use of external memory. When predicting the name of a mathematical object or a lemma, the model looked up the definition from earlier in the proof. Examples of this behavior are shown in Table 8 . In example 1, the model retrieves a definition within the body of a lemma, markov_inequality. In example 2, it retrieves the definition of a previously defined concept subgraph_threshold. In example 3, it retrieves the definition of orthonormal_system. We manually checked 10 examples where the model made a prediction of lemma names, and 8 out of 10 times model found the body of the lemma it needs to predict. In the other two cases, the model also looked up materials in the immediate vicinity. To the best of our knowledge, this is the first demonstration that attention is capable of looking up definitions and function bodies from a large corpus. The Isabelle case study used a model with two memory layers of size 32K.
5 CONCLUSION
We present a simple extension to the Transformer architecture, called kNN-augmented attention, which dramatically increases the length of the context that a language model can attend to by using k-nearest-neighbor lookup into a large external memory. We demonstrate the effectiveness of external memory in a series of language modeling experiments over a variety of long-document datasets, including LaTeX documents, source code, formal proofs, and books.
The Memorizing Transformer shows large improvements in perplexity over the baseline for all of the data sets and architectures that we studied; it is comparable to a vanilla transformer that has 5 times the number of parameters. Perplexity continues to improve with increasing memory size, although there is a point of diminishing returns. Moreover, external memory continues to provide benefits even as the transformer is scaled up from 200M to 8B parameters. Perhaps most intriguingly, a Memorizing Transformer does not need to be pre-trained from scratch; it is possible obtain large gains from adding memory to an existing pre-trained model, and then fine-tuning it.
Unlike other forms of attention, kNN retrieval can be easily scaled up to huge memory sizes, and is thus potentially able to leverage vast knowledge bases or code repositories. How to make the best use of this capability is a topic for future work.
A.1 ABLATION STUDIES
In the following section, we performed ablation studies to investigate the effects of various hyperparameters. Unless otherwise specified, we carried out these experiments with a memorizing transformer with context size 512, XL cache 512 with a memory size of 8192.
Multiple kNN layers. We experimented with using two kNN layers, rather than just one. However, we did not see further benefits brought by more than multiple retrieval layers.
kNN layer index We experimented with adding the external memory to layer 3, 6, 9 and 12 in a 12-layer transformer, with results shown in Table 14 . We found that adding memory to the middle of the layer stack will obtain the best result, whereas adding memory to layers either too close to the input or to the output obtained less gains. Random seeds We measured the statistical significant of the results reported. We did 3 runs with 3 random seeds for Transformer XL of size 512, and also a memorizing transformer with memory size 8192. We measured the standard deviation of perplexities after 500K steps of training, shown in Table 16 . We saw the standard deviation between different runs of the same experiment appears to be much smaller than the gap between different models. 
B WHAT DOES THE MODEL RETRIEVE FROM MEMORY?
Retrieving citation names On arXiv math, several examples are shown in Table 17 , which includes both the retrieved token and its surrounding context. We observe that many of the gains in crossentropy loss took place when trying to predict the name of bibitems, citations, or references, by looking up the references and citations used previously in the paper. Such lookups usually span over the entire paper, which is much longer than 8192 tokens, providing a plausible explanation for the gain beyond memory size of 8192. Retrieving function names from the codebase As with the arXiv papers, we also studied which tokens the model retrieved from memory. As might be expected, the model is often looking up the names of functions, and variables, as shown in Table 18 . 
https://arxiv.com/help/bulk_data
https://console.cloud.google.com/marketplace/product/github/github-repos
https://www.isa-afp.org/topics.html
https://isabelle.in.tum.de/
References
[1]Ainslie Joshua, Ontañón Santiago, Alberti Chris, Cvicek Vaclav, Fisher Zachary, Pham Philip, et al.. ETC: encoding long and structured inputs in transformers. EMNLP 2020;:.
[2]Austin Jacob, Odena Augustus, Nye Maxwell, Bosma Maarten, Michalewski Henryk, Dohan David, et al.. Program synthesis with large language models.  2021;:.
[3]Beltagy Iz, Peters MatthewE, Cohan Arman. Longformer: The long-document transformer.  2020;:.
[4]Bradbury James, Frostig Roy, Hawkins Peter, Johnson MatthewJames, Leary Chris, Maclaurin Dougal, et al.. JAX: composable transformations of Python+NumPy programs.  2018;:.
[5]Tom B, Brown Benjamin, Mann Nick, Ryder Melanie, Subbiah Jared, Kaplan Prafulla, et al.. Language models are few-shot learners. NeurIPS 2020;:.
[6]Krzysztof Marcin Choromanski Valerii, Likhosherstov David, Dohan Xingyou, Song Andreea, Gane Tamás, Sarlós Peter, et al.. Rethinking attention with performers. ICLR 2021;:.
[7]Cobbe Karl, Kosaraju Vineet, Bavarian Mohammad, Hilton Jacob, Nakano Reiichiro, Hesse Christopher, et al.. Training verifiers to solve math word problems.  2021;:.
[8]Dai Zihang, Yang Zhilin, Yang Yiming, Carbonell JaimeG, Viet Le Quoc, Salakhutdinov Ruslan. Transformer-XL: Attentive language models beyond a fixed-length context. ACL 2019;:.
[9]Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina. BERT: pre-training of deep bidirectional transformers for language understanding. ACL 2019;:.
[10]Fan Angela, Lavril Thibaut, Grave Edouard, Joulin Armand, Sukhbaatar Sainbayar. Addressing some limitations of transformers with feedback memory.  2020;:.
[11]Fan Angela, Gardent Claire, Braud Chloé, Bordes Antoine. Augmenting transformers with KNN-based composite memory for dialog. Transactions of the Association for Computational Linguistics 2021;9:82--99.
[12]Grave Edouard, Joulin Armand, Usunier Nicolas. Improving neural language models with a continuous cache. ICLR 2017;:.
[13]Guo Ruiqi, Sun Philip, Lindgren Erik, Geng Quan, Simcha David, Chern Felix, et al.. Accelerating large-scale inference with anisotropic vector quantization. ICML 2020;:.
[14]Gupta Ankit, Dar Guy, Goodman Shaya, Ciprut David, Berant Jonathan. Memory-efficient transformers via top-k attention.  2021;:.
[15]Guu Kelvin, Lee Kenton, Tung Zora, Pasupat Panupong, Chang Ming-Wei. Retrieval augmented language model pre-training. ICML 2020;:.
[16]Hahn Christopher, Schmitt Frederik, Kreber JensU, Norman Rabe Markus, Finkbeiner Bernd. Teaching temporal logics to neural networks. In ICLR 2021;:.
[17]Heek Jonathan, Levskaya Anselm, Oliver Avital, Ritter Marvin, Rondepierre Bertrand, Steiner Andreas, et al.. Flax: A neural network library and ecosystem for JAX.  2020;:.
[18]Henry Alex, Prudhvi Raj, Dachapally Shubham, Shantaram Pawar Yuxuan, Chen . Query-key normalization for transformers. EMNLP 2020;:.
[19]Johnson Jeff, Douze Matthijs, Jégou Hervé. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 2021;:.
[20]Khandelwal Urvashi, Levy Omer, Jurafsky Dan, Zettlemoyer Luke, Lewis Mike. Generalization through memorization: Nearest neighbor language models. ICLR 2020;:.
[21]Kitaev Nikita, Kaiser Łukasz, Levskaya Anselm. Reformer: The efficient transformer. ICLR 2020;:.
[22]Kudo Taku, Richardson John. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP 2018;:.
[23]Lample Guillaume, Sablayrolles Alexandre, Ranzato Marc'aurelio, Denoyer Ludovic, Jégou Hervé. Large memory layers with product keys.  2019;:.
[24]Lewis Mike, Ghazvininejad Marjan, Ghosh Gargi, Aghajanyan Armen, Wang Sida, Zettlemoyer Luke. Pre-training via paraphrasing.  None;:.
[25]Lewis Patrick, Perez Ethan, Piktus Aleksandra, Petroni Fabio, Karpukhin Vladimir, Goyal Naman, et al.. Retrieval-augmented generation for knowledge-intensive NLP tasks.  None;:.
[26]Li Wenda, Yu Lei, Wu Yuhuai, Paulson LawrenceC. Isarstep: a benchmark for high-level mathematical reasoning. ICLR 2021;:.
[27]Polu Stanislas, Sutskever Ilya. Generative language modeling for automated theorem proving.  2020;:.
[28]Norman Rabe Markus, Lee Dennis, Bansal Kshitij, Szegedy Christian. Mathematical reasoning via self-supervised skip-tree training. ICLR 2021;:.
[29]Jack W Rae Anna, Potapenko , Siddhant M, Jayakumar Chloe, Hillier TimothyP, Lillicrap . Compressive transformers for long-range sequence modelling. ICLR 2020;:.
[30]Raffel Colin, Shazeer Noam, Roberts Adam, Lee Katherine, Narang Sharan, Matena Michael, et al.. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 2020;:.
[31]Hongyu Ren Hanjun, Dai Zihang, Dai Mengjiao, Yang Jure, Leskovec Dale, Schuurmans Bo, et al.. Combiner: Full attention transformer with sparse computation cost.  2021;:.
[32]Roy Aurko, Saffar Mohammad, Vaswani Ashish, Grangier David. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics 2021;9:53--68.
[33]Shazeer Noam, Stern Mitchell. Adafactor: Adaptive learning rates with sublinear memory cost. ICML 2018;:.
[34]Sukhbaatar Sainbayar, Grave Edouard, Lample Guillaume, Jegou Herve, Joulin Armand. Augmenting self-attention with persistent memory.  2019;:.
[35]Sukhbaatar Sainbayar, Ju Da, Poff Spencer, Roller Stephen, Szlam Arthur, Weston Jason, et al.. Not all memories are created equal: Learning to forget by expiring. ICML 2021;:.
[36]Sun Simeng, Krishna Kalpesh, Mattarella-Micke Andrew, Iyyer Mohit. Do long-range language models actually use long-range context?. EMNLP 2021;:.
[37]Tay Yi, Dehghani Mostafa, Bahri Dara, Metzler Donald. Efficient transformers: A survey.  2020;:.
[38]Tay Yi, Dehghani Mostafa, Abnar Samira, Shen Yikang, Bahri Dara, Pham Philip, et al.. Long range arena: A benchmark for efficient transformers. ICLR 2021;:.
[39]Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez AidanN, et al.. Attention is all you need.  2017;:.
[40]Wang Qingxiang, Brown Chad, Kaliszyk Cezary, Urban Josef. Exploration of neural machine translation in autoformalization of mathematics in mizar. International Conference on Certified Programs and Proofs 2020;:.
[41]Wang Sinong, Li BelindaZ, Khabsa Madian, Fang Han, Ma Hao. Linformer: Self-attention with linear complexity.  2020;:.
[42]Wang Xun, Zhang Haozhi, Huang Weilin, Scott MatthewR. Cross-batch memory for embedding learning. CVPR 2020;:.
[43]Williams RonaldJ, Peng Jing. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation 1990;:.
[44]Yogatama Dani, De Masson D'autume Cyprien, Kong Lingpeng. Adaptive semiparametric language models. ACL 2021;9:362--373.
[45]Zaheer Manzil, Guruganesh Guru, Kumar Avinava, Dubey Joshua, Ainslie Chris, Alberti Santiago, et al.. Big bird: Transformers for longer sequences.  2020;:.
[46]Zemlyanskiy Yury, Ainslie Joshua, Michiel De Jong Philip, Pham Ilya, Eckstein Fei, Sha . H-transformer-1d: Fast one-dimensional hierarchical attention for sequences. ACL: Human Language Technologies, 2021. Zhenhai Zhu and Radu Soricut 2021;:.
[2203.08913] Memorizing Transformers
https://arxiv.org/abs/2203.08913
https://arxiv.org/abs/2004.05150
Longformer: The Long-Document Transformer
2020-04-10
Beltagy; Iz; Peters; Matthew E; Cohan; Arman
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2004.05150">Download PDF</a></p><blockquote>
 Abstract: Transformer-based models are unable to process long sequences due to their
self-attention operation, which scales quadratically with the sequence length.
To address this limitation, we introduce the Longformer with an attention
mechanism that scales linearly with sequence length, making it easy to process
documents of thousands of tokens or longer. Longformer's attention mechanism is
a drop-in replacement for the standard self-attention and combines a local
windowed attention with a task motivated global attention. Following prior work
on long-sequence transformers, we evaluate Longformer on character-level
language modeling and achieve state-of-the-art results on text8 and enwik8. In
contrast to most prior work, we also pretrain Longformer and finetune it on a
variety of downstream tasks. Our pretrained Longformer consistently outperforms
RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop
and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a
Longformer variant for supporting long document generative sequence-to-sequence
tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Iz Beltagy [<a href="/show-email/65b5ac40/2004.05150">view email</a>]
 <br />
 <strong><a href="/abs/2004.05150v1">[v1]</a></strong>
 Fri, 10 Apr 2020 17:54:09 UTC (228 KB)<br /><strong>[v2]</strong>
Wed, 2 Dec 2020 17:52:35 UTC (1,357 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2004.05150

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:2004.05150 (cs)
 [Submitted on 10 Apr 2020 (v1), last revised 2 Dec 2020 (this version, v2)]

 Title: Longformer: The Long-Document Transformer

 Authors: Iz Beltagy, Matthew E. Peters, Arman Cohan
 Download PDF
 Abstract: Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
 Comments: Version 2 introduces the Longformer-Encoder-Decoder (LED) model
 Subjects: Computation and Language (cs.CL) 
 Cite as: arXiv:2004.05150 [cs.CL] 
 (or arXiv:2004.05150v2 [cs.CL] for this version) 
 https://doi.org/10.48550/arXiv.2004.05150 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Iz Beltagy [view email]
 [v1] Fri, 10 Apr 2020 17:54:09 UTC (228 KB)
 [v2] Wed, 2 Dec 2020 17:52:35 UTC (1,357 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 2004
 Change to browse by:
 cs

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 3 blog links

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Iz Beltagy
 Matthew E. Peters
 Arman Cohan
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Longformer: The Long-Document Transformer
Date Published: 2020-04-10
Authors:
Iz  Beltagy, Allen Institute for Artificial Intelligence, Seattle, WA, USA, beltagy@allenai.org
Matthew E Peters, Allen Institute for Artificial Intelligence, Seattle, WA, USA, matthewp@allenai.org
Arman  Cohan, Allen Institute for Artificial Intelligence, Seattle, WA, USA, armanc@allenai.org
Abstract
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on Wiki-Hop and TriviaQA. 1
1 Introduction
Transformers (Vaswani et al., 2017) have achieved state-of-the-art results in a wide range of natural language tasks including generative language modeling (Dai et al., 2019; Radford et al., 2019) and discriminative language understanding (Devlin et al., 2019) . This success is partly due to the self-attention component which enables the network to capture contextual information from the entire sequence. While powerful, the memory and computational requirements of self-attention grow quadratically with sequence length, making it infeasible (or very expensive) to process long sequences on current hardware.
To address this limitation, we present Longformer, a modified Transformer architecture with Figure 1 : Longformer's memory usage scales linearly with the sequence length, unlike the full self-attention mechanism that runs out of memory for long sequences on current GPUs. Longformer's GPU-kernel is nearly as fast as the highly optimized full self-attention operation, and nearly 6X faster than naive Pytorch. a self-attention operation that scales linearly with the sequence length, making it versatile for processing long documents (Fig. 1 ). This is an advantage for natural language tasks such as long document classification, question answering (QA), and coreference resolution, where existing approaches partition or shorten the long context into smaller sequences that fall within the typical 512 token limit of BERT-style pretrained models. Such partitioning could potentially result in loss of important cross-partition information, and to mitigate this problem, existing methods often rely on complex architectures to address such interactions. On the other hand, our proposed Longformer is able to build contextual representations of the entire context using multiple layers of attention, reducing the need for task-specific architectures.
Recent work has addressed the computational inefficiency of Transformers on long sequences (see Tab. 1). However, they primarily focus on autoregressive language modeling, while the application of long document transformers to document-level NLP tasks in the transfer learning setting (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019) has remained largely unexplored. We address this gap and show that Longformer's attention mechanism can act as a drop-in replacement for the self-attention mechanism in pretrained Transformers, and leads to gains across a suite of document NLP tasks.
Longformer's attention mechanism is a combination of a windowed local-context self-attention and an end task motivated global attention that encodes inductive bias about the task. Through ablations and controlled trials we show both attention types are essential -the local attention is primarily used to build contextual representations, while the global attention allows Longformer to build full sequence representations for prediction.
We first evaluate Longformer on autoregressive character-level language modeling using a combination of windowed and a new dilated attention pattern, allowing the model to process sequences of up to 32K characters on modern GPUs. We achieve state-of-the-art results on text8 and enwik8 benchmark datasets, demonstrating the effectiveness of Longformer in long document modeling.
Then, to evaluate Longformer's ability to replace the full self-attention operation of existing pretrained models, we pretrain it with the masked language modeling (MLM) objective, continuing from the RoBERTa (Liu et al., 2019) released checkpoint. After pretraining, we apply it to downstream language tasks through finetuning and demonstrate that Longformer consistently outperforms RoBERTa on a wide range of document-level natural language tasks including text classification, QA, and coreference resolution, achieving state-ofthe-art results on two of these datasets.
2 Related Work
Long-Document Transformers Tab. 1 summarizes recent prior work on long documents. Two types of self-attention approaches have been explored. The first is a left-to-right (ltr) approach that processes the document in chunks moving from left-to-right. While such models have been successful in autoregressive language modeling, they are unsuitable for transfer learning approaches with tasks that benefit from bidirectional context.
Our work falls within the other general approach that defines some form of sparse attention pattern and avoids computing the full quadratic attention matrix multiplication. The model with the most similar attention pattern to ours is Sparse Transformer (Child et al., 2019) , which uses a form of dilated sliding window of blocks of size 8x8 provided by BlockSparse (Gray et al., 2017) . While both models use custom CUDA kernels, ours is implemented in TVM (Chen et al., 2018 ) ( §3.2) making it more customizable and maintainable than BlockSparse which is implemented in C++, and designed for a specific versions of TensorFlow. We also introduce additional task motivated global attention patterns suitable for common NLP tasks ( §3) and show they are essential for good performance in the transfer learning setting.
A few models tried tasks other than autoregressive language modeling, which is a step forward because arguably focusing on language modeling as the primary evaluation has led to the development of models with limited applicability. BP-Transformer (Ye et al., 2019) evaluated on machine translation (MT), but didn't explore the pretrainfinetune setting. Blockwise attention (Qiu et al., 2019) pretrained their models and evaluated on question answering (QA). However, the evaluation is limited as it doesn't include language modeling, and the QA datasets are of relatively short documents,2 therefore the effectiveness of this model on long document tasks remains unexplored.
Task-specific Models for Long Documents
Many task-specific approaches have been developed to workaround the 512 limit of pretrained transformer models like BERT. The simplest approach just truncates the document, commonly used for classification (Xie et al., 2019 ). Another approach chunks the document into chunks of length 512 (could be overlapping), processes each chunk separately, then combines the activations with a task specific model (Joshi et al., 2019) . A third approach popular for multihop and open domain QA tasks uses a two-stage model where the first stage retrieves relevant documents that are passed onto the second stage for answer extrac- tion (Clark and Gardner, 2017; Chen et al., 2017) .
All of these approaches suffer from information loss due to truncation or cascading errors from the two stage approach. In contrast, Longformer can process long sequences without truncating or chunking, allowing us to adopt a much simpler approach that concatenates the available context and processes it in a single pass.
3 Model
The original Transformer model has a self-attention component with O(n 2 ) time and memory complexity where n is the input sequence length and thus, is not efficient to scale to long inputs. To address this challenge, we sparsify the full self-attention matrix according to an "attention pattern" specifying pairs of input locations attending to one another. Unlike the full self-attention, our proposed attention pattern scales linearly with the input sequence, making it efficient for longer sequences. In the following, we discuss the design and implementation of this attention pattern.
3.1 Attention Pattern
Fig. 2 summarizes the configurations of our proposed attention pattern.
Sliding Window Given the importance of local context (Kovaleva et al., 2019) , our attention pattern employs a fixed-size window attention surrounding each token. Using multiple stacked layers of such windowed attention results in a large receptive field, where top layers have access to all input locations and have the capacity to build representations that incorporate information across the entire input. More formally, in this attention pattern, given a fixed window size w, each token attends to 1 2 w tokens on each side (Fig. 2b ). The computation complexity of this pattern is O(n×w), which scales linearly with input sequence length n. To make this attention pattern efficient, w should be small compared with n. However, as mentioned above, a model with typical multiple stacked transformer will have a large receptive field. This is analogues to CNNs where stacking layers of small kernels leads to high level features that are built from a large portion of the input (receptive field) (Wu et al., 2019) . In our case, with a transformer of layers, the receptive field size is × w (assuming w is fixed for all layers). Depending on the application, it might be helpful to use different values of w for each layer to balance between efficiency and model representation capacity ( §4.1).
Dilated Sliding Window
To further increase the receptive field without increasing computation, the sliding window can be "dilated". This is analogues to dilated CNNs (van den Oord et al., 2016) where the window has gaps of size dilation d (Fig. 2c ). Assuming a fixed d and w for all layers, the receptive field is × d × w, which can reach tens of thousands of tokens even for small values of d.
In multi-headed attention, each attention head computes a different attention score. We found settings with different dilation configurations per head improves performance by allowing some heads without dilation to focus on local context, while others with dilation focus on longer context.
Global Attention
In state-of-the-art BERT-style models for natural language tasks, the optimal input representation differs from language modeling and varies by task. For masked language modeling (MLM), the model uses local context to predict the masked word, while for classification, the model aggregates the representation of the whole sequence into a special token ([CLS] in case of BERT). For QA, the question and document are concatenated, allowing the model to compare the question with the document through self-attention.
In our case, the windowed and dilated attention are not flexible enough to learn task-specific representations. Accordingly, we add "global attention" on few pre-selected input locations. Importantly, we make this attention operation symmetric: that is, a token with a global attention attends to all tokens across the sequence, and all tokens in the sequence attend to it. Fig. 2d shows an example of a sliding window attention with global attention at a few tokens at custom locations. For example for classification, global attention is used for the [CLS] token while in QA global attention is provided on all question tokens. Note that, since the number of such tokens is small relative and independent of n (the total number of tokens in the input sequence) the complexity of the combined local and global attention is still O(n).
While specifying global attention is task specific, it is much simpler than existing task specific approaches that chunk/shorten the input into smaller sequences and often use complex architecture to combine information across these chunks. Further, it increases the representational power of the model as it allows building contextual representations across the entire sequence.
Linear Projections for Global Attention Recall that given the linear projections Q, K, V , the Transformer model (Vaswani et al., 2017) computes attention scores as follows:
Attention(Q, K, V ) = softmax QK T √ d k VWe use two sets of projections, Q s , K s , V s to compute attention scores of sliding window attention, and Q g , K g , V g to compute attention scores for the global attention. The additional projections provide flexibility to model the different types of attention, which we show is critical for best performance on downstream tasks. Q g , K g , V g are all initialized with values that match Q s , K s , V s .
3.2 CUDA Kernels for our Attention Pattern
In regular transformers, attention scores are computed as in Eqn. 1. The expensive operation is the matrix multiplication QK T because both Q and K have n (sequence length) projections.
Our dilated sliding window attention pattern is not straightforward to implement using modern deep learning libraries. Implementing it requires a form of banded matrix multiplication (matrix multiplication where the output is all zero except certain diagonals) that is not supported in existing deep learning libraries like PyTorch/Tensorflow. In addition, naive implementations with loops is unusably slow. To address this challenge, we provide a highly efficient custom CUDA kernel that implements these attention operations and allows parallelizing the operation on GPU threads.
Tensor Virtual Machine (TVM) We build our custom CUDA kernel using TVM (Chen et al., 2018) , a deep learning compiler stack that compiles high level description of a function into optimized device-specific code. Using TVM, we describe our form of banded matrix multiplication in high-level python constructs, then TVM generates the corresponding CUDA code and compiles it for GPUs.
Fig. 1 compares the runtime and memory of the full self-attention, our TVM implementation of the dilated sliding window, and a naive implementation of this attention. It is clear the full attention implementation is fast but it takes significant amount of memory because it needs to store all n 2 values. The naive implementation with loops is not memory consuming because it only stores the non-zero values, however it is significantly slow and impractical to use. Finally, our implementation is both fast and memory efficient because it only computes and stores the non-zero values.3 
4 Autoregressive Language Modeling
Autoregressive or left-to-right language modeling is loosely defined as estimating the probability distribution of an existing token/character given its previous tokens/characters in an input sequence. This task is considered one of the fundamental tasks in natural language and recent prior work on modeling long sequences using transformers has relied on this task as their primary evaluation (Dai et al., 2019; Rae et al., 2020; Sukhbaatar et al., 2019) . Similarly, we develop and evaluate our model on autoregressive language modeling.
4.1 Attention Pattern
For autoregressive language modeling we use our dilated sliding window attention. Following Sukhbaatar et al. (2019) we use differing window sizes across the layers. In particular, we use small window sizes for the lower layers and increase window sizes as we move to higher layers. This allows the top layers to learn higher-level representation of the entire sequence while having the (Dai et al., 2019) 41M -1.06 Reformer (Kitaev et al., 2020) --1.05 Adaptive (Sukhbaatar et al., 2019) 39M 1.04 1.02 BP-Transformer (Ye et al., 2019) 38M -1.02 Our Longformer 41M 1.02 1.00 lower layers capture local information. In addition, it provides balance between efficiency (smaller window sizes are less computationally expensive due to fewer nonzero values) and performance (larger window sizes have richer representation power and often result in performance improvements).
We do not use dilated sliding windows for lower layers to maximize their capacity to learn and utilize the immediate local context. For the higher layers, we use a small amount of increasing dilation only on 2 heads. This gives the model the ability to directly attend to distant tokens without sacrificing local context.
4.2 Experiment Setup
Task and Datasets We focus on character-level autoregressive language modeling because the sequences are naturally longer than those of wordlevel language modeling, making them suitable for evaluating our model. We used text8 and enwik8 (Mahoney, 2009) for evaluation, two standard and widely used character-level language modeling datasets.
Training Ideally, we would like to train our model on the largest window size and sequence length we can fit in a modern GPU memory. However, we found that the model needs a large number of gradient updates to learn the local context first; before learning to utilize longer context. To accom-modate this, we adopt a staged training procedure where we increase the attention window size and sequence length across multiple training phrases. In particular, in the first phase we start with a short sequence length and window size, then on each subsequent phase, we double the window size and the sequence length, and halve the learning rate. This makes training fast, while keeping the slow part (longest sequences and window sizes) to the end. We train the model over 5 total phases with starting sequence length of 2,048 and ending sequence length of 23,040 on the last phase (see Appendix A for detailed configurations of each phase).
Evaluation At evaluation time, we are able to run our model on sequence of length 32,256. Following prior work (Dai et al., 2019; Sukhbaatar et al., 2019) , during evaluation we split the dataset into overlapping sequences of size 32,256 with a step of size 512, and report the performance on the last 512 tokens on the sequence.
Hyperparameters Our model only specifies how the self-attention component works, and it is agnostic to the other design choices for the transformer model. We use relative position embeddings with sinusoidal weights as in Dai et al. (2019) . We use two different model sizes, a small (12 layers, 512 hidden size) model as in Dai et al. (2019) , and a large (30 layers, 512 hidden size) model as in Child et al. (2019) . We employed mixed precision training (floating points 16 and 32) using apex4 to reduce memory consumption and speed-up training. However, we kept the attention computation in fp32 to avoid numerical instability issues. 5 We used gradient checkpointing (Chen et al., 2016) to reduce memory usage, and ran our experiments on 48GB RTX8000 GPUs. Refer to Appendix A for a more detailed list of hyperparameters.
4.2.1 Results
Tab. 2 and 3 summarize evaluation results on text8 and enwik8 datasets. We achieve a new state-of-the-art on both text8 and enwik8 using the small models with BPC of 1.10 and 1.00 on text8 and enwik8 respectively, demonstrating the effectiveness of our model.
For large models, given how expensive these experiments are, and following recent work (Kitaev et al., 2020; Rae et al., 2020) et al., 2020) are not good fit for the pretrainingfinetuning paradigm as discussed in §2.
4.2.2 Ablation Study
To show the importance of the design choices of our attention patterns, we tried different variants and report their controlled experiment results. To make the ablation study more manageable, we train each configuration for 150K steps6 with phase 1 configuration on a small model on text8, then report the BPC performance on the dev set. The top of Tab. 4 demonstrates the impact of different ways of configuring the window sizes per layer. We observe that increasing the window size from the bottom to the top layer leads to the best performance, arranging them in the reverse way leads to worse performance, and using a fixed window size (the average of window sizes of the other configuration) leads to a performance that it is in between. The bottom of Tab. 4 shows the impact of adding dilation. Adding some dilation to two heads leads to some improvement compared with no dilation at all.
5 Pretraining and Finetuning
Current state-of-the-art systems for many NLP tasks finetune a pretrained model with task supervision (e.g. BERT). One of our main motivations is to develop such a model suitable for long document tasks. To do so, we pretrained Longformer
Source
Tokens Avg doc len Books (Zhu et al., 2015) 0.5B 95.9K English Wikipedia 2.1B 506 Realnews (Zellers et al., 2019) 1.8B 1.7K Stories (Trinh and Le, 2018) 2.1B 7.8K We pretrain Longformer with masked language modeling (MLM), where the goal is to recover randomly masked tokens in a sequence. Since MLM pretraining is expensive, we continue pretraining from the RoBERTa (Liu et al., 2019) released checkpoint, while only making the minimal changes necessary to support Longformer's attention mechanism. Note that our attention pattern can be plugged into any pretrained transformer model without the need to change the model architecture.
Attention Pattern We use the sliding window attention with window size of 512 on all layers. This matches RoBERTa's sequence length, and therefore uses the same amount of computation as RoBERTa. 8Position Embeddings RoBERTa uses learned absolute position embeddings with the maximum position being 512. To support longer documents, we add extra position embeddings to support up to position 4,096. To leverage RoBERTa's pretrained weights, instead of randomly initializing the new position embeddings, we initialize them by copying the 512 position embeddings from RoBERTa multiple times as analysis of BERT's attention heads shows a strong learned bias to attending to local context, including the previous or next token (Clark et al., 2019) . Using the copy initialization preserves this local structure everywhere except at the partition boundaries. Despite its simplicity, we found this to be a very effective (see Tab. 6), allowing Longformer pretraining to rapidly converge with a small number of gradient updates.
Pretraining Data In order to allow the model to learn long dependencies in pretraining, we compiled a corpus of long documents. Some of these data sources were also included in the original RoBERTa pretraining including the Books corpus (Zhu et al., 2015) plus English Wikipedia. We additionally included one third of a subset of the Realnews dataset (Zellers et al., 2019) with documents longer than 1,200 tokens as well as one third of the Stories (Trinh and Le, 2018) corpus. Our goal was to include a mix of long and short documents to both allow the model to learn longer dependencies while not to forget information from the original RoBERTa pretraining. The statistics of the pretraining data is shown in Tab. 5.
Continued MLM Pretraining
We train 9 two sizes of Longformer, a base model and a large model. Both models are trained for 65K gradient updates with sequences length 4,096, batch size 64 (2 18 tokens), maximum learning rate of 3e-5, linear warmup of 500 steps, followed by a power 3 polynomial decay. The rest of the hyperparameters are the same as RoBERTa.
Tab. 6 shows the BPC on the development set of our training corpus. The first row shows a 1.846 BPC using RoBERTa-base, which is comparable to the 1.880 BPC reported on the RoBERTa paper on their corpus. This indicates our training corpus is from a distribution close to that used to train RoBERTa. The following two rows show the performance of Longformer before pretraining with randomly initialized position embeddings and with copied position embeddings. The significant difference indicates the importance of the copy initialization, and the relative small difference between the RoBERTa BPC and the initialized BPC indicates that our sliding window attention is working well with the RoBERTa weights. The following two rows show the impact of continuing pretraining. Traininig for 2K steps improves BPC from 1.957 to 1.753, which further decreases to 1.705 after 65K steps, demonstrating the model is learning to better utilize the sliding window attention and longer context. Similar patterns are observed with RoBERTa-large and Longformer-large.
5.1 Tasks
We apply Longformer to multiple long document tasks, including QA, coreference resolution and classification. Tab. 7 shows the evaluation datasets 9 using fairseq (Ott et al., have contexts significantly longer than 512 wordpieces. Our primary goal is to evaluate whether our attention mechanism can act as a replacement for the standard self-attention mechanism in BERT style models, and to perform controlled trials against a strong baseline. We are also interested in evaluating whether we can replace complicated task specific models necessitated by BERT's limited context with simpler models that just concatenate all available context into a single sequence.
Our baseline is a RoBERTa based model that breaks the context into the longest possible segment, passes each individually through RoBERTa, and concatenates the activations for further processing. For QA tasks, we also concatenate the question to each segment so that RoBERTa can condition it's contextual representations of the context on the question. The Longformer variant replaces the RoBERTa self-attention mechanism with our windowed attention used during pretraining, plus a task motivated global attention. The global attention uses additional linear projections ( §3.1).
Question answering
We consider three datasets with long contexts: WikiHop (Welbl et al., 2018) , Wikipedia setting of TriviaQA (Joshi et al., 2017) , and HotpotQA, distractor dev setting (Yang et al., 2018) . 10For WikiHop and TriviaQA we follow the simple QA model of BERT (Devlin et al., 2019) dataset-specific prediction layer. WikiHop uses a classification layer for the candidate while Trivi-aQA uses the loss function of Clark and Gardner (2017) to predict answer span. We include global attention to question tokens and answer candidates for WikiHop and to question tokens for TriviaQA.
For HotpotQA, we experiment with two different models, a single stage multitask model that jointly predicts evidence and answer spans, and a twostage model that first extracts evidence paragraphs and passes them to a second stage for answer extraction. The single stage model combines a span extraction loss with a question type (yes/no/span) classification head over the first token and an evidence extraction loss predicted from special tokens at the end of sentences and paragraphs. We also include global attention to sentence and paragraph tokens. Note that both approaches are significantly simpler than recent SOTA models that include pipeline approaches of multiple stages and customized architectures (e.g., (Tu et al., 2019; Chen et al., 2019; Tu et al., 2020; Groeneveld et al., 2020) ). See Appendix B for further details about the models and hyperparameters.
Coreference Resolution We use OntoNotes (Pradhan et al., 2012) , and the model from Joshi et al. (2019) , a modification of the system from Lee et al. (2018) to replace ELMo with BERT. The Longformer system is a straightforward adaption of the baseline model by replacing RoBERTa with Longformer and extending the sequence length. We didn't use global attention for this task.
Document Classification
We evaluate on IMDB (Maas et al., 2011) and Hyperpartisan news detection (Kiesel et al., 2019) datasets. 11 IMDB is a standard sentiment classification datasets consisting of movie reviews. While most documents in this dataset are short, about 13.6% of them are 11 For Hyperpartisan we split the training data into train/dev/test sets using standard 90/10/10 splits and performed each experiment five times with different seeds to control variability associated with the small dataset. larger than 512 wordpieces (Tab. 7). Documents in Hyperpartisan are relatively long, and it is small with only 645 documents making it a good test for Longformer's ability to adapt to limited data. We use global attention on the [CLS] token.
5.2 Results
Main Result Tab. 8 summarizes the results of all our finetuning experiments. We observe that Longformer consistently outperforms the RoBERTa baseline. Its performance gain is especially obvious for tasks that require long context such as WikiHop and Hyperpartisan. For TriviaQA, the improvement is more modest as the local context is often sufficient to answer the question. In the case of HotpotQA, the supporting fact auxiliary supervision allows models to easily find relevant contexts and then focus on local context, leading to smaller gains. This is contrasted with WikiHop that only includes distant supervision of intermediate reasoning chains, where our approach excels by reasoning over the entire context. On the IMDB and OntoNotes datasets the performance gains are smaller. For IMDB, the majority of the dataset consists of short documents and thus it is expected to see smaller improvements. For OntoNotes, we found that the distance between any two mentions is typically quite small so that a baseline that processes smaller chunks separately is able to stitch together mentions into coreference chains without considering cross chunk interactions.
Longformer-large for QA We also evaluate the performance of Longformer-large on long context QA tasks. Tab. 9 shows that our Longformer-large achieves new state-of-the-art results on WikiHop and TriviaQA by large margins (3.6 and 4 points respectively). Tab. 10 summarizes results of Hot-potQA, and, as expected, using Longformer-large improves the result compared to Longformer-base. The two-stage model improves the results even further, likely because of the increased capacity that allows each stage to specialize on one task. (Glaß et al., 2019) (not GNN) 79.4 86.2 -HGN (Fang et al., 2019) 81.0 87.9 73.0 C2F Reader (Shao et al., 2020) --73.9
Table 10 : HotpotQA results, distractor setting on the dev set. All numbers are F1 scores. 13 This model matches performance of TAP2 (Glaß et al., 2019) , which is the the best performing single model that doesn't use a form of graph neural networks (GNN; Kipf and Welling, 2017). However, our model is simpler than TAP2 where they use a three-stage approach consisting of extracting paragraphs, evidence sentences and finally answer spans with an additional specialized span selection pretraining. All the better performing models for HotpotQA (Shao et al., 2020; Fang et al., 2019) use multi-stage approaches plus GNNs or graph network of entities, which seem to encode an important inductive bias for the task. 12 Furthermore, the 10 paragraphs come from 10 different documents and the concatenated context is therefore not coherent. In this case we suspect that Longformer has less chance to use information from pretraining due to difference between the pretraining and finetuning input structure. Additional pretraining tasks could help further improve results.
5.3 Ablations on WikiHop
Tab. 11 presents an ablation study for WikiHop. All results use Longformer-base, trained for 5 epochs with identical hyperparameters except where noted. Longformer benefits from longer sequences, global attention, separate projection matrices for global attention, MLM pretraining, and longer training.
In addition, when configured as in RoBERTa-base (seqlen: 512, and n 2 attention) Longformer performs slightly worse then RoBERTa-base, confirm- 12 We can encode this inductive bias using global attention over entities, but we leave this for future work. 13 We report numbers from dev set scores of published papers. Missing values are not publicly available. Leaderboard test results will be included in the future.
Model
Accuracy ing that performance gains are not due to additional pretraining.
6 Conclusion and Future Work
We present Longformer, a transformer-based model that is scalable for processing long documents and that makes it easy to perform a wide range of document-level NLP tasks without chunking/shortening the long input and without complex architecture to combine information across these chunks. Longformer employs an attention pattern that combines local and global information while also scaling linearly with the sequence length.
Longformer achieves state-of-the-art results on the character-level language modeling tasks of text8 and enwik8. When pretrained, Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on Wiki-Hop and TriviaQA. For future work, we would like to explore other attention patterns that are more efficient by dynamically adapting to the input. We also would like to apply our model to other relevant long document tasks such as summarization.
A Character LM Hyperparameters
We evaluate on text8 and enwik8, both contain 100M characters from Wikipedia split into 90M, 5M, 5M for train, dev, test. Our implementation is based on the Transformer-XL (Dai et al., 2019 ) code14 with the memory mechanism disabled. Our hyperparameters and stage configurations are listed in Tab. 12. Our CUDA kernel supports the autoregressive mode where each token attends to a window of previous tokens only. Our implementation also includes a version of the relative position embedding that is compatible with our dilated sliding window attention.
We ran the small model experiments on 4 RTX8000 GPUs for 16 days. For the large model, we ran experiments on 8 RTX8000 GPUs for 13 days. Most of our hyperparameter search is similar to the ablation in Tab. 4 where we run the configuration for 150K steps on text8. We experimented with absolute position embeddings and learned position embeddings, dropout values of [0.1, 0.2] (small model) and [0.1, 0.4] (large model), prelayernorm and post-layernorm (Xiong et al., 2020) , learning rate (LR) of phase1 of values [2.5e-5, 5e-4, 1e-4] constant and cosine LR schedules, and different configurations for dilation (on all heads, on 2 heads, no dilation). Number of gradient updates/phase reported in Tab. 12 is determined by running each phase until the validation BPC stops getting better.
B Task specific model details
All the QA and classification models are implemented using PyTorch-Lightning15 .
WikiHop Instances in WikiHop consist of: a question, answer candidates (ranging from two candidates to 79 candidates), supporting contexts (ranging from three paragraphs to 63 paragraphs), and the correct answer. The dataset does not provide any intermediate annotation for the multihop reasoning chains, requiring models to instead infer them from the indirect answer supervision.
To prepare the data for input to Longformer and RoBERTa, we first tokenize the question, answer candidates, and support contexts using RoBERTa's wordpiece tokenizer.
Then we concatenate the question and answer candidates with special tokens as
[q] question [/q] [ent] candidate1 [/ent] ... [ent] candidateN [/ent].
The contexts are also concatenated using RoBERTa's document delimiter tokens as separators:  context1  ...  contextM . The special tokens [q], [/q], [ent], [/ent] were added to the RoBERTa vocabulary and randomly initialized before task finetuning.
After preparing the input data, we compute activations from the top layer of each model as follows. We take the question and answer candidates and concatenate them to as much context as possible up to the model sequence length (512 for RoBERTa, 4,096 for Longformer), run the sequence through the model, collect the output activations, and repeat until all of the context is exhausted (for all models except Longformer-large, where we just include the first 4,096 length sequence due to memory requirements). Then all activations for all chunks are concatenated into one long sequence. In the case of Longformer, we use global attention to the entire question and answer candidate sequence.
For prediction, we attach a linear layer to each [ent] that outputs a single logit, average over all logits for each candidate across the chunks, apply a softmax and use the cross entropy loss with the correct answer candidate.
Training used the Adam optimizer with linear warmup over 200 gradient updates to a maximum LR, and linear decay over the remainder of training. We used gradient accumulation to effective batch size of 32 instances, checking the development accuracy every 250 gradient updates and reported the maximum development accuracy. Other hyperparameters (dropout, weight decay) were identical to RoBERTa pretraining.
In general, we ran minimal hyperparameter trials, but for fair comparison between Longformer and RoBERTa ran an identical hyperparameter search with Longformer-base and RoBERTa-base. This consisted of a grid search of LR in [2e-5, 3e-5, 5e-5] and number epochs in [5, 10, 15] . The best Longformer-base configuration used lr=3e-5, 15 epochs. We ran two hyperparameter trials for Longformer-large, lr=3e-5 and number epochs in [5, 15] (the 5 epoch model had higher dev accuracy of 77.6, and was the single model submitted to the public leaderboard for test set evaluation). All mod- els were trained on a single RTX8000 GPU, with Longformer-base taking about a day for 5 epochs.
TriviaQA TriviaQA has more than 100K question, answer, document triplets for training. Documents are Wikipedia articles, and answers are named entities mentioned in the article. The span that answers the question is not annotated, but it is found using simple text matching. Similar to WikiHop, we tokenize the question and the document using RoBERTa's tokenizer, then form the input as [s] question [/s] document [/s]. We truncate the document at 4,096 wordpiece to avoid it being very slow. Afterwards, we get the activations from RoBERTa and Longformer similar to WikiHop (discussed above). We use global attention on all question tokens.
For prediction, we add one layer that predicts the beginning and end of the answer span. Because of the distant supervision nature of the training data (no gold answer spans), we use the loss function of Clark and Gardner (2017) which works like an OR that the model only needs to get one answer span right, not all of them.
Hyperparameters of the best configuration are listed in Tab. 13. All other hyperparameters are similar to RoBERTa's. For hyperparameter search, we only tuned LR for the RoBERTa baseline and tried rates [3e-5, 5e-5, 1e-4], then used the best, which is 3e-5, for all subsequent experiments with no further tuning. We trained the Longformer-large with the best configuration once and submitted its output to the leaderboard. We ran our experiments on 32GB V100 GPUs. Small model takes 1 day to train on 4 GPUs, while large model takes 1 day on 8 GPUs.
HotpotQA
HotpotQA dataset involves answering questions from a set of 10 paragraphs from 10 different Wikipedia articles where 2 paragraphs are relevant to the question and the rest are distractors. It includes 2 tasks of answer span extraction and evidence sentence identification. Our model for HotpotQA combines both answer span extraction and evidence extraction in one joint model. We also experimented with a two-stage model with similar setup that uses longformer first for evidence extraction and then extract answers using a BERT-based QA model. The second stage of our two-stage model is based on Groeneveld et al. (2020) . Similar to Wikihop and TriviaQA, to prepare the data for input to Longformer and RoBERTa, we concatenate question and then all the 10 paragraphs in one long context. We particularly use the following input format with special tokens: "
[CLS] [q] question [/q] [p] sent 1,1 [s] sent 1,2 [s] ... [p] sent 2,1 [s] sent 2,2 [s] ..."
where [s] and [p] are special tokens representing sentences and paragraphs. The special tokens were added to the RoBERTa vocabulary and randomly initialized before task finetuning. For Longformer, we use global attention to input tokens as well as sentence and paragraph tokens. For answer span extraction we use BERT's QA model (Devlin et al., 2019) with addition of a question type (yes/no/span) classification head over the first special token ([CLS]). For evidence extraction we apply 2 layer feedforward networks on top of the representations corresponding to sentence and paragraph tokens to get the corresponding evidence prediction scores and use binary cross entropy loss to train the model. We combine span, question classification, sentence, and paragraphs losses and train the model in a multitask way using linear combination of losses. Our experiments are done on RTX8000 GPUs and training each epoch takes approximately half a day on 4 GPUs. We trained the model using Adam optimizer with linear warmup (1000 steps) and linear decay. We used minimal hyperpareter tuning using LRs of 3e-5 and 5e-5 and epochs of 3 to 7 and found the model with LR of 3e-5 and 6 epochs to work best. We conduct the same hyperparameter search for the RoBERTa baseline as well. The rest of hyperparameters are reported in Tab 13. Coreference model details The coreference model is a straightforward adaptation of the coarseto-fine BERT based model from Joshi et al. (2019) . After preprocessing each document with the RoBERTa wordpiece tokenizer, it splits each document into non-overlapping segments up to the maximum sequence length, then concatenates the activations for the coarse-to-fine clustering stage that forms coreference clusters. The maximum sequence length was 384 for RoBERTa-base, chosen after three trials from [256, 384, 512] using the default hyperparameters in the original implementation. 16 For Longformer-base the sequence length was 4,096. Similar to the original implementation, 16 https://github.com/mandarjoshi90/coref different learning rates were used for the pretrained RoBERTa parameters and the randomly initialized task parameters. Using a larger learning rate in the task parameters allows the optimizer to adjust them farther from their randomly initialized values without destroying the information in the pretrained RoBERTa parameters. Hyperparameter searches were minimal and consisted of grid searches of for both RoBERTa and Longformer for a fair comparison. The best configuration for Longformer-base was RoBERTa lr=1e-5, task lr=1e-4. All other hyperparameters were the same as in the original implementation. Training takes about 10 hours on a single GPU.
Our implementation is a superhack that involves PyTorch and Tensorflow sharing a single process and GPU. To avoid re-implementing the complicated coarse-to-fine logic from Tensorflow in PyTorch (that involves a highly optimized custom GPU kernel originally released by Lee et al. (2018) ), we devised a system where the lower transformer portion of the model passes activations and gradients back and forth between PyTorch and Tensorflow. The input tensors are first run through the transformer in PyTorch, the activations are collected from the top layer, transferred from GPU to CPU then from CPU to Tensorflow and back to GPU to run the coarse-to-fine clustering and compute the loss. Then gradients are back propogated in Tensorflow to the top of the transformer and the process reversed to transfer them to PyTorch for back propogation through the remainder of the model. Separate optimizers are maintained with identical LR schedules for parameter updates. The overhead in this approach is minimal compared to the overall cost of running the model.
Text classification For classification, following BERT, we used a simple binary cross entropy loss on top of a first [CLS] token with addition of global attention to [CLS] . We used Adam optimizer with batch sizes of 32 and linear warmup and decay with warmup steps equal to 0.1 of the total training steps. For both IMDB and Hyperpartisan news we did grid search of LRs [3e-5, 5e-5] and epochs [10, 15, 20] and found the model with [3e-5] and epochs 15 to work best. Experiments were done on a single RTX8000 GPU.
SQuAD contexts typically fit within the 512 limit, and MRQA is constructed by dropping long-document examples.
It is worth noting that theoretically, a perfectly optimized sliding window attention operation should be faster than the n 2 computation. However, achieving this level of performance requires special knowledge of low-level GPU programming, similar to implementing a highly optimized matrix multiplication. Our current implementation is sufficiently fast and practical to use.
https://github.com/NVIDIA/apex
We found that using fp16 in attention operation results in floating point overflow and NaNs in later stages of training.
An obvious caveat is that there is a chance the end performance will not agree with the performance at step 150K. However, this is a reasonable approximation that saves the huge cost of running all these configurations to completion.
Sequences up to 16K are possible on current GPUs.
We tried adding the additional dilation pattern on a few heads as in §4.1 but found it to hurt performance, likely because it is not compatible with the pretrained RoBERTa weights. We suspect that retraining such model from scratch might be needed to get improved performance
We use the full version of TriviaQA and HotpotQA, not the simplified versions in MRQA(Fisch et al., 2019).
https://github.com/kimiyoung/ transformer-xl
https://github.com/PyTorchLightning/ pytorch-lightning
References
[1]Al-Rfou Rami, Choe Dokook, Constant Noah, Guo Mandy, Jones Llion. Character-level language modeling with deeper self-attention. AAAI 2018;:.
[2]Chen Danqi, Fisch Adam, Weston Jason, Bordes Antoine. Reading wikipedia to answer opendomain questions. ACL 2017;:.
[3]Chen Jifan, Lin Shih-Ting, Durrett Greg. Multi-hop question answering via reasoning chains.  2019;:.
[4]Chen Tianqi, Moreau Thierry, Jiang Ziheng, Zheng Lianmin, Yan Eddie, Shen Haichen, et al.. TVM: An automated end-to-end optimizing compiler for deep learning. 13th USENIX Symposium on Operating Systems Design and Implementation 2018;18:578--594.
[5]Chen Tianqi, Xu Bing, Zhang Chiyuan, Guestrin Carlos. Training deep nets with sublinear memory cost.  2016;:.
[6]Child Rewon, Gray Scott, Radford Alec, Sutskever Ilya. Generating long sequences with sparse transformers.  2019;:.
[7]Clark Christopher, Gardner Matt. Simple and effective multi-paragraph reading comprehension. ACL 2017;:.
[8]Clark Kevin, Khandelwal Urvashi, Levy Omer, Manning ChristopherD. What does bert look at? an analysis of bert's attention.  2019;:.
[9]Andrew M, Dai Quoc V, Le . Semi-supervised sequence learning. Advances in Neural Information Processing Systems 2015;28:.
[10]Dai Zihang, Yang Zhilin, Yang Yiming, Carbonell JaimeG, Le QuocV, Salakhutdinov Ruslan. Transformer-xl: Attentive language models beyond a fixed-length context. ACL 2019;:.
[11]Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019;:.
[12]Fang Yuwei, Sun Siqi, Gan Zhe, Pillai Rohit, Wang Shuohang, Liu Jing Jing. Hierarchical graph network for multi-hop question answering.  2019;:.
[13]Fisch Adam, Talmor Alon, Jia Robin, Seo Minjoon, Choi Eunsol, Chen Danqi. MRQA 2019 shared task: Evaluating generalization in reading comprehension. Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP 2019;:.
[14]Glaß Michael, Massimiliano Gliozzo Alfio, Chakravarti Rishav, Ferritto Anthony, Pan Lin, Bhargav Gaudani, et al.. Span selection pre-training for question answering.  2019;:.
[15]Gray Scott, Radford Alec, Kingma DiederikP. Gpu kernels for block-sparse weights.  2017;:.
[16]Groeneveld Dirk, Khot Tushar, Mausam Ashish, Sabhwaral . A simple yet strong pipeline for HotpotQA.  2020;:.
[17]Howard Jeremy, Ruder Sebastian. Universal language model fine-tuning for text classification. ACL 2018;:.
[18]Joshi Mandar, Choi Eunsol, Weld DanielS, Zettlemoyer Luke. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. ACL 2017;:.
[19]Joshi Mandar, Levy Omer, Zettlemoyer Luke, Weld Daniel. BERT for coreference resolution: Baselines and analysis. EMNLP-IJCNLP 2019;:.
[20]Kiesel Johannes, Mestre Maria, Shukla Rishabh, Vincent Emmanuel, Adineh Payam, Corney David, et al.. SemEval-2019 task 4: Hyperpartisan news detection. Proceedings of the 13th International Workshop on Semantic Evaluation 2019;:829--839.
[21]Thomas N, Kipf Max, Welling . Semisupervised classification with graph convolutional networks.  2017;:.
[22]Kitaev Nikita, Kaiser Lukasz, Levskaya Anselm. Reformer: The efficient transformer.  2020;:.
[23]Kovaleva OlgaV, Romanov Alexey, Rogers Anna, Rumshisky Anna. Revealing the dark secrets of bert. EMNLP/IJCNLP 2019;:.
[24]Lee Kenton, He Luheng, Zettlemoyer Luke. Higher-order coreference resolution with coarse-tofine inference. NAACL 2018;:.
[25]Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, et al.. Roberta: A robustly optimized bert pretraining approach.  2019;:.
[26]Maas AndrewL, Daly RaymondE, Pham PeterT, Huang Dan, Ng AndrewY, Potts Christopher. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies 2011;:142--150.
[27]Mahoney Matt. Large text compression benchmark.  2009;:.
[28]Van Den Oord Aäron, Dieleman Sander, Zen Heiga, Simonyan Karen, Vinyals Oriol, Graves Alex, et al.. Wavenet: A generative model for raw audio. SSW 2016;:.
[29]Ott Myle, Edunov Sergey, Baevski Alexei, Fan Angela, Gross Sam, Ng Nathan, et al.. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019;:.
[30]Peters MatthewE, Neumann Mark, Iyyer Mohit, Gardner Matt, Clark Christopher, Lee Kenton, et al.. Deep contextualized word representations.  2018;:.
[31]Sameer Pradhan Alessandro, Moschitti Nianwen, Xue Olga, Uryupina Yuchen, Zhang . CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. Joint Conference on EMNLP and CoNLL -Shared Task 2012;:1--40.
[32]Qiu Jiezhong, Ma Hao, Levy Omer, Yih Scott, Wang Sinong, Tang Jie. Blockwise selfattention for long document understanding.  2019;:.
[33]Radford Alec, Wu Jeffrey, Child Rewon, Luan David, Amodei Dario, Sutskever Ilya. Language models are unsupervised multitask learners.  2019;:.
[34]Rae JackW, Potapenko Anna, Siddhant M, Jayakumar TimothyP, Lillicrap . Compressive transformers for long-range sequence modelling.  2020;:.
[35]Shao Nan, Cui Yiming, Liu Ting, Wang Shijin, Hu Guoping. Is graph structure necessary for multi-hop reasoning?.  2020;:.
[36]Sukhbaatar Sainbayar, Grave Edouard, Bojanowski Piotr, Joulin Armand. Adaptive attention span in transformers. ACL 2019;:.
[37]Trieu H, Trinh QuocV, Le . A simple method for commonsense reasoning.  2018;:.
[38]Tu Ming, Huang Jinke, He Xiaodong, Zhou Bowen. Graph sequential network for reasoning over sequences.  2020;:.
[39]Tu Ming, Huang Kevin, Wang Guangtao, Huang Jing, He Xiaodong, Zhou Bufang. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents.  2019;:.
[40]Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez AidanN, et al.. Attention is all you need. NIPS 2017;:.
[41]Welbl Johannes, Stenetorp Pontus, Riedel Sebastian. Constructing datasets for multi-hop reading comprehension across documents.  2018;6:287--302.
[42]Wu Felix, Fan Angela, Baevski Alexei, Dauphin Yann, Auli Michael, Xie ;, et al.. Pay less attention with lightweight and dynamic convolutions.  2019;:.
[43]Xiong Ruibin, Yang Yunchang, He Di, Zheng Kai, Shu Xin Zheng Chen, Xing Huishuai, et al.. On layer normalization in the transformer architecture.  2020;:.
[44]Yang Zhilin, Qi Peng, Zhang Saizheng, Bengio Yoshua, Cohen WilliamW, Salakhutdinov Ruslan, et al.. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. EMNLP 2018;:.
[45]Ye Zihao, Guo Qipeng, Gan Quan, Qiu Xipeng, Zhang Zheng. Bp-transformer: Modelling long-range context via binary partitioning.  2019;:.
[46]Zellers Rowan, Holtzman Ari, Rashkin Hannah, Bisk Yonatan, Farhadi Ali, Roesner Franziska, et al.. Defending against neural fake news. NeurIPS 2019;:.
[47]Zhu Yukun, Kiros Ryan, Zemel RichardS, Salakhutdinov Ruslan, Urtasun Raquel, Torralba Antonio, et al.. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. IEEE International Conference on Computer Vision (ICCV) 2015;:19--27.
Longformer: The Long-Document Transformer
https://arxiv.org/abs/2004.05150
https://arxiv.org/abs/2009.14794
Rethinking Attention with Performers
2020-09-30
Choromanski; Krzysztof; Likhosherstov; Valerii; Dohan; David; Song; Xingyou; Gane; Andreea; Sarlos; Tamas; Hawkins; Peter; Davis; Jared; Mohiuddin; Afroz; Kaiser; Lukasz; Belanger; Colwell; Lucy; Weller; Adrian
<div><div>
 
 
 <div><p>Authors:<a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Choromanski%2C+K">Krzysztof Choromanski</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Likhosherstov%2C+V">Valerii Likhosherstov</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Dohan%2C+D">David Dohan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Song%2C+X">Xingyou Song</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gane%2C+A">Andreea Gane</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sarlos%2C+T">Tamas Sarlos</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hawkins%2C+P">Peter Hawkins</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Davis%2C+J">Jared Davis</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mohiuddin%2C+A">Afroz Mohiuddin</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kaiser%2C+L">Lukasz Kaiser</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Belanger%2C+D">David Belanger</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Colwell%2C+L">Lucy Colwell</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Weller%2C+A">Adrian Weller</a></p></div>
 
 
 
 <p><a href="/pdf/2009.14794">Download PDF</a></p><blockquote>
 Abstract: We introduce Performers, Transformer architectures which can estimate regular
(softmax) full-rank-attention Transformers with provable accuracy, but using
only linear (as opposed to quadratic) space and time complexity, without
relying on any priors such as sparsity or low-rankness. To approximate softmax
attention-kernels, Performers use a novel Fast Attention Via positive
Orthogonal Random features approach (FAVOR+), which may be of independent
interest for scalable kernel methods. FAVOR+ can be also used to efficiently
model kernelizable attention mechanisms beyond softmax. This representational
power is crucial to accurately compare softmax with other kernels for the first
time on large-scale tasks, beyond the reach of regular Transformers, and
investigate optimal attention-kernels. Performers are linear architectures
fully compatible with regular Transformers and with strong theoretical
guarantees: unbiased or nearly-unbiased estimation of the attention matrix,
uniform convergence and low estimation variance. We tested Performers on a rich
set of tasks stretching from pixel-prediction through text models to protein
sequence modeling. We demonstrate competitive results with other examined
efficient sparse and dense attention methods, showcasing effectiveness of the
novel attention-learning paradigm leveraged by Performers.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Valerii Likhosherstov [<a href="/show-email/e755a50c/2009.14794">view email</a>]
 <br />
 <strong><a href="/abs/2009.14794v1">[v1]</a></strong>
 Wed, 30 Sep 2020 17:09:09 UTC (10,282 KB)<br />
 <strong><a href="/abs/2009.14794v2">[v2]</a></strong>
 Tue, 16 Feb 2021 21:40:24 UTC (13,996 KB)<br />
 <strong><a href="/abs/2009.14794v3">[v3]</a></strong>
 Tue, 9 Mar 2021 16:26:47 UTC (13,996 KB)<br /><strong>[v4]</strong>
Sat, 19 Nov 2022 12:45:21 UTC (27,987 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2009.14794

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:2009.14794 (cs)
 [Submitted on 30 Sep 2020 (v1), last revised 19 Nov 2022 (this version, v4)]

 Title: Rethinking Attention with Performers

 Authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller
 Download PDF
 Abstract: We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.
 Comments: Published as a conference paper + oral presentation at ICLR 2021. 38 pages. See this https URL for protein language model code, and this https URL for Performer code. See this https URL for Google AI Blog
 Subjects: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Machine Learning (stat.ML)  
 Cite as: arXiv:2009.14794 [cs.LG]   
 (or arXiv:2009.14794v4 [cs.LG] for this version)   
 https://doi.org/10.48550/arXiv.2009.14794   
 Focus to learn more   
 arXiv-issued DOI via DataCite   
 

 Submission history

 From: Valerii Likhosherstov [view email]
 [v1] Wed, 30 Sep 2020 17:09:09 UTC (10,282 KB)
 [v2] Tue, 16 Feb 2021 21:40:24 UTC (13,996 KB)
 [v3] Tue, 9 Mar 2021 16:26:47 UTC (13,996 KB)
 [v4] Sat, 19 Nov 2022 12:45:21 UTC (27,987 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 2009
 Change to browse by:
 cs
 cs.CL
 stat
 stat.ML

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 5 blog links

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Krzysztof Choromanski
 Valerii Likhosherstov
 David Dohan
 Xingyou Song
 Andreea Gane
 …
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

RETHINKING ATTENTION WITH PERFORMERS
Date Published: 2022-11-19
Authors:
Krzysztof  Choromanski
Valerii  Likhosherstov, University of Cambridge
David  Dohan
Xingyou  Song
Andreea  Gane
Tamas  Sarlos
Peter  Hawkins
Jared  Davis
Afroz  Mohiuddin
Lukasz  Kaiser
David  Belanger
Lucy  Colwell, University of Cambridge, lcolwell@google.com
Adrian  Weller, University of Cambridge
  Google
Abstract
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attentionkernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can also be used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.
1 INTRODUCTION AND RELATED WORK
Transformers (Vaswani et al., 2017; Dehghani et al., 2019) are powerful neural network architectures that have become SOTA in several areas of machine learning including natural language processing (NLP) (e.g. speech recognition (Luo et al., 2020) ), neural machine translation (NMT) (Chen et al., 2018) , document generation/summarization, time series prediction, generative modeling (e.g. image generation (Parmar et al., 2018) ), music generation (Huang et al., 2019) , and bioinformatics (Rives et al., 2019; Madani et al., 2020; Ingraham et al., 2019; Elnaggar et al., 2019; Du et al., 2020) .
Transformers rely on a trainable attention mechanism that identifies complex dependencies between the elements of each input sequence. Unfortunately, the regular Transformer scales quadratically with the number of tokens L in the input sequence, which is prohibitively expensive for large L and precludes its usage in settings with limited computational resources even for moderate values of L. Several solutions have been proposed to address this issue (Beltagy et al., 2020; Gulati et al., 2020; Chan et al., 2020; Child et al., 2019; Bello et al., 2019) . Most approaches restrict the attention mechanism to attend to local neighborhoods (Parmar et al., 2018) or incorporate structural priors on attention such as sparsity (Child et al., 2019) , pooling-based compression (Rae et al., 2020) clustering/binning/convolution techniques (e.g. (Roy et al., 2020) which applies k-means clustering to learn dynamic sparse attention regions, or (Kitaev et al., 2020) , where locality sensitive hashing is used to group together tokens of similar embeddings), sliding windows (Beltagy et al., 2020) , or truncated targeting (Chelba et al., 2020) . There is also a long line of research on using dense attention matrices, but defined by low-rank kernels substituting softmax (Katharopoulos et al., 2020; Shen et al., 2018) . Those methods critically rely on kernels admitting explicit representations as dot-products of finite positive-feature vectors.
The approaches above do not aim to approximate regular attention, but rather propose simpler and more tractable attention mechanisms, often by incorporating additional constraints (e.g. identical query and key sets as in (Kitaev et al., 2020) ), or by trading regular with sparse attention using more layers (Child et al., 2019) . Unfortunately, there is a lack of rigorous guarantees for the representation power produced by such methods, and sometimes the validity of sparsity patterns can only be verified empirically through trial and error by constructing special GPU operations (e.g. either writing C++ CUDA kernels (Child et al., 2019) or using TVMs (Beltagy et al., 2020) ). Other techniques which aim to reduce Transformers' space complexity include reversible residual layers allowing one-time activation storage in training (Kitaev et al., 2020) and shared attention weights (Xiao et al., 2019) . These constraints may impede application to long-sequence problems, where approximations of the attention mechanism are not sufficient. Approximations based on truncated back-propagation (Dai et al., 2019) are also unable to capture long-distance correlations since the gradients are only propagated inside a localized window. Other methods propose biased estimation of regular attention but only in the non-causal setting and with large mean squared error (Wang et al., 2020) .
In response, we introduce the first Transformer architectures, Performers, capable of provably accurate and practical estimation of regular (softmax) full-rank attention, but of only linear space and time complexity and not relying on any priors such as sparsity or low-rankness. Performers use the Fast Attention Via positive Orthogonal Random features (FAVOR+) mechanism, leveraging new methods for approximating softmax and Gaussian kernels, which we propose. We believe these methods are of independent interest, contributing to the theory of scalable kernel methods. Consequently, Performers are the first linear architectures fully compatible (via small amounts of fine-tuning) with regular Transformers, providing strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and lower variance of the approximation.
FAVOR+ can be also applied to efficiently model other kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, that are beyond the reach of regular Transformers, and find for them optimal attention-kernels. FAVOR+ can also be applied beyond the Transformer scope as a more scalable replacement for regular attention, which itself has a wide variety of uses in computer vision (Fu et al., 2019) , reinforcement learning (Zambaldi et al., 2019) , training with softmax cross entropy loss, and even combinatorial optimization (Vinyals et al., 2015) .
We test Performers on a rich set of tasks ranging from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing the effectiveness of the novel attention-learning paradigm leveraged by Performers. We emphasize that in principle, FAVOR+ can also be combined with other techniques, such as reversible layers (Kitaev et al., 2020) or cluster-based attention (Roy et al., 2020) .
2 FAVOR+ MECHANISM & POSITIVE ORTHOGONAL RANDOM FEATURES
Below we describe in detail the FAVOR+ mechanism -the backbone of the Performer s architecture. We introduce a new method for estimating softmax (and Gaussian) kernels with positive orthogonal random features which FAVOR+ leverages for the robust and unbiased estimation of regular (softmax) attention and show how FAVOR+ can be applied for other attention-kernels.
2.1 PRELIMINARIES -REGULAR ATTENTION MECHANISM
Let L be the size of an input sequence of tokens. Then regular dot-product attention (Vaswani et al., 2017) is a mapping which accepts matrices Q, K, V ∈ R L×d as input where d is the hidden dimension (dimension of the latent representation). Matrices Q, K, V are intermediate representations of the input and their rows can be interpreted as queries, keys and values of the continuous dictionary data structure respectively. Bidirectional (or non-directional (Devlin et al., 2018) ) dot-product attention has the following form, where A ∈ R L×L is the so-called attention matrix:
Att ↔ (Q, K, V) = D -1 AV, A = exp(QK / √ d), D = diag(A1 L ).
(1) Here exp(•) is applied elementwise, 1 L is the all-ones vector of length L, and diag(•) is a diagonal matrix with the input vector as the diagonal. Time and space complexity of computing (1) are O(L 2 d) and O(L 2 + Ld) respectively, because A has to be stored explicitly. Hence, in principle, dot-product attention of type ( 1) is incompatible with end-to-end processing of long sequences. Bidirectional attention is applied in encoder self-attention and encoder-decoder attention in Seq2Seq architectures.
Another important type of attention is unidirectional dot-product attention which has the form:
Att → (Q, K, V) = D -1 AV, A = tril(A), D = diag( A1 L ),where tril(•) returns the lower-triangular part of the argument matrix including the diagonal. As discussed in (Vaswani et al., 2017) , unidirectional attention is used for autoregressive generative modelling, e.g. as self-attention in generative Transformers as well as the decoder part of Seq2Seq Transformers.
We will show that attention matrix A can be approximated up to any precision in time O(Ld 2 log(d)).
For comparison, popular methods leveraging sparsity via Locality-Sensitive Hashing (LSH) techniques (Kitaev et al., 2020) have O(Ld 2 log L) time complexity. In the main body of the paper we will describe FAVOR+ for bidirectional attention. Completely analogous results can be obtained for the unidirectional variant via the mechanism of prefix-sums (all details in the Appendix B.1).
2.2 GENERALIZED KERNELIZABLE ATTENTION
FAVOR+ works for attention blocks using matrices A ∈ R L×L of the form A(i, j) = K(q i , k j ), with q i /k j standing for the i th /j th query/key row-vector in Q/K and kernel K : R d × R d → R + defined for the (usually randomized) mapping: φ : R d → R r + (for some r > 0) as:
K(x, y) = E[φ(x) φ(y)].
(3)
We call φ(u) a random feature map for u ∈ R d . For Q , K ∈ R L×r with rows given as φ(q i ) and φ(k i ) respectively, Equation 3 leads directly to the efficient attention mechanism of the form: The above scheme constitutes the FA-part of the FAVOR+ mechanism. The remaining OR+ part answers the following questions: (1) How expressive is the attention model defined in Equation 3, and in particular, can we use it in principle to approximate regular softmax attention ? (2) How do we implement it robustly in practice, and in particular, can we choose r L for L d to obtain desired space and time complexity gains? We answer these questions in the next sections.
Att ↔ (Q, K, V) = D -1 (Q ((K ) V)), D = diag(Q ((K ) 1 L )).2.3 HOW TO AND HOW NOT TO APPROXIMATE SOFTMAX-KERNELS FOR ATTENTION
It turns out that by taking φ of the following form for functions f 1 , ..., f l : R → R, function g : R d → R and deterministic vectors ω i or ω 1 , ..., ω m iid ∼ D for some distribution D ∈ P(R d ):
φ(x) = h(x) √ m (f 1 (ω 1 x), ..., f 1 (ω m x), ..., f l (ω 1 x), ..., f l (ω m x)),we can model most kernels used in practice. Furthermore, in most cases D is isotropic (i.e. with pdf function constant on a sphere), usually Gaussian. For example, by taking h(x) = 1, l = 1 and D = N (0, I d ) we obtain estimators of the so-called PNG-kernels (Choromanski et al., 2017 ) (e.g. f 1 = sgn corresponds to the angular kernel). Configurations: h(x) = 1, l = 2, f 1 = sin, f 2 = cos correspond to shift-invariant kernels, in particular D = N (0, I d ) leads to the Gaussian kernel K gauss (Rahimi & Recht, 2007) . The softmax-kernel which defines regular attention matrix A is given as:
SM(x, y) def = exp(x y).In the above, without loss of generality, we omit √ d-renormalization since we can equivalently renormalize input keys and queries. Since: SM(x, y) = exp( x 2 2 )K gauss (x, y) exp( y 2 2 ), based on what we have said, we obtain random feature map unbiased approximation of SM(x, y) using trigonometric functions with:
h(x) = exp( x 2 2 ), l = 2, f 1 = sin, f 2 = cos. We call it SM trig m (x, y).
There is however a caveat there. The attention module from (1) constructs for each token, a convex combination of value-vectors with coefficients given as corresponding renormalized kernel scores. That is why kernels producing non-negative scores are used. Applying random feature maps with potentially negative dimension-values (sin / cos) leads to unstable behaviours, especially when kernel scores close to 0 (which is the case for many entries of A corresponding to low relevance tokens) are approximated by estimators with large variance in such regions. This results in abnormal behaviours, e.g. negative-diagonal-values renormalizers D -1 , and consequently either completely prevents training or leads to sub-optimal models. We demonstrate empirically that this is what happens for SM trig m and provide detailed theoretical explanations showing that the variance of SM trig m is large as approximated values tend to 0 (see: Section 3). This is one of the main reasons why the robust random feature map mechanism for approximating regular softmax attention was never proposed.
We propose a robust mechanism in this paper. Furthermore, the variance of our new unbiased positive random feature map estimator tends to 0 as approximated values tend to 0 (see: Section 3). Lemma 1 (Positive Random Features (PRFs) for Softmax). For x, y ∈ R d , z = x + y we have:
SM(x, y) = E ω∼N (0,I d ) exp ω x- x 2 2 exp ω y- y 2 2 = ΛE ω∼N (0,I d ) cosh(ω z),where Λ = exp(-
x 2 + y 2 2
) and cosh is hyperbolic cosine. Consequently, softmax-kernel admits a positive random feature map unbiased approximation with h(x) = exp(-
x 2 2 ), l = 1, f 1 = exp and D = N (0, I d ) or: h(x) = 1 √ 2 exp(-x 2 2 ), l = 2, f 1 (u) = exp(u), f 2 (u) = exp(-u
) and the same D (the latter for further variance reduction). We call related estimators: SM In Fig. 2 we visualize the advantages of positive versus standard trigonometric random features. In critical regions, where kernel values are small and need careful approximation, our method outperforms its counterpart. In Section 4 we further confirm our method's advantages empirically, using positive features to efficiently train softmax-based linear Transformers. If we replace in (7) ω with √ d ω ω , we obtain the so-called regularized softmax-kernel SMREG which we can approximate in a similar manner, simply changing D = N (0,
I d ) to D = Unif( √ dS d-1
), a distribution corresponding to Haar measure on the sphere of radius
√ d in R d , obtaining estimator SMREG + m .
As we show in Section 3, such random features can also be used to accurately approximate regular softmax-kernel.
2.4 ORTHOGONAL RANDOM FEATURES (ORFS)
The above constitutes the R+ part of the FAVOR+ method. It remains to explain the O-part. To further reduce the variance of the estimator (so that we can use an even smaller number of random features r), we entangle different random samples ω 1 , ..., ω m to be exactly orthogonal. This can be done while maintaining unbiasedness whenever isotropic distributions D are used (i.e. in particular in all kernels we considered so far) by the standard Gram-Schmidt orthogonalization procedure (see (Choromanski et al., 2017) for details). ORFs is a well-known method, yet it turns out that it works particularly well with our introduced PRFs for softmax. This leads to the first theoretical results showing that ORFs can be applied to reduce the variance of softmax/Gaussian kernel estimators for any dimensionality d rather than just asymptotically for large enough d (as is the case for previous methods, see: next section) and leads to the first exponentially small bounds on large deviations probabilities that are strictly smaller than for non-orthogonal methods. Positivity of random features plays a key role in these bounds. The ORF mechanism requires m ≤ d, but this will be the case in all our experiments. The pseudocode of the entire FAVOR+ algorithm is given in Appendix B.
Our theoretical results are tightly aligned with experiments. We show in Section 4 that PRFs+ORFs drastically improve accuracy of the approximation of the attention matrix and enable us to reduce r which results in an accurate as well as space and time efficient mechanism which we call FAVOR+.
3 THEORETICAL RESULTS
We present here the theory of positive orthogonal random features for softmax-kernel estimation. All these results can be applied also to the Gaussian kernel, since as explained in the previous section, one can be obtained from the other by renormalization (see: Section 2.3). All proofs and additional more general theoretical results with a discussion are given in the Appendix. Lemma 2 (positive (hyperbolic) versus trigonometric random features). The following is true:
MSE( SM trig m (x, y)) = 1 2m exp( x + y 2 )SM -2 (x, y)(1 -exp(-x -y 2 )) 2 , MSE( SM + m (x, y)) = 1 m exp( x + y 2 )SM 2 (x, y)(1 -exp(-x + y 2 )), MSE( SM hyp+ m (x, y)) = 1 2 (1 -exp(-x + y 2 ))MSE( SM + m (x, y)),for independent random samples ω i , and where MSE stands for the mean squared error.
Thus, for SM(x, y) → 0 we have: MSE( SM trig m (x, y)) → ∞ and MSE( SM + m (x, y)) → 0. Furthermore, the hyperbolic estimator provides additional accuracy improvements that are strictly better than those from SM + 2m (x, y) with twice as many random features. The next result shows that the regularized softmax-kernel is in practice an accurate proxy of the softmax-kernel in attention. Theorem 1 (regularized versus softmax-kernel). Assume that the L ∞ -norm of the attention matrix for the softmax-kernel satisfies: A ∞ ≤ C for some constant C ≥ 1. Denote by A reg the corresponding attention matrix for the regularized softmax-kernel. The following holds:
inf i,j A reg (i, j) A(i, j) ≥ 1 - 2 d 1 3 + o 1 d 1 3
, and sup i,j
A reg (i, j)
A(i, j) ≤ 1.Furthermore, the latter holds for d ≥ 2 even if the L ∞ -norm condition is not satisfied, i.e. the regularized softmax-kernel is a universal lower bound for the softmax-kernel.
Consequently, positive random features for SMREG can be used to approximate the softmax-kernel.
Our next result shows that orthogonality provably reduces mean squared error of the estimation with positive random features for any dimensionality d > 0 and we explicitly provide the gap. 
MSE( SM ort+ m (x, y)) ≤ MSE( SM + m (x, y))- 2(m -1) m(d + 2) SM(x, y) -exp - x 2 + y 2 2 2 .
(10) Furthermore, completely analogous result holds for the regularized softmax-kernel SMREG.
For the regularized softmax-kernel, orthogonal features provide additional concentration results -the first exponentially small bounds for probabilities of estimators' tails that are strictly better than for non-orthogonal variants for every d > 0. Our next result enables us to explicitly estimate the gap. Theorem 3. Let x, y ∈ R d . The following holds for any a > SMREG(x, y), θ > 0 and m ≤ d:
P[ SMREG + m (x, y) > a] ≤ exp(-θma)M Z (θ) m , P[ SMREG ort+ m (x, y) > a] ≤ exp(-θma) M Z (θ) m -exp - m 2 ( x 2 + y 2 ) θ 4 m(m -1) 4(d + 2) x + y 4
where SMREG ort+ m (x, y) stands for the modification of SMREG
+ m (x, y) with ORFs, X = Λ exp( √ d ω ω 2 (x + y)), ω ∼ N (0, I d ), Λ is as in Lemma 1 and M Z is the moment generating function of Z.
We see that ORFs provide exponentially small and sharper bounds for critical regions where the softmax-kernel is small. Below we show that even for the SM trig mechanism with ORFs, it suffices to take m = Θ(d log(d)) random projections to accurately approximate the attention matrix (thus if not attention renormalization, PRFs would not be needed). In general, m depends on the dimensionality d of the embeddings, radius R of the ball where all queries/keys live and precision parameter (see: Appendix F.6 for additional discussion), but does not depend on input sequence length L. Theorem 4 (uniform convergence for attention approximation). Assume that L 2 -norms of queries/keys are upper-bounded by R > 0. Define l = Rd -1 4 and take h * = exp( l 2 2 ). Then for any > 0, δ = (h * ) 2 and the number of random projections m = Θ( d δ 2 log( 4d
3 4 R δ )
) the following holds for the attention approximation mechanism leveraging estimators SM trig with ORFs: A -A ∞ ≤ with any constant probability, where A approximates the attention matrix A.
4 EXPERIMENTS
We implemented our setup on top of pre-existing Transformer training code in Jax (Frostig et al., 2018) optimized with just-in-time (jax.jit) compilation, and complement our theory with empirical evidence to demonstrate the practicality of FAVOR+ in multiple settings. Unless explicitly stated, a Performer replaces only the attention component with our method, while all other components are exactly the same as for the regular Transformer. For shorthand notation, we denote unidirectional/causal modelling as (U) and bidirectional/masked language modelling as (B).
In terms of baselines, we use other Transformer models for comparison, although some of them are restricted to only one case -e.g. Reformer (Kitaev et al., 2020) is only (U), and Linformer (Wang et al., 2020) is only (B). Furthermore, we use PG-19 (Rae et al., 2020) as an alternative (B) pretraining benchmark, as it is made for long-length sequence training compared to the (now publicly unavailable) BookCorpus (Zhu et al., 2015) + Wikipedia dataset used in BERT (Devlin et al., 2018) and Linformer. All model and tokenization hyperparameters are shown in Appendix A. 
4.1 COMPUTATIONAL COSTS
We compared speed-wise the backward pass of the Transformer and the Performer in (B) setting, as it is one of the main computational bottlenecks during training, when using the regular default size (n heads , n layers , d f f , d) = (8, 6, 2048, 512) , where d f f denotes the width of the MLP layers.
We observed (Fig. 3 ) that in terms of L, the Performer reaches nearly linear time and sub-quadratic memory consumption (since the explicit O(L 2 ) attention matrix is not stored). In fact, the Performer achieves nearly optimal speedup and memory efficiency possible, depicted by the "X"-line when attention is replaced with the "identity function" simply returning the V-matrix. The combination of both memory and backward pass efficiencies for large L allows respectively, large batch training and lower wall clock time per gradient step. Extensive additional results are demonstrated in Appendix E by varying layers, raw attention, and architecture sizes.
4.2 SOFTMAX ATTENTION APPROXIMATION ERROR
We further examined the approximation error via FAVOR+ in Fig. 4 . We demonstrate that 1. Orthogonal features produce lower error than unstructured (IID) features, 2. Positive features produce lower error than trigonometric sin/cos features. These two empirically validate the PORF mechanism. To further improve overall approximation of attention blocks across multiple iterations which further improves training, random samples should be periodically redrawn (Fig. 5 , right). This is a cheap procedure, but can be further optimized (Appendix B.2).
4.3 SOFTMAX APPROXIMATION ON TRANSFORMERS
Even if the approximation of the attention mechanism is tight, small errors can easily propagate throughout multiple Transformer layers (e.g. MLPs, multiple heads), as we show in Fig. 14 (Appendix). In other words, the model's Lipschitz constant can easily scale up small attention approximation error, which means that very tight approximations may sometimes be needed. Thus, when applying FAVOR(+)'s softmax approximations on a Transformer model (i.e. "Performer-X-SOFTMAX"), we demonstrate that:
1. Backwards compatibility with pretrained models is available as a benefit from softmax approximation, via small finetuning (required due to error propagation) even for trigonometric features (Fig. 5 , left) on the LM1B dataset (Chelba et al., 2014) . However, when on larger dataset PG-19, 2. Positive (POS) softmax features (with redrawing) become crucial for achieving performance matching regular Transformers (Fig. 5 , right). We further benchmark the Performer on both (U) and (B) cases by training a 36-layer model using protein sequences from the Jan. 2019 release of TrEMBL (Consortium, 2019), similar to (Madani et al., 2020) . In Fig. 6 , the Reformer and Linformer significantly drop in accuracy on the protein dataset. Furthermore, the usefulness of generalized attention is evidenced by Performer-RELU (taking f = ReLU in Equation 5) achieving the highest accuracy in both (U) and (B) cases. Our proposed softmax approximation is also shown to be tight, achieving the same accuracy as the exact-softmax Transformer and confirming our theoretical claims from Section 3. 
4.5 LARGE LENGTH TRAINING -COMMON DATASETS
On the standard (U) ImageNet64 benchmark from (Parmar et al., 2018) with L = 12288 which is unfeasible for regular Transformers, we set all models to use the same (n heads , d f f , d) but varying n layers . Performer/6-layers matches the Reformer/12-layers, while the Performer/12-layers matches the Reformer/24-layers (Fig. 7 : left). Depending on hardware (TPU or GPU), we also found that the Performer can be 2x faster than the Reformer via Jax optimizations for the (U) setting.
For a proof of principle study, we also create an initial protein benchmark for predicting interactions among groups of proteins by concatenating protein sequences to length L = 8192 from TrEMBL, long enough to model protein interaction networks without the large sequence alignments required by existing methods (Cong et al., 2019) . In this setting, a regular Transformer overloads memory even at a batch size of 1 per chip, by a wide margin. Thus as a baseline, we were forced to use a significantly smaller variant, reducing to (n heads , n layers , d f f , d) = (8, {1, 2, 3}, 256, 256). Meanwhile, the Performer trains efficiently at a batch size of 8 per chip using the standard (8, 6, 2048, 512) architecture. We see in Fig. 7 (right subfigure) that the smaller Transformer (n layer = 3) is quickly bounded at ≈ 19%, while the Performer is able to train continuously to ≈ 24%. 
6 BROADER IMPACT
We believe that the presented algorithm can be impactful in various ways:
Biology and Medicine: Our method has the potential to directly impact research on biological sequence analysis by enabling the Transformer to be applied to much longer sequences without constraints on the structure of the attention matrix. The initial application that we consider is the prediction of interactions between proteins on the proteome scale. Recently published approaches require large evolutionary sequence alignments, a bottleneck for applications to mammalian genomes (Cong et al., 2019) . The potentially broad translational impact of applying these approaches to biological sequences was one of the main motivations of this work. We believe that modern bioinformatics can immensely benefit from new machine learning techniques with Transformers being among the most promising. Scaling up these methods to train faster more accurate language models opens the door to the ability to design sets of molecules with pre-specified interaction properties. These approaches could be used to augment existing physics-based design strategies that are of critical importance for example in the development of new nanoparticle vaccines (Marcandalli et al., 2019) .
Environment: As we have shown, Performers with FAVOR+ are characterized by much lower compute costs and substantially lower space complexity which can be directly translated to CO 2 emission reduction (Strubell et al., 2019) and lower energy consumption (You et al., 2020) , as regular Transformers require very large computational resources.
Research on Transformers:
We believe that our results can shape research on efficient Transformers architectures, guiding the field towards methods with strong mathematical foundations. Our research may also hopefully extend Transformers also beyond their standard scope (e.g. by considering the Generalized Attention mechanism and connections with kernels). Exploring scalable Transformer architectures that can handle L of the order of magnitude few thousands and more, preserving accuracy of the baseline at the same time, is a gateway to new breakthroughs in bio-informatics, e.g. language modeling for proteins, as we explained in the paper. Our presented method can be potentially a first step.
Backward Compatibility: Our Performer can be used on the top of a regular pre-trained Transformer as opposed to other Transformer variants. Even if up-training is not required, FAVOR+ can still be used for fast inference with no loss of accuracy. We think about this backward compatibility as a very important additional feature of the presented techniques that might be particularly attractive for practitioners.
Attention Beyond Transformers: Finally, FAVOR+ can be applied to approximate exact attention also outside the scope of Transformers. This opens a large volume of new potential applications including: hierarchical attention networks (HANS) (Yang et al., 2016) , graph attention networks (Velickovic et al., 2018) , image processing (Fu et al., 2019) , and reinforcement learning/robotics (Tang et al., 2020) .
APPENDIX: RETHINKING ATTENTION WITH PERFORMERS A HYPERPARAMETERS FOR EXPERIMENTS
This optimal setting (including comparisons to approximate softmax) we use for the Performer is specified in the Generalized Attention (Subsec. A.4), and unless specifically mentioned (e.g. using name "Performer-SOFTMAX"), "Performer" refers to using this generalized attention setting.
A.1 METRICS
We report the following evaluation metrics:
1. Accuracy: For unidirectional models, we measure the accuracy on next-token prediction, averaged across all sequence positions in the dataset. For bidirectional models, we mask each token with 15% probability (same as (Devlin et al., 2018) ) and measure accuracy across the masked positions.
2. Perplexity: For unidirectional models, we measure perplexity across all sequence positions in the dataset. For bidirectional models, similar to the accuracy case, we measure perplexity across the masked positions.
3. Bits Per Dimension/Character (BPD/BPC): This calculated by loss divided by ln(2).
We used the full evaluation dataset for TrEMBL in the plots in the main section, while for other datasets such as ImageNet64 and PG-19 which have very large evaluation dataset sizes, we used random batches (>2048 samples) for plotting curves.
A.1.1 PG-19 PREPROCESSING
The PG-19 dataset (Rae et al., 2020) is presented as a challenging long range text modeling task. It consists of out-of-copyright Project Gutenberg books published before 1919. It does not have a fixed vocabulary size, instead opting for any tokenization which can model an arbitrary string of text. We use a unigram SentencePiece vocabulary (Kudo & Richardson, 2018) with 32768 tokens, which maintains whitespace and is completely invertible to the original book text. Perplexities are calculated as the average log-likelihood per token, multiplied by the ratio of the sentencepiece tokenization to number of tokens in the original dataset. The original dataset token count per split is: train=1973136207, validation=3007061, test=6966499. Our sentencepiece tokenization yields the following token counts per split: train=3084760726, valid=4656945, and test=10699704. This gives log likelihood multipliers of train=1.5634, valid=1.5487, test=1.5359 per split before computing perplexity, which is equal to exp(log likelihood multiplier * loss).
Preprocessing for TrEMBL is extensively explained in Appendix C.
A.2 TRAINING HYPERPARAMETERS
Unless specifically stated, all Performer + Transformer runs by default used 0.5 grad clip, 0.1 weight decay, 0.1 dropout, 10 -3 fixed learning rate with Adam hyperparameters (β 1 = 0.9, β 2 = 0.98, = 10 -9 ), with batch size maximized (until TPU memory overload) for a specific model.
All 36-layer protein experiments used the same amount of compute (i.e. 16x16 TPU-v2, 8GB per chip). For concatenated experiments, 16x16 TPU-v2's were also used for the Performer, while 8x8's were used for the 1-3 layer (d = 256) Transformer models (using 16x16 did not make a difference in accuracy).
Note that Performers are using the same training hyperparameters as Transformers, yet achieving competitive results -this shows that FAVOR can act as a simple drop-in without needing much tuning.
A.3 APPROXIMATE SOFTMAX ATTENTION DEFAULT VALUES
The optimal values, set to default parameters1 , are: renormalize_attention = True, numerical stabilizer = 10 -6 , number of features = 256, ortho_features = True, ortho_scaling = 0.0.
A.4 GENERALIZED ATTENTION DEFAULT VALUES
The optimal values, set to default parameters2 , are: renormalize_attention = True, numerical stabilizer = 0.0, number of features = 256, kernel = ReLU, kernel_epsilon = 10 -3 .
A.5 REFORMER DEFAULT VALUES
For the Reformer, we used the same hyperparameters as mentioned for protein experiments, without gradient clipping, while using the defaults 3 (which instead use learning rate decay) for ImageNet-64.
In both cases, the Reformer used the same default LSH attention parameters.
A.6 LINFORMER DEFAULT VALUES Using our standard pipeline as mentioned above, we replaced the attention function with the Linformer variant via Jax, with δ = 10 -6 , k = 600 (same notation used in the paper (Wang et al., 2020) ), where δ is the exponent in a renormalization procedure using e -δ as a multiplier in order to approximate softmax, while k is the dimension of the projections of the Q and K matrices. As a sanity check, we found that our Linformer implementation in Jax correctly approximated exact softmax's output within 0.02 error for all entries.
Note that for rigorous comparisons, our Linformer hyperparameters are even stronger than the defaults found in (Wang et al., 2020) , as:
• We use k = 600, which is more than twice than the default k = 256 from the paper, and also twice than our default m = 256 number of features.
• We also use redrawing, which avoids "unlucky" projections on Q and K.
B MAIN ALGORITHM: FAVOR+
We outline the main algorithm for FAVOR+ formally:
Algorithm 1: FAVOR+ (bidirectional or unidirectional).
Input : Q, K, V ∈ R L×d , isBidirectional -binary flag. Result: Att ↔ (Q, K, V) ∈ R L×L if isBidirectional, Att → (Q, K, V) ∈ R L×L otherwise.
Compute Q and K as described in Section 2.2 and Section 2.3 and take
C := [V 1 L ]; if isBidirectional then Buf 1 := (K ) C ∈ R M ×(d+1) , Buf 2 := Q Buf 1 ∈ R L×(d+1) ; else
Compute G and its prefix-sum tensor G PS according to (11);
Buf 2 := G PS 1,:,: Q 1 . . . G PS L,:,: Q L ∈ R L×(d+1) ; end [Buf 3 buf 4 ] := Buf 2 , Buf 3 ∈ R L×d , buf 4 ∈ R L ; return diag(buf 4 ) -1 Buf 3 ; B.
1 UNIDIRECTIONAL CASE AND PREFIX SUMS
We explain how our analysis from Section 2.2 can be extended to the unidirectional mechanism in this section. Notice that this time attention matrix A is masked, i.e. all its entries not in the lower-triangular part (which contains the diagonal) are zeroed (see also Fig. 8 ).
Figure 8 : Visual representation of the prefix-sum algorithm for unidirectional attention. For clarity, we omit attention normalization in this visualization. The algorithm keeps the prefix-sum which is a matrix obtained by summing the outer products of random features corresponding to keys with value-vectors. At each given iteration of the prefix-sum algorithm, a random feature vector corresponding to a query is multiplied by the most recent prefix-sum (obtained by summing all outer-products corresponding to preceding tokens) to obtain a new row of the matrix AV which is output by the attention mechanism.
For the unidirectional case, our analysis is similar as for the bidirectional case, but this time our goal is to compute tril(Q (K ) )C without constructing and storing the L × L-sized matrix tril(Q (K ) ) explicitly, where d+1) . In order to do so, observe that ∀1 ≤ i ≤ L:
C = [V 1 L ] ∈ R L×(
[tril(Q (K ) )C] i = G PS i,:,: × Q i , G PS i,:,: = i j=1 G j,:,: , G j,:,: = K j C j ∈ R M ×(d+1) (11)
where G, G PS ∈ R L×M ×(d+1) are 3d-tensors. Each slice G PS :,l,p is therefore a result of a prefix-sum (or cumulative-sum) operation applied to G :,l,p : G PS i,l,p = i j=1 G i,l,p . An efficient algorithm to compute the prefix-sum of L elements takes O(L) total steps and O(log L) time when computed in parallel (Ladner & Fischer, 1980; Cormen et al., 2009) . See Algorithm 1 for the whole approach.
B.2 ORTHOGONAL RANDOM FEATURES -EXTENSIONS
As mentioned in the main text, for isotropic Ω (true for most practical applications, including regular attention), instead of sampling ω i independently, we can use orthogonal random features (ORF) (Yu et al., 2016; Choromanski et al., 2017; 2018b) : these maintain the marginal distributions of samples ω i while enforcing that different samples are orthogonal. If we need m > d, ORFs still can be used locally within each d × d block of W (Yu et al., 2016) .
ORFs were introduced to reduce the variance of Monte Carlo estimators (Yu et al., 2016; Choromanski et al., 2017; 2018b; 2019a; Rowland et al., 2019; Choromanski et al., 2018a; 2019b) and we showed in the theoretical and experimental sections from the main body that they do indeed lead to more accurate approximations and substantially better downstream results. There exist several variants of the ORF-mechanism and in the main body we discussed only the base one (that we refer to here as regular). Below we briefly review the most efficient ORF mechanisms (based on their strengths and costs) to present the most complete picture.
(1) Regular ORFs [R-ORFs]: Applies Gaussian orthogonal matrices (Yu et al., 2016) . Encodes matrix W of ω-samples (with different rows corresponding to different samples) in O(md) space. Provides algorithm for computing Wx in O(md) time for any x ∈ R d . Gives unbiased estimation. Requires one-time O(md 2 ) preprocessing (Gram-Schmidt orthogonalization).
(2) Hadamard/Givens ORFs [H/G-ORFs]: Applies random Hadamard (Choromanski et al., 2017) or Givens matrices (Choromanski et al., 2019b) . Encodes matrix W in O(m) or O(m log(d)) space. Provides algorithm for computing Wx in O(m log(d)) time for any x ∈ R d . Gives small bias (tending to 0 with d → ∞).
B.3 TIME AND SPACE COMPLEXITY -DETAILED ANALYSIS
We see that a variant of bidirectional FAVOR+ using iid samples or R-ORFs has O(md + Ld + mL) space complexity as opposed to Θ(L 2 + Ld) space complexity of the baseline. Unidirectional FAVOR+ using fast prefix-sum pre-computation in parallel (Ladner & Fischer, 1980; Cormen et al., 2009) has O(mLd) space complexity to store G PS which can be reduced to O(md + Ld + mL) by running a simple (though non-parallel in L) aggregation of G PS i,:,: without storing the whole tensor G PS in memory. From Subsec. B.2, we know that if instead we use G-ORFs, then space complexity is reduced to O(m log(d) + Ld + mL) and if the H-ORFs mechanism is used, then space is further reduced to O(m+Ld+mL) = O(Ld+mL). Thus for m, d L all our variants provide substantial space complexity improvements since they do not need to store the attention matrix explicitly.
The time complexity of Algorithm 1 is O(Lmd) (note that constructing Q and K can be done in time O(Lmd)). Note that the time complexity of our method is much lower than O(L 2 d) of the baseline for L m.
As explained in Subsec. B.2, the R-ORF mechanism incurs an extra one-time O(md 2 ) cost (negligible compared to the O(Lmd) term for L d). H-ORFs or G-ORFs do not have this cost, and when FAVOR+ uses them, computing Q and K can be conducted in time O(L log(m)d) as opposed to O(Lmd) (see: Subsec. B.2). Thus even though H/G-ORFs do not change the asymptotic time complexity, they improve the constant factor from the leading term. This might play an important role in training very large models.
The number of random features m allows a trade-off between computational complexity and the level of approximation: bigger m results in higher computation costs, but also in a lower variance of the estimate of A. In the theoretical section from the main body we showed that in practice we can take M = Θ(d log(d)).
Observe that the FAVOR+ algorithm is highly-parallelizable, and benefits from fast matrix multiplication and broadcasted operations on GPUs or TPUs. (Madani et al., 2020) , we argue that it includes most of the relevant sequences. Specifically, the TrEMBL dataset consists of the subset of UniProtKB sequences that have been computationally analyzed but not manually curated, and accounts for ≈ 99.5% of the total number of sequences in the UniProtKB dataset5 .
C EXPERIMENTAL DETAILS FOR PROTEIN MODELING TASKS
Following the methodology described in Madani et al. (Madani et al., 2020) , we used both an OOD-Test set, where a selected subset of Pfam families are held-out for valuation, and an IID split, where the remaining protein sequences are split randomly into train, valid, and test tests. We held-out the following protein families (PF18369, PF04680, PF17988, PF12325, PF03272, PF03938, PF17724, PF10696, PF11968, PF04153, PF06173, PF12378, PF04420, PF10841, PF06917, PF03492, PF06905, PF15340, PF17055, PF05318), which resulted in 29,696 OOD sequences. We note that, due to deduplication and potential TrEMBL version mismatch, our OOD-Test set does not match exactly the one in Madani et al. (Madani et al., 2020) . We also note that this OOD-Test selection methodology does not guarantee that the evaluation sequences are within a minimum distance from the sequences used during training. In future work, we will include rigorous distance based splits.
The statistics for the resulting dataset splits are reported in Table 1 . In the standard sequence modeling task, given the length statistics that are reported in the table, we clip single sequences to maximum length L = 1024, which results in few sequences being truncated significantly.
In the long sequence task, the training and validation sets are obtained by concatenating the sequences, separated by an end-of-sequence token, and grouping the resulting chain into non-overlapping sequences of length L = 8192. A random baseline, with uniform probability across all the vocabulary tokens at every position, has accuracy 5% (when including only the 20 standard amino acids) and 4% (when also including the 5 anomalous amino acids (Consortium, 2019)). However, the empirical frequencies of the various amino acids in our dataset may be far from uniform, so we also consider an empirical baseline where the amino acid probabilities are proportional to their empirical frequencies in the training set.
C.2 EMPIRICAL BASELINE
Figure 9 shows the estimated empirical distribution. We use both the standard and anomalous amino acids, and we crop sequences to length 1024 to match the data processing performed for the Transformer models. The figure shows only the 20 standard amino acids, colored by their class, for comparison with the visualization on the TrEMBL web page6 .
C.3 TABULAR RESULTS
Table 2 contains the results on the single protein sequence modeling task (L = 1024). We report accuracy and perplexity as defined in Appendix A: ). We note that the empirical baseline results are applicable to both the unidirectional (UNI) and bidirectional (BID) models.
C.4 ATTENTION MATRIX ILLUSTRATION
In this section we illustrate the attention matrices produced by a Performer model. We focus on the bidirectional case and choose one Performer model trained on the standard single-sequence TrEMBL task for over 500K steps. The same analysis can be applied to unidirectional Performers as well.
We note that while the Transformer model instantiates the attention matrix in order to compute the attention output that incorporates the (queries Q, keys K, values V ) triplet (see Eq. 1 in the main paper), the FAVOR mechanism returns the attention output directly (see Algorithm 1). To account for this discrepancy, we extract the attention matrices by applying each attention mechanism twice: once on each original (Q, K, V ) triple to obtain the attention output, and once on a modified (Q, K, V • ) triple, where V • contains one-hot indicators for each position index, to obtain the attention matrix.
The choice of V • ensures that the dimension of the attention output is equal to the sequence length, and that a non-zero output on a dimension i can only arise from a non-zero attention weight to the i th sequence position. Indeed, in the Transformer case, when comparing the output of this procedure with the instantiated attention matrix, the outputs match.
Attention matrix example. We start by visualizing the attention matrix for an individual protein sequence. We use the BPT1_BOVIN protein sequence7 , one of the most extensively studied globular proteins, which contains 100 amino acids. In Figure 10 , we show the attention matrices for the first 4 layers. Note that many heads show a diagonal pattern, where each node attends to its neighbors, and some heads show a vertical pattern, where each head attends to the same fixed positions. These patterns are consistent with the patterns found in Transformer models trained on natural language (Kovaleva et al., 2019) . In Figure 12 we highlight these attention patterns by focusing on the first 25 tokens, and in Figure 11 , we illustrate in more detail two attention heads.
Amino acid similarity. Furthermore, we analyze the amino-acid similarity matrix estimated from the attention matrices produced by the Performer model, as described in Vig et al. (Vig et al., 2020) . We aggregate the attention matrix across 800 sequences. The resulting similarity matrix is illustrated in Figure 13 . Note that the Performer recognizes highly similar amino acid pairs such as (D, E) and (F, Y).
Figure 10 : We show the attention matrices for the first 4 layers and all 8 heads (each row is a layer, each column is head index, each cell contains the attention matrix across the entire BPT1_BOVIN protein sequence). Note that many heads show a diagonal pattern, where each node attends to its neighbors, and some heads show a vertical pattern, where each head attends to the same fixed positions. Figure 12 : We highlight the attention patterns by restricting our attention to the first 25 tokens (note that we do not renormalize the attention to these tokens). The illustration is based on Vig et al. (Vig, 2019; Vig & Belinkov, 2019) . Note that, similar to prior work on protein Transformers (Madani et al., 2020) , the attention matrices include both local and global patterns. 
A C D E F G H I K L M N P Q R S T V W Y A C D E F G H I K L M N P Q R S T V W Y 0.0 0.2 0.4 0.6 0.8 1.0 A C D E F G H I K L M N P Q R S T V W Y A C D E F G H I K L M N P Q R S T V W Y 0
D.1 BACKWARDS COMPATIBILITY -ERROR PROPAGATION
Although mentioned previously (Sec. 4.2) that the Performer with additional finetuning is backwards compatible with the Transformer, we demonstrate below in Fig. 14 that error propagation due to nonattention components of the Transformer is one of the primary reasons that pretrained Transformer weights cannot be immediately used for inference on the corresponding Performer.
Figure 14 : Output approximation errors between a vanilla Transformer and a Performer (with orthogonal features) for varying numbers of layers.
D.2 APPROXIMATE SOFTMAX -EXTENDED PROPERTIES
We show the following properties of our softmax approximation, in Fig. 15 :
Redrawing: While the benefits of redrawing features was shown in Subsec. 4.3 of the main body of the paper, we also demonstrate its benefits when there are multiple layers with large scale (16x16 TPU-v2) training.
Unidirectional: While we have shown on TrEMBL that Performer with generalized ReLU attention outperforms softmax, we also show that approximate softmax attention can still be a solid choice, for example on ImageNet64 (U). After 100K steps of training, the Performer-ReLU, Performer-Softmax, and Performer-Softmax (SMREG) variants achieve respectively, 3.67, 3.69, 3.67 BPD.
Instability of Trigonometric Features:
We see the full view of the unstable training curve when using Trigonometric softmax. We investigated Generalized Attention mechanisms (mentioned in Sec. 2.2) on TrEMBL when L = 512 for various kernel functions. This is similar to (Tsai et al., 2019) which also experiments with various attention kernels for natural language. Using hyperparameter sweeps across multiple variables in FAVOR, we compared several kernels and also renormalization on/off (Fig. 16 and Fig. 17 ), where Renormalize corresponds to applying D -1 operator in attention, as for the standard mechanism, though we noticed that disabling it does not necessarily hurt accuracy) to produce the best training configuration for the Performer. We note that the effective batch size slightly affects the rankings (as shown by the difference between 2x2 and 4x4 TPU runs) -we by default use the generalized ReLU kernel with other default hyperparameters shown in Appendix A, as we observed that they are empirically optimal for large batch size runs (i.e. 8x8 or 16x16 TPU's).
Figure 16 : To emphasize the highest accuracy runs but also show the NaN issues with certain kernels which caused runs to stop early, we set both x and y axes to be log-scale. We tested kernels defined by different functions f (see: Sec. 2.2): sigmoid, exponential, ReLU, absolute, gelu, cosine (original softmax approximation), tanh, and identity. All training runs were performed on 2x2 TPU-v2's, 128 batch size per device.
Figure 17 : We also performed a similar setup as Fig. 16 for 4x4 TPU-v2's. D.4 COMPARISON WITH LINEAR TRANSFORMER We use the attention implementation of the Linear Transformer from (Katharopoulos et al., 2020) , which mainly involves setting our feature map φ(x) = elu(x) + 1, where elu(x) is the shifted-eLU function from (Clevert et al., 2016) . 
E COMPUTATION COSTS -EXTENDED RESULTS
In this subsection, we empirically measure computational costs in terms wall clock time on forward and backward passes for three scenarios in Fig. 20: 1. Performer, with varying number of layers. We show that our method can scale up to (but not necessarily limited to) even 20 layers.
2. Attention time complexities when comparing standard attention (from Transformer) and FAVOR (from Performer). Note that the maximum memory size here is not reflective of the maximum memory size in an actual model (shown below), as this benchmark requires computing explicit tensors (causing memory increases) in Jax, while a model does not.
3. Time complexities when comparing the Transformer and Performer models. "X" (OPT) denotes the maximum possible speedup achievable, when attention simply returns the Vvector, showing that the Performer is nearly optimal. We see that the maximum possible power of 2 length allowed on a V100 GPU (16GB) is 2 15 = 32768 using regular dimensions.
Since some of the computational bottleneck in the Transformer may originate from the extra feed-forward layers (Kitaev et al., 2020) , we also benchmark the "Small" version, i.e.
(n heads , n layers , d f f , d) = (1, 6, 64, 64) as well, when the attention component is the dominant source of computation and memory. We remind the reader that the "Regular" version consists of (n heads , n layers , d f f , d) = (8, 6, 2048, 512) . 
F THEORETICAL RESULTS
We provide here the proofs of all theoretical results presented in the paper.
F.1 PROOF OF LEMMA 1
Proof. We first deduce that for any a, b ∈ R d SM(x, y) = exp(x y) = exp(-
x 2 /2) • exp( x + y 2 /2) • exp(-y 2 /2).
Next, let w ∈ R d . We use the fact that
(2π) -d/2 exp(-w -c 2 2 /2)dw = 1
for any c ∈ R d and derive:
exp( x + y 2 /2) = (2π) -d/2 exp( x + y 2 /2) exp(-w -(x + y) 2 /2)dw = (2π) -d/2 exp(-w 2 /2 + w (x + y) -x + y 2 /2 + x + y 2 /2)dw = (2π) -d/2 exp(-w 2 /2 + w (x + y))dw = (2π) -d/2 exp(-w 2 /2) • exp(w x) • exp(w y)dw = E ω∼N (0 d ,I d ) [exp(ω x) • exp(ω y)].
That completes the proof of the first part of the lemma. An identity involving hyperbolic cosine function is implied by the fact that for every u ∈ R d and ω ∼ N (0, I d ) the following is true:
E[exp(ω u)] = ∞ i=0 E[(ω u) 2i ] (2i)! = 1 2 ∞ i=0 E[(ω u) 2i ] + E[(-ω u) 2i ] (2i)! . ()
The cancellation of the odd moments E[(ω u) 2i+1 ] follows directly from the fact that ω is taken from the isotropic distribution (i.e. distribution with pdf function constant on each sphere). That completes the proof.
F.2 PROOF OF LEMMA 2
Proof. Denote: z = x + y and ∆ = xy. Note that by using standard trigonometric identities (and the fact that the variance of the sum of independent random variables is the sum of variances of those random variables), we can get the following for ω ∼ N (0, I d ):
MSE( SM trig m (x, y)) = 1 m exp( x 2 + y 2 )Var(cos(ω ∆)).Using the fact that (see: Lemma 1 in (Yu et al., 2016) ; note that in that lemma they use notation: z for what we denote as: ∆ ):
Var(cos(ω ∆)) = 1 2 (1 -exp(-∆ 2 )) 2 ,we obtain:
MSE( SM trig m (x, y)) = 1 2m exp( x 2 + y 2 )(1 -exp(-∆ 2 )) 2 = 1 2m exp( z 2 )SM -2 (x, y)(1 -exp(-∆ 2 )) 2 ,which completes the first part of the proof. To obtain the formula for: MSE( SM + m (x, y)) notice first that:
E ω∼N (0,I d ) [exp(ω z)] = exp( z 2 2 ).The above immediately follows from the fact that positive random feature maps provide unbiased estimation of the softmax-kernel, thus the following is true:
SM(x, y) = exp(- x 2 + y 2 2 )E ω∼N (0,I d ) [exp(ω z)].Therefore we obtain:
MSE( SM + m (x, y)) = 1 m exp(-( x 2 + y 2 ))Var(exp(ω z)) = 1 m exp(-( x 2 + y 2 )) E[exp(2ω z)] -(E[exp(ω z)]) 2 = 1 m exp(-( x 2 + y 2 ))(exp(2 z 2 ) -exp( z 2 )),where the last equality follows from Equation 16. Therefore we have:
MSE( SM + m (x, y)) = 1 m exp(-( x 2 + y 2 )) exp( z 2 )(exp( z 2 ) -1) = 1 m exp( z 2 )SM 2 (x, y)(1 -exp(-z 2 )).Finally,
MSE( SM
hyp+ m (x, y)) = 1 4m exp(- x 2 + y 2 2 ) 2 (Var(exp(ω z)) + Var(exp(-ω z))+ 2Cov(exp(ω z)), exp(-ω z)))) = 1 4m exp(- x 2 + y 2 2 ) 2 (2Var(exp(ω z))+ 2Cov(exp(ω z)), exp(-ω z))))) = 1 2m exp(-( x 2 + y 2 )) (Var(exp(ω z)) + 1 -(E[exp(ω z)]) 2 ) = 1 2m exp(-( x 2 + y 2 )) (exp(2 z 2 ) -exp( z 2 ) + 1 -exp( z 2 )) = 1 2m exp(-( x 2 + y 2 ))(exp( z 2 ) -1) 2 = 1 2 (1 -exp(-z 2 ))MSE( SM + m (x, y)).In the chain of equalities above we used the fact that random variables exp(ω z) and exp(-ω z) have the same distribution. This is true since ω and -ω have the same distribution (ω is Gaussian). That completes the proof.
F.3 PROOF OF THEOREM 1
Proof. Let x, y ∈ R d be respectively a query/key. Note that from the definition of SMREG(x, y) we have for z = x + y:
SMREG(x, y) = exp(- x 2 + y 2 2 ) ∞ k=0 1 (2k)! z 2k d k E ω∼N (0,I d ) [( ω ω 2 e 1 ) 2k ],where e 1 def = (1, 0, ..., 0) ∈ R d . To obtain the above we used the fact that N (0, I d ) is isotropic (that in particular implies zeroing of the even terms in the Taylor expansion).
Let us denote:
A(k, d) def = E ω∼N (0,I d ) [( ω ω 2 e 1 ) 2k ]. It turns out that: A(2k, d) = (2k -1)!! (d + 2k -2)(d + 2k -4) • ... • d . ()
The proof of that fact can be found in the supplement of (Choromanski et al., 2018b ), yet we provide it below for completeness and the convenience of the Reader:
Lemma 3. Expression A(2k, d) satisfies the following for k ∈ N :
A(2k, d) = (2k -1)!! (d + 2k -2)(d + 2k -4) • ... • d .Proof. Note first that for d ≥ 2 the density function p d (θ) of the angle between a vector r ∈ R d chosen uniformly at random from the unit sphere and e 1 is given by the following formula:
p d (θ) = sin d-2 (θ) π 0 sin d-2(θ) dθ . ()
Let us denote:
F (k, d) def = π 0 cos k (θ) sin d (θ)dθ.
Using partial integration, we get:
π 0 cos k (θ) sin d (θ)dθ = π 0 cos k-1 (θ) sin d (θ)(sin(θ)) dθ = cos k-1 (θ) sin d+1 (θ)| π 0 - π 0 sin(θ)((k -1) cos k-2 (θ)(-sin(θ)) sin d (θ)+ d k (θ) sin d-1 (θ))dθ.Thus we conclude that:
F (k, d) = k-1 d+1 F (k -2, d + 2). Therefore we F (2k, d) = (2k -1)!! (d + 1)(d + 3) • ... • (d + 2k -1) π 0 sin d+2k (θ)dθ.We again conduct partial integration and get:
π 0 sin d (θ)dθ = - 1 d sin d-1 (θ) cos(θ)| π 0 + d -1 d π 0 sin d-2 (θ)dθ = d -1 d π 0 sin d-2 (θ)dθ.Therefore we conclude that:
A(2k, d) = 1 d-3 d-2 d-5 d-4 • ... (2k -1)!! (d -1)(d + 1) • ... • (d + 2k -3) d + 2k -3 d + 2k -2 d + 2k -5 d + 2k -4 • .... = (2k -1)!! (d + 2k -2)(d + 2k -4) • ... • d ,which completes the proof.
Applying the above lemma, we get:
SMREG(x, y) = exp(- x 2 + y 2 2 ) ∞ k=0 1 (2k)! z 2k d k (2k -1)!! (d + 2k -2)(d + 2k -4) • ... • d = exp(- x 2 + y 2 2 ) ∞ k=0 w k k! f (k, d),where
w = z 2 2 and f (k, d) = d k (d+2k-2)(d+2k-4)•...•d . Thus we obtain: SMREG(x, y) SM(x, y) = e -w ∞ k=0 w k k! f (k, d).Note first that for k ≥ 1 we have: f (k, d) ≤ 1, thus:
SMREG(x, y) ≤ SM(x, y).Published as a conference paper at ICLR 2021
We also have for l = d 
3 : SMREG(x, y) SM(x, y) = e -w l k=0 w k k! f (k, d) + e -w ∞ k=l+1 w k k! f (k, d) ≥ f (l, d)e -w l k=0 w k k! + e -w ∞ k=l+1 w k k! f (k, d) ≥ f (l, d)(1 -e -w ∞ k=l+1 w k k! ) = f (l, d)(1 -P[Po(w) > l]),where Po(w) stands for the random variable of Poisson distribution with parameter w. Therefore we get for t = ln( l w ): SMREG(x, y) SM(x, y)
≥ (1 - 2l -2 d ) l (1 -P[Po(w) > l]) ≥ exp(l ln(1 - 2l -2 d ))(1 -P[tPo(w) ≥ tl]) = exp l ∞ i=1 (-1) i ( 2l-2 d ) i i (1 -P[exp(tPo(w) -tl) ≥ 1]) ≥ exp(- 2 d 1 3 + o( 1 d 1 3 ))(1 -exp(-tl)E[exp(tPo(w))]) = exp(- 2 d 1 3 + o( 1 d 1 3 ))(1 -exp(-w -l(t -1))),where the last equality is implied by the formula for the Laplace Transform for the Poisson random variable:
E[exp(tPo(w))] = exp(w(exp(t) -1)).Notice that: w = z 2 2 = ln(SM(x,x))+ln(SM(y,y))+2 ln(SM(x,y)) 2 ≤ 2 ln(C). We conclude that:
SMREG(x, y) SM(x, y) ≥ (1 - 2 d 1 3 + o( 1 d 1 3 ))(1 -C -2 ( d 1 3 2e • ln(C) ) -d 1 3 ) = 1 - 2 d 1 3 + o( 1 d 1 3
). (35)
That completes the proof.
F.4 PROOFS OF THEOREM 2,THEOREM 3 & BEAUTIFUL FUNCTIONS
We will provide here much more general theoretical results which will imply Theorem 3 and Theorem 2. We need the following definition: Definition 1. We say that function F : R n → R is beautiful if F can be expressed as:
F Ω,g (z) = E ω∼Ω [g(ω z)],for a probabilistic isotropic distribution Ω, and where g : R → R is an entire function with nonnegative power-series coefficients (i.e. g(x) = ∞ i=0 a i x i for every x ∈ R and with a i ≥ 0 for i = 0, 1, ...). In the formula above we assume that the expectation on the RHS exists.
Interestingly, beautiful functions can be used to define softmax and consequently, Gaussian kernels (both standard and regularized), leading to our PRF mechanism presented in the main body of the paper, as we explain below. Remark 1. If one takes Ω = N (0, I d )(note that N (0, I d ) is isotropic) and g : x → exp(x) (such g is clearly entire with nonnegative power-series coefficient) then the following is true for z = x + y: SM(x, y) = exp(-x 2 + y 2 2 )F Ω,g (z).
(37)
Similarly: SMREG(x, y) = exp(-x 2 + y 2
2
)F Ωreg,g (z), where Ω reg stands for the distribution corresponding to Haar measure on the sphere of radius √ d (which is clearly isotropic). Therefore general concentration results for Monte Carlo estimators of beautiful functions immediately imply corresponding results for the (standard and regularized) softmax (and thus also Gaussian) kernel.
We will consider two estimators of the beautiful functions from Definition 1 that directly lead (through Remark 1) to: PRF-based approximation of the softmax-kernel and its enhanced version with orthogonal features. Standard Monte Carlo estimator samples independently ω iid 1 , ..., ω iid m iid ∼ Ω, where m stands for the number of samples and then computes:
F iid m (z) def = 1 m m i=1 g((ω iid i ) z).Orthogonal Monte Carlo estimator samples ω ort 1 , ..., ω ort m (m ≤ d) in such a way that marginally we have: ω ort i ∼ Ω, but (ω ort i ) ω ort j = 0 for i = j (such an orthogonal ensemble can be always created if Ω is isotropic, as we already mentioned in the main body of the paper). We define:
F ort m (z) def = 1 m m i=1 g((ω ort i ) z).F.4.1 ORTHOGONALITY UNIVERSALLY IMPROVES CONCENTRATION Denote by M Z (θ) = E[e θZ ] a moment generating function of the random variable Z. Note first that estimators of beautiful functions based on standard Monte Carlo procedure using independent vectors ω iid i guarantee strong concentration bounds since independent ω i s provide a way to obtain exponentially small upper bounds on failure probabilities through moment generating functions. We summarize this classic observation which is a standard application of Markov's Inequality below.
Lemma 4. Consider an estimator F iid m (z) of the beautiful function F evaluated at z. Then the following holds for any a > F (z), θ > 0:
P[ F iid m (z) > a] ≤ exp(θma)M X (θ) m ,where X = g(w z), w ∼ D.
The above result provides us with exponentially small (in Legendre Transform) upper bounds on tail probabilities for the standard estimator. Below we provide our two main theoretical results.
Theorem 5 (orthogonality provides smaller tails). If F Ω,g is a beautiful function then the following holds for m ≤ d, X as in Lemma 4 and any a > F (z), θ > 0:
P[ F ort m (z)) > a] ≤ exp(-θma) M X (θ) m - θ 4 m(m -1) 4d 2 (d + 2) a M -2 0 a 2 1 z 4 (E ω 2 ) 2 . ()
This result shows that features obtained from the ensembles of pairwise orthogonal random vectors provide exponentially small bounds on tail probabilities and that these bounds are strictly better than for estimators using unstructured features. Furthermore, the result is universal, i.e. holds for any dimensionality d, not just asymptotically for d large enough.
We also obtain similar result regarding mean squared errors (MSEs) of the considered estimators: Theorem 6. If F Ω,g is a beautiful function then the following holds for m ≤ d:
MSE( F ort m (z)) ≤ MSE( F iid m (z)) -(1 - 1 m ) 2 d + 2 (F Ω,g (z) -a 0 ) 2 . ()
As before, an orthogonal estimator leads to better concentration results and as before, this is the case for any d > 0, not only asymptotically for large enough d.
Note that from what we have said above, Theorem 2 and Theorem 3 follow immediately from Theorem 6 and Theorem 5 respectively.
Thus in the remainder of this section we will prove Theorem 6 and Theorem 5. 
https://github.com/google-research/google-research/blob/master/ performer/fast_attention
https://github.com/google-research/google-research/blob/master/ performer/fast_attention
https://github.com/google/trax/blob/master/trax/supervised/configs/ reformer_imagenet64.gin
https://www.uniprot.org/statistics/TrEMBL
https://www.uniprot.org/uniprot/
https://www.uniprot.org/statistics/TrEMBL
https://www.uniprot.org/uniprot/P00974
References
[1]Bello Irwan, Zoph Barret, Vaswani Ashish, Shlens Jonathon, Quoc V, Le . Attention augmented convolutional networks.  2019;:.
[2]Beltagy Iz, Peters MatthewE, Cohan Arman. Longformer: The long-document transformer.  2020;:.
[3]Chan William, Saharia Chitwan, Hinton GeoffreyE, Norouzi Mohammad, Jaitly Navdeep. Imputer: Sequence modelling via imputation and dynamic programming.  2020;:.
[4]Chelba Ciprian, Mikolov Tomas, Schuster Mike, Ge Qi, Brants Thorsten, Koehn Phillipp, et al.. One billion word benchmark for measuring progress in statistical language modeling. INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association 2014;:2635--2639.
[5]Chelba Ciprian, Xu Chen Mia, Bapna Ankur, Shazeer Noam. Faster transformer decoding: N-gram masked self-attention.  2020;:.
[6]Xu Mia,  Chen, Firat Orhan, Bapna Ankur, Johnson Melvin, Macherey Wolfgang, et al.. The best of both worlds: Combining recent advances in neural machine translation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 2018;1:76--86.
[7]Child Rewon, Gray Scott, Radford Alec, Sutskever Ilya. Generating long sequences with sparse transformers.  2019;:.
[8]Choromanski Krzysztof, Downey Carlton, Boots Byron. Initialization matters: Orthogonal predictive state recurrent neural networks. 6th International Conference on Learning Representations, ICLR 2018 2018;:.
[9]Choromanski Krzysztof, Rowland Mark, Sarlós Tamás, Sindhwani Vikas, Turner RichardE, Weller Adrian. The geometry of random features. International Conference on Artificial Intelligence and Statistics 2018;2018:1--9.
[10]Choromanski Krzysztof, Pacchiano Aldo, Pennington Jeffrey, Tang Yunhao. KAMA-NNs: Low-dimensional rotation based neural networks. The 22nd International Conference on Artificial Intelligence and Statistics 2019;89:236--245.
[11]Choromanski Krzysztof, Rowland Mark, Chen Wenyu, Weller Adrian. Unifying orthogonal Monte Carlo methods. Proceedings of the 36th International Conference on Machine Learning, ICML 2019;97:1203--1212.
[12]Krzysztof Marcin Choromanski Mark, Rowland Adrian, Weller . The unreasonable effectiveness of structured random orthogonal embeddings. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017;:219--228.
[13]Clevert Djork-Arné, Unterthiner Thomas, Hochreiter Sepp. Fast and accurate deep network learning by exponential linear units (elus). 4th International Conference on Learning Representations 2016;:.
[14]Cong Qian, Anishchenko Ivan, Ovchinnikov Sergey, Baker David. Protein interaction networks revealed by proteome coevolution. Science 2019;365:185--189.
[15]. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research 2019;47:506--D515.
[16]Dai Zihang, Yang Zhilin, Yang Yiming, Cohen WilliamW, Carbonell Jaime, Le QuocV, et al.. Transformer-XL: Language modeling with longer-term dependency.  2019;:.
[17]Dehghani Mostafa, Gouws Stephan, Vinyals Oriol, Uszkoreit Jakob, Kaiser Lukasz. Universal transformers. 7th International Conference on Learning Representations 2019;:.
[18]Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina. BERT: pre-training of deep bidirectional transformers for language understanding.  2018;:.
[19]Du Yilun, Meier Joshua, Ma Jerry, Fergus Rob, Rives Alexander. Energy-based models for atomic-resolution protein conformations.  2020;:.
[20]Elnaggar Ahmed, Heinzinger Michael, Dallago Christian, Rost Burkhard. End-to-end multitask learning, from protein language to protein features without alignments. bioRxiv 2019;:.
[21]Frostig Roy, Johnson Matthew, Leary Chris. Compiling machine learning programs via highlevel tracing. Conference on Machine Learning and Systems 2018;:.
[22]Fu Jun, Liu Jing, Tian Haijie, Li Yong, Bao Yongjun, Fang Zhiwei, et al.. Dual attention network for scene segmentation. IEEE Conference on Computer Vision and Pattern Recognition 2019;:3146--3154.
[23]Gulati Anmol, Qin James, Chiu Chung-Cheng, Parmar Niki, Zhang Yu, Yu Jiahui, et al.. Conformer: Convolution-augmented transformer for speech recognition.  2020;:.
[24]Huang Cheng-Zhi Anna, Vaswani Ashish, Uszkoreit Jakob, Simon Ian, Hawthorne Curtis, Shazeer Noam, et al.. Music transformer: Generating music with long-term structure. 7th International Conference on Learning Representations 2019;:.
[25]Ingraham John, Garg Vikas, Barzilay Regina, Jaakkola Tommi. Generative models for graphbased protein design. Advances in Neural Information Processing Systems 2019;:15794--15805.
[26]Katharopoulos Angelos, Vyas Apoorv, Pappas Nikolaos, Fleuret François. Transformers are rnns: Fast autoregressive transformers with linear attention.  2006;:.
[27]Kitaev Nikita, Kaiser Lukasz, Levskaya Anselm. Reformer: The efficient transformer. 8th International Conference on Learning Representations, ICLR 2020 2020;:.
[28]Kovaleva Olga, Romanov Alexey, Rogers Anna, Rumshisky Anna. Revealing the dark secrets of bert.  2019;:.
[29]Kudo Taku, Richardson John. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  2018;:.
[30]Ladner RichardE, Fischer MichaelJ. Parallel prefix computation. J. ACM 1980;27:831--838.
[31]Lin Han, Chen Haoxian, Zhang Tianyi, Laroche Clément, Choromanski Krzysztof. Demystifying orthogonal Monte Carlo and beyond.  2020;:.
[32]Luo Haoneng, Zhang Shiliang, Lei Ming, Xie Lei. Simplified self-attention for transformer-based end-to-end speech recognition.  2020;:.
[33]Madani Ali, Mccann Bryan, Naik Nikhil, Shirish Keskar Nitish, Anand Namrata, Eguchi RaphaelR, et al.. Progen: Language modeling for protein generation.  2020;:.
[34]Marcandalli Jessica, Fiala Brooke, Ols Sebastian, Perotti Michela, De Van Der Schueren Willem, Snijder Joost, et al.. Induction of potent neutralizing antibody responses by a designed protein nanoparticle vaccine for respiratory syncytial virus. Cell 2019;176:1420--1431.
[35]Nangia Nikita, Bowman SamuelR. Listops: A diagnostic dataset for latent tree learning. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2018 2018;:92--99.
[36]Parmar Niki, Vaswani Ashish, Uszkoreit Jakob, Kaiser Lukasz, Shazeer Noam, Ku Alexander, et al.. Image transformer. Proceedings of the 35th International Conference on Machine Learning, ICML 2018 2018;80:4052--4061.
[37]Rae JackW, Potapenko Anna, Siddhant M, Jayakumar Chloe, Hillier TimothyP, Lillicrap . Compressive transformers for long-range sequence modelling. International Conference on Learning Representations 2020;:.
[38]Rahimi Ali, Recht Benjamin. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems 2007;:1177--1184.
[39]Rives Alexander, Goyal Siddharth, Meier Joshua, Guo Demi, Ott Myle, Zitnick C, et al.. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioArxiv.  None;04:.
[40]Rowland Mark, Hron Jiri, Tang Yunhao, Choromanski Krzysztof, Sarlós Tamás, Weller Adrian. Orthogonal estimation of Wasserstein distances. The 22nd International Conference on Artificial Intelligence and Statistics 2019;89:186--195.
[41]Roy Aurko, Saffar Mohammad, Vaswani Ashish, Grangier David. Efficient content-based sparse attention with routing transformers.  2020;:.
[42]Shen Zhuoran, Zhang Mingyuan, Yi Shuai, Yan Junjie, Zhao Haiyu. Factorized attention: Self-attention with linear complexities.  2018;:.
[43]Strubell Emma, Ganesh Ananya, Mccallum Andrew. Energy and policy considerations for deep learning in NLP.  2019;:.
[44]Tang Yujin, Nguyen Duong, Ha David. Neuroevolution of self-interpretable agents.  2020;:.
[45]Tay Yi, Dehghani Mostafa, Abnar Samira, Shen Yikang, Bahri Dara, Pham Philip, et al.. Long range arena: A benchmark for efficient transformers.  2021;:.
[46]Tsai Yao-Hung Hubert, Bai Shaojie, Yamada Makoto, Morency Louis-Philippe, Salakhutdinov Ruslan. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing 2019;:4335--4344.
[47]Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez AidanN, et al.. Attention is all you need. Advances in Neural Information Processing Systems 2017;30:5998--6008.
[48]Velickovic Petar, Cucurull Guillem, Casanova Arantxa, Romero Adriana, Liò Pietro, Bengio Yoshua. Graph attention networks. 6th International Conference on Learning Representations, ICLR 2018 2018;:.
[49]Vig Jesse. A multiscale visualization of attention in the transformer model.  2019;:.
[50]Vig Jesse, Belinkov Yonatan. Analyzing the structure of attention in a transformer language model.  2019;:.
[51]Vig Jesse, Madani Ali, Varshney LavR, Xiong Caiming, Socher Richard, Fatema Nazneen, et al.. Bertology meets biology: Interpreting attention in protein language models.  2020;:.
[52]Vinyals Oriol, Fortunato Meire, Jaitly Navdeep. Pointer networks. Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015;:2692--2700.
[53]Wang Sinong, Li BelindaZ, Khabsa Madian, Fang Han, Ma Hao. Linformer: Self-attention with linear complexity.  2020;:.
[54]Xiao Tong, Li Yinqiao, Zhu Jingbo, Yu Zhengtao, Liu Tongran. Sharing attention weights for fast transformer. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence 2019;:5292--5298.
[55]Yang Zichao, Yang Diyi, Dyer Chris, He Xiaodong, Smola AlexanderJ, Hovy EduardH. Hierarchical attention networks for document classification. NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2016;:1480--1489.
[56]You Haoran, Li Chaojian, Xu Pengfei, Fu Yonggan, Wang Yue, Chen Xiaohan, et al.. Drawing early-bird tickets: Toward more efficient training of deep networks. International Conference on Learning Representations 2020;:.
[57]Yu FelixX, Theertha Suresh Ananda, Krzysztof Marcin, Choromanski , Daniel N, Holtmann-Rice Sanjiv, et al.. Orthogonal random features. Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016;:1975--1983.
[58]Flores Zambaldi Vinícius, Raposo David, Santoro Adam, Bapst Victor, Li Yujia, Babuschkin Igor, et al.. Deep reinforcement learning with relational inductive biases. 7th International Conference on Learning Representations 2019;:.
[59]Zhu Yukun, Kiros Ryan, Zemel RichardS, Salakhutdinov Ruslan, Urtasun Raquel, Torralba Antonio, et al.. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision, ICCV 2015 2015;:19--27.
RETHINKING ATTENTION WITH PERFORMERS
https://arxiv.org/abs/2009.14794
https://arxiv.org/abs/1905.07799
Adaptive Attention Span in Transformers
2019-05-19
Sukhbaatar; Sainbayar; Grave; Edouard; Bojanowski; Piotr; Joulin; Armand
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/1905.07799">Download PDF</a></p><blockquote>
 Abstract: We propose a novel self-attention mechanism that can learn its optimal
attention span. This allows us to extend significantly the maximum context size
used in Transformer, while maintaining control over their memory footprint and
computational time. We show the effectiveness of our approach on the task of
character level language modeling, where we achieve state-of-the-art
performances on text8 and enwiki8 by using a maximum context of 8k characters.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Sainbayar Sukhbaatar [<a href="/show-email/6137cf2a/1905.07799">view email</a>]
 <br />
 <strong><a href="/abs/1905.07799v1">[v1]</a></strong>
 Sun, 19 May 2019 19:43:14 UTC (534 KB)<br /><strong>[v2]</strong>
Thu, 8 Aug 2019 06:58:31 UTC (534 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:1905.07799

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:1905.07799 (cs)
 [Submitted on 19 May 2019 (v1), last revised 8 Aug 2019 (this version, v2)]

 Title: Adaptive Attention Span in Transformers

 Authors: Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, Armand Joulin
 Download PDF
 Abstract: We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.
 Comments: Accepted to ACL 2019 
 Subjects: Machine Learning (cs.LG) ; Machine Learning (stat.ML)
 Cite as: arXiv:1905.07799 [cs.LG] 
 (or arXiv:1905.07799v2 [cs.LG] for this version) 
 https://doi.org/10.48550/arXiv.1905.07799 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Sainbayar Sukhbaatar [view email]
 [v1] Sun, 19 May 2019 19:43:14 UTC (534 KB)
 [v2] Thu, 8 Aug 2019 06:58:31 UTC (534 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 1905
 Change to browse by:
 cs
 stat
 stat.ML

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Sainbayar Sukhbaatar
 Edouard Grave
 Piotr Bojanowski
 Armand Joulin
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs and how to get involved.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Adaptive Attention Span in Transformers
Date Published: 2019-08-08
Authors:
Sainbayar  Sukhbaatar
Edouard  Grave, egrave@fb.com
Piotr  Bojanowski, bojanowski@fb.com
Armand  Joulin, ajoulin@fb.com
Abstract
We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.
1 Introduction
Language models are at the core of many NLP applications, like machine translation or dialogue. Recently, much progress has been made by a new neural network called Transformer (Vaswani et al., 2017) . Part of its success is due to its ability to capture long term dependencies. This is achieved by taking long sequences as inputs and explicitly compute the relations between every token via a mechanism called the "self-attention" layer (Al-Rfou et al., 2019) .
While this layer allows for information to propagate across long distances, it has a computational and memory cost that scales quadratically with the size of the input sequence. As a consequence, Transformers hardly scale to sequences of more than a thousand tokens. This is particularly problematic in the case of character level language modeling where dependencies are often spread over a few thousands time steps.
In this work, we propose an alternative to the self-attention layer to reduce the computational burden of a Transformer. Our layer learns its optimal context size, resulting in a network where each attention layer gathers information on their own context. In practice, we observe that this leads to Transformer with small context in the low-level layers and very large ones for the last layers. With this modification, we are able to scale input sequences to more than 8k tokens with no loss of performance, nor additional computational or memory cost. We validate our approach on the task of character level language modeling where we reach state-of-the-art performances while reducing the number of FLOPS. The code to reproduce our results is publicly available1 .
2.1 Sequential transformer network
Language modeling is the problem of assigning a probability to a sequence of tokens (w 1 , . . . , w T ):
P (w 1 , . . . , w T ) = T t=1 P (w t | w t-1 , . . . , w 1 ).
Recent progress was made with a new autoregressive model called Sequential Transformer (Vaswani et al., 2017) . A Transformer is made of a sequence of layers that are composed of a block of parallel self-attention layers followed by a feedforward network. We refer to Vaswani et al. (2017) for the details on the structure. In this paper, we make a couple of modifications to the Transformer model: we use the relative position embeddings of Shaw et al. (2018) and the caching mechanism of Dai et al. (2019) to speed up the train and test time.
Self-attention layer. A core mechanism of a transformer network is the self-attention layer, which consists of multiple attention heads working in parallel. Each attention head applies the attention mechanism of Bahdanau et al. (2015) to its own input. Given a token t in a sequence, the head first computes similarities with its past, i.e., any token r in the span [t -S, t):
s tr = x t W q (W k x r + p t-r ) ,where W k and W q are the "key" and "query" matrices, and p is the relative position embedding. The attention weights are then obtained by applying a softmax function on these similarities:
a tr = exp (s tr ) t-1 q=t-S exp (s tq ) ,Finally, the head outputs a vector y t by taking the average of the past representations weighted by their attention weights:
y t = t-1 r=t-S a tr W v x r ,where W v is called the "value" matrix. Outputs from different heads are then concatenated together and multiplied by an output matrix W o before feeding to the next layer.
Similar to the memory access mechanisms of Sukhbaatar et al. (2015) , it pulls information from the past to update the current token representation. Repeating this mechanism in consecutive layers allows for information to flow over long distances. However, for each input token, each attention head scales linearly in memory and time in the context size, or attention span. There are typically 12 layers with 8 heads each that processes 512 tokens simultaneously. This drastically limits the maximum attention span used in Transformers.
2.2 Adaptive attention span
Each attention head of a Transformer shares the same attention span S. This assumes that every head requires the same span to form its representation. As shown in Figure 1 , this assumption does not hold in the context of character level language modeling: some heads (e.g., Head A) focus on the recent history, while others take information from the whole available context (e.g., Head B). In this section, we propose to learn the attention span of each head independently to reduce their computational and memory cost.
x m z (x) 1 z z + R
For each head, we add a masking function to control for the span of the attention. A masking function is a non-increasing function that maps a distance to a value in [0, 1]. We take the following soft masking function m z parametrized by a real value z in [0, S]:
m z (x) = min max 1 R (R + z -x) , 0 , 1 ,
where R is a hyper-parameter that controls its softness. This soft masking function is inspired by Jernite et al. ( 2017). In Figure 2 , we show the shape of this piecewise function as a function of the distance. The attention weights from Eq. 2 are then computed on the masked span, i.e.,
a tr = m z (t -r) exp (s tr ) t-1 q=t-S m z (t -q) exp (s tq )
.
We add a 1 penalization on the parameters z i for each attention head i of the model to the loss function:
L = -log P (w 1 , . . . , w T ) + λ M i z i ,
where λ > 0 is the regularization hyperparameter, and M is the number of heads in each layer. Our formulation is differentiable in the parameters z i and we learn them jointly with the rest of the model.
Dynamic attention span.
As an extension, we consider a dynamic computation approach (Graves, 2016) where the attention span dynamically change based on the current input (Luong et al., 2015; Shu and Nakayama, 2017) . At a time step t, the span parameter z t of an attention head is then a function of the input parametrized by a vector v and a scalar b, i.e., z t = Sσ(v T x t + b). We penalize z t in the same way as before and learn the parameters v, b jointly with the rest of the parameters.
3 Experiments
In this section, we evaluate the impact of our adaptive attention mechanism in the experimental setting of Al-Rfou et al. ( 2019) for character level language modeling.
Dataset. We use the text8 and enwik8 datasets of Mahoney (2011) . The both dataset have 100M tokens. We report bit per character (bpc) on dev and test set.
Implementation details. We experiment with two sizes of models. Our small models have 12 layers and a hidden size of d h = 512, except for the feedforward ReLU layers, which have 2048 units. The large models have 24 layers with a hidden size of d h = 768, and a ReLU size of 4096. All models have 8 attention heads in each layer. Token and position embedding parameters are initialized from N (0, 1), and the projection matrices W {q,k,v,o} are initialized from
U(-1/ √ d h , 1/ √ d h ).
A single set of position embeddings p t is shared across all the heads.
In adaptive-span models, we reprameterized the span parameter z by z = Sz , where z ∈ [0, 1] is initialized to 0. In dynamic-span models, the bias term b is initialized -4 to make initial spans small. We set the hyperparameters λ = 2 × 10 -6 and R = 32 for the both type of models, except λ is reduced to 0.5 × 10 -6 when S = 8192 because z was not growing longer than 4000.
We use Adagrad with a batch size of 64 and fixed learning rate of 0.07 and 32k warm-up steps. Our warm-up strategy differs from Vaswani et al. (2017) : we linearly increase learning rate from zero to the final learning rate. Gradients of each module are clipped at 0.03 for better stability. At train time, we use a block of 512 consecutive characters and compute the loss and gradient for each of those 512 characters.
In small models, we apply dropout with a rate of 0.3 to the attention and the feedforward ReLU activations. We train small models for 600K steps (900K steps when S = 8192), which takes about 2 ∼ 3 days on 8 V100 GPUs depending on the attention span limit. Large models are trained with a dropout rate of 0.4 until the validation performance stopped improving (250K steps for text8 and 150K steps for enwik8), and then further trained for 20K steps with a learning rate divided by 10.
Results.
In Table 1 , we compare our sequential Transformer with the adaptive spans ("Adaptive-Span") of Sec. 2.2 to models of Al-Rfou et al. (2019) and Dai et al. (2019) . For small models, our model outperforms the other Transformers by 0.07 bcp while significantly reducing the memory usage for large attention span. Interestingly, even with a limit on span sets to 8192, the average span is only 314. Similar results are obtained on enwik8 as shown in Table 2 , where the adaptive-span model outperformed similar sized models with a significantly smaller average span. Our large models achieved state-of-the-art performances on both datasets with fewer parameters and FLOPS.
In Figure 3 , we compare the fixed and adaptive span small Transformers as we increase the attention span limit S. The performance of both models improve as the limit increase (see Figure 3 (left)), but the adaptive-span model benefits more from longer span. As shown on the Figure 3 (center), a Transformer with adaptive spans controls its average spans, leading to reduction of up to 70% in the number of FLOPS for the inference with large spans (see Figure 3(right) ).
Impact on the attention span. In Figure 4 , we show the final attention spans of every attention heads of our small adaptive-span model with S = 4096. Even though all the span sizes are initialized to the same value, we see large varieties in their final values. We can see that the lowest 5 layers have the smallest possible attention span, which is R = 32 of the masking function. This indicates that lower layers in a Transformer model do not really require a long attention span in this particular task. In contrast, few attention heads in the higher layers have very long spans, exceeding several thousand. Although there is a general tendency of higher layers having longer attention spans, it is not a simple monotonic function of the layer height.
Impact on the number of FLOPS. Having a smaller attention span has a direct impact on the total number of FLOPS necessary for computing one-step prediction. In a standard fixed-span Table 1 : Character level language modeling on text8. We report bpc for the dev and test sets, as well as, the number of parameters, the average attention spans and total number of FLOPS (an estimate of the number of FLOPS necessary for computing one step prediction). model, the total number of FLOPS is mostly controlled by the feed-forward layer (accounting for 62% of FLOPS when S = 256). However, as the span increase, the attention layer dominates the computation (82% of FLOPS when S = 8192), making it hard to scale to longer sequences. In contrast, the learning of an attention span keeps computation at a relatively constant level even as S increase as shown in Figure 3(right) .
The memory usage is also dominated by the attention layer as the attention span increase. Thus, reducing the average span will also reduce the memory usage. However, because all heads in a single layer attend to common state vectors, the maximum span within each layer will determine the memory usage. The same is true for the number of FLOPS if all heads of a layer are computed together, as often done for better efficiency.
In practice, the largest fixed-span model that can fit in memory for training had a span of S = 2048 (batches had to be split when S = 4096), and it took about 550ms per batch. In contrast, an adaptive-span model with a 4 times longer span of S = 8192 fit in memory and took about similar time per batch.
Dynamic span. In Table 3 , we show the adaptive and dynamic spans achieved the same performance with comparable average spans on text8.
Figure 5 shows how the average dynamic span adapts to the input sequence. The span increases at the beginning of words and in the middle of composed words, e.g., to predict the "l" in "overlook".
4 Conclusion
In this work, we present a novel self-attention layer with an adaptive span. This mechanism allows for models with longer context, and thus with the capability to catch longer dependencies. We have shown the importantce of this feature in the context of character level modeling where information is spread over great distances.
https://github.com/facebookresearch/ adaptive-span
References
[1]Al-Rfou Rami, Choe Dokook, Constant Noah, Guo Mandy, Jones Llion. Character-level language modeling with deeper self-attention. Proceedings of the 33rd AAAI Conference on Artificial Intelligence 2019;:.
[2]Bahdanau Dzmitry, Cho Kyunghyun, Bengio Yoshua. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations 2015;:.
[3]Dai Zihang, Yang Zhilin, Yang Yiming, Carbonell JaimeG, Le QuocV, Salakhutdinov Ruslan. Transformer-xl: Attentive language models beyond a fixed-length context.  2019;:.
[4]Graves Alex. Adaptive computation time for recurrent neural networks.  2016;:.
[5]Jernite Yacine, Grave Edouard, Joulin Armand, Mikolov Tomas. Variable computation in recurrent neural networks.  2017;:.
[6]Luong Thang, Pham Hieu, Manning ChristopherD. Effective approaches to attention-based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing 2015;:.
[7]Mahoney Matt. Large text compression benchmark.  2011;:.
[8]Shaw Peter, Uszkoreit Jakob, Vaswani Ashish. Self-attention with relative position representations. Proceedings of the 2018 Conference of the North American Chapter 2018;:.
[9]Shu Raphael, Nakayama Hideki. An empirical study of adequate vision span for attention-based neural machine translation. Proceedings of the First Workshop on Neural Machine Translation 2017;:.
[10]Sukhbaatar Sainbayar, Szlam Arthur, Weston Jason, Fergus Rob. End-to-end memory networks. Advances in Neural Information Processing Systems 2015;28:.
[11]Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez AidanN, et al.. Attention is all you need. Advances in Neural Information Processing Systems 2017;30:.
Adaptive Attention Span in Transformers
https://arxiv.org/abs/1905.07799
https://proceedings.mlr.press/v119/katharopoulos20a.html
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
2020-11-21
Angelos Katharopoulos; Apoorv Vyas; Nikolaos Pappas; François Fleuret
<div><article>
 
<p><i>Proceedings of the 37th International Conference on Machine Learning</i>, PMLR 119:5156-5165, 2020.
 </p> 
 
 <h4>Abstract</h4>
 <p>
 Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
 </p>
 <h4>Cite this Paper</h4>
<hr />
<div><p> BibTeX</p> <p><code>
@InProceedings{pmlr-v119-katharopoulos20a,
 title = 	 {Transformers are {RNN}s: Fast Autoregressive Transformers with Linear Attention},
 author = {Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran{\c{c}}ois},
 booktitle = 	 {Proceedings of the 37th International Conference on Machine Learning},
 pages = 	 {5156--5165},
 year = 	 {2020},
 editor = 	 {III, Hal Daumé and Singh, Aarti},
 volume = 	 {119},
 series = 	 {Proceedings of Machine Learning Research},
 month = 	 {13--18 Jul},
 publisher = {PMLR},
 pdf = 	 {http://proceedings.mlr.press/v119/katharopoulos20a/katharopoulos20a.pdf},
 url = 	 {https://proceedings.mlr.press/v119/katharopoulos20a.html},
 abstract = 	 {Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.}
}
</code></p>
</div>
<div><p> Endnote</p> <p><code>%0 Conference Paper
%T Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
%A Angelos Katharopoulos
%A Apoorv Vyas
%A Nikolaos Pappas
%A François Fleuret
%B Proceedings of the 37th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2020
%E Hal Daumé III
%E Aarti Singh	
%F pmlr-v119-katharopoulos20a
%I PMLR
%P 5156--5165
%U https://proceedings.mlr.press/v119/katharopoulos20a.html
%V 119
%X Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
</code></p>
</div>
<div><p> APA</p> <p><code>
Katharopoulos, A., Vyas, A., Pappas, N. &amp; Fleuret, F.. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:5156-5165 Available from https://proceedings.mlr.press/v119/katharopoulos20a.html.
</code></p>
</div>
<hr />
 
 <h4>Related Material</h4>
 
 
</article>||||I|||| Proceedings of Machine Learning Research
 Volume 119JMLR MLOSS FAQ Submission Format

 [edit]

 Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

 Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret
 Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5156-5165, 2020.

 Abstract

 Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

 Cite this Paper

 BibTeX
 @InProceedings{pmlr-v119-katharopoulos20a, title = {Transformers are {RNN}s: Fast Autoregressive Transformers with Linear Attention}, author = {Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran{\c{c}}ois}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5156--5165}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/katharopoulos20a/katharopoulos20a.pdf}, url = {https://proceedings.mlr.press/v119/katharopoulos20a.html}, abstract = {Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.} }
 Copy to ClipboardDownload
 Endnote
 %0 Conference Paper %T Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention %A Angelos Katharopoulos %A Apoorv Vyas %A Nikolaos Pappas %A François Fleuret %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-katharopoulos20a %I PMLR %P 5156--5165 %U https://proceedings.mlr.press/v119/katharopoulos20a.html %V 119 %X Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
 Copy to ClipboardDownload
 APA
 Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F.. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5156-5165 Available from https://proceedings.mlr.press/v119/katharopoulos20a.html.
 Copy to ClipboardDownload

 Related Material

 * Download PDF
 * Software
 * Supplementary PDF

 This site last compiled Sun, 06 Nov 2022 07:33:10 +0000

 Github Account
 Copyright © The authors and PMLR 2022. MLResearchPress
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5156-5165, 2020.
  
 
 Abstract
 
 Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
 
 Cite this Paper
  BibTeX  
@InProceedings{pmlr-v119-katharopoulos20a,
 title = 	 {Transformers are {RNN}s: Fast Autoregressive Transformers with Linear Attention},
 author = {Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran{\c{c}}ois},
 booktitle = 	 {Proceedings of the 37th International Conference on Machine Learning},
 pages = 	 {5156--5165},
 year = 	 {2020},
 editor = 	 {III, Hal Daumé and Singh, Aarti},
 volume = 	 {119},
 series = 	 {Proceedings of Machine Learning Research},
 month = 	 {13--18 Jul},
 publisher = {PMLR},
 pdf = 	 {http://proceedings.mlr.press/v119/katharopoulos20a/katharopoulos20a.pdf},
 url = 	 {https://proceedings.mlr.press/v119/katharopoulos20a.html},
 abstract = 	 {Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.}
}
 
  Endnote  %0 Conference Paper
%T Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
%A Angelos Katharopoulos
%A Apoorv Vyas
%A Nikolaos Pappas
%A François Fleuret
%B Proceedings of the 37th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2020
%E Hal Daumé III
%E Aarti Singh	
%F pmlr-v119-katharopoulos20a
%I PMLR
%P 5156--5165
%U https://proceedings.mlr.press/v119/katharopoulos20a.html
%V 119
%X Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\bigO{N^2}$ to $\bigO{N}$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our \emph{Linear Transformers} achieve similar performance to vanilla Transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
 
  APA  
Katharopoulos, A., Vyas, A., Pappas, N. &amp; Fleuret, F.. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5156-5165 Available from https://proceedings.mlr.press/v119/katharopoulos20a.html.
 
 
 Related Material
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
https://proceedings.mlr.press/v119/katharopoulos20a.html
https://proceedings.neurips.cc/paper/2020/hash/f6a8dd1c954c8506aadc764cc32b895e-Abstract.html
Fast Transformers with Clustered Attention
2020-01-01
Vyas; Apoorv; Katharopoulos; Angelos; Fleuret; François
<div><div>
<h4>Fast Transformers with Clustered Attention</h4>
<p>
 Part of
 <a href="https://proceedings.neurips.cc/paper_files/paper/2020">Advances in Neural Information Processing Systems 33 (NeurIPS 2020)
 </a>
</p>
<br/>
<h4>Authors</h4>
<p><i>Apoorv Vyas, Angelos Katharopoulos, François Fleuret</i></p>
<h4>Abstract</h4>
<p>Transformers have been proven a successful model for a variety of tasks in
sequence modeling. However, computing the attention matrix, which is their key
component, has quadratic complexity with respect to the sequence length, thus
making them prohibitively expensive for large sequences. To address this, we
propose clustered attention, which instead of computing the attention
for every query, groups queries into clusters and computes attention just for
the centroids. To further improve this approximation, we use the computed
clusters to identify the keys with the highest attention per query and compute
the exact key/query dot products. This results in a model with linear
complexity with respect to the sequence length for a fixed number of clusters.
We evaluate our approach on two automatic speech recognition datasets and show
that our model consistently outperforms vanilla transformers for a given
computational budget. Finally, we demonstrate that our model can approximate
arbitrarily complex attention distributions with a minimal number of clusters
by approximating a pretrained BERT model on GLUE and SQuAD benchmarks with only
25 clusters and no loss in performance.</p>
</div><div>
<p>Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.</p>
<p>Use the "Report an Issue" link to request a name change.</p>
</div><p>Do not remove: This comment is monitored to verify that the site is working properly</p></div>||||I|||| NeurIPS Proceedings
 * 
 * 
 Search


 Fast Transformers with Clustered Attention

 Part of Advances in Neural Information Processing Systems 33 (NeurIPS 2020)

 AuthorFeedback Bibtex MetaReview Paper Review Supplemental


 Authors

 Apoorv Vyas, Angelos Katharopoulos, François Fleuret

 Abstract

 Transformers have been proven a successful model for a variety of tasks in sequence modeling. However, computing the attention matrix, which is their key component, has quadratic complexity with respect to the sequence length, thus making them prohibitively expensive for large sequences. To address this, we propose clustered attention, which instead of computing the attention for every query, groups queries into clusters and computes attention just for the centroids. To further improve this approximation, we use the computed clusters to identify the keys with the highest attention per query and compute the exact key/query dot products. This results in a model with linear complexity with respect to the sequence length for a fixed number of clusters. We evaluate our approach on two automatic speech recognition datasets and show that our model consistently outperforms vanilla transformers for a given computational budget. Finally, we demonstrate that our model can approximate arbitrarily complex attention distributions with a minimal number of clusters by approximating a pretrained BERT model on GLUE and SQuAD benchmarks with only 25 clusters and no loss in performance.


 Name Change Policy

 ×

 Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.

 Use the "Report an Issue" link to request a name change.

 Report an Issue | Name Change Policy
 Do not remove: This comment is monitored to verify that the site is working properly
Fast Transformers with Clustered Attention
 Part of
 Advances in Neural Information Processing Systems 33 (NeurIPS 2020)
 
Authors
 Apoorv Vyas, Angelos Katharopoulos, François Fleuret 
Abstract
Transformers have been proven a successful model for a variety of tasks in
sequence modeling. However, computing the attention matrix, which is their key
component, has quadratic complexity with respect to the sequence length, thus
making them prohibitively expensive for large sequences. To address this, we
propose clustered attention, which instead of computing the attention
for every query, groups queries into clusters and computes attention just for
the centroids. To further improve this approximation, we use the computed
clusters to identify the keys with the highest attention per query and compute
the exact key/query dot products. This results in a model with linear
complexity with respect to the sequence length for a fixed number of clusters.
We evaluate our approach on two automatic speech recognition datasets and show
that our model consistently outperforms vanilla transformers for a given
computational budget. Finally, we demonstrate that our model can approximate
arbitrarily complex attention distributions with a minimal number of clusters
by approximating a pretrained BERT model on GLUE and SQuAD benchmarks with only
25 clusters and no loss in performance.
 
Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.
Use the "Report an Issue" link to request a name change.
 Do not remove: This comment is monitored to verify that the site is working properly
Fast Transformers with Clustered Attention
https://proceedings.neurips.cc/paper/2020/hash/f6a8dd1c954c8506aadc764cc32b895e-Abstract.html
https://arxiv.org/abs/2004.10102
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
2020-04-21
Kobayashi; Goro; Kuribayashi; Tatsuki; Yokoi; Sho; Inui; Kentaro
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2004.10102">Download PDF</a></p><blockquote>
 Abstract: Attention is a key component of Transformers, which have recently achieved
considerable success in natural language processing. Hence, attention is being
extensively studied to investigate various linguistic capabilities of
Transformers, focusing on analyzing the parallels between attention weights and
specific linguistic phenomena. This paper shows that attention weights alone
are only one of the two factors that determine the output of attention and
proposes a norm-based analysis that incorporates the second factor, the norm of
the transformed input vectors. The findings of our norm-based analyses of BERT
and a Transformer-based neural machine translation system include the
following: (i) contrary to previous studies, BERT pays poor attention to
special tokens, and (ii) reasonable word alignment can be extracted from
attention mechanisms of Transformer. These findings provide insights into the
inner workings of Transformers.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Goro Kobayashi [<a href="/show-email/44a93fab/2004.10102">view email</a>]
 <br />
 <strong><a href="/abs/2004.10102v1">[v1]</a></strong>
 Tue, 21 Apr 2020 15:22:27 UTC (6,955 KB)<br /><strong>[v2]</strong>
Tue, 6 Oct 2020 15:15:38 UTC (6,691 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2004.10102

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:2004.10102 (cs)
 [Submitted on 21 Apr 2020 (v1), last revised 6 Oct 2020 (this version, v2)]

 Title: Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

 Authors: Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro Inui
 Download PDF
 Abstract: Attention is a key component of Transformers, which have recently achieved considerable success in natural language processing. Hence, attention is being extensively studied to investigate various linguistic capabilities of Transformers, focusing on analyzing the parallels between attention weights and specific linguistic phenomena. This paper shows that attention weights alone are only one of the two factors that determine the output of attention and proposes a norm-based analysis that incorporates the second factor, the norm of the transformed input vectors. The findings of our norm-based analyses of BERT and a Transformer-based neural machine translation system include the following: (i) contrary to previous studies, BERT pays poor attention to special tokens, and (ii) reasonable word alignment can be extracted from attention mechanisms of Transformer. These findings provide insights into the inner workings of Transformers.
 Comments: 19 pages, accepted by EMNLP 2020 
 Subjects: Computation and Language (cs.CL) 
 Cite as: arXiv:2004.10102 [cs.CL] 
 (or arXiv:2004.10102v2 [cs.CL] for this version)
 https://doi.org/10.48550/arXiv.2004.10102 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Goro Kobayashi [view email]
 [v1] Tue, 21 Apr 2020 15:22:27 UTC (6,955 KB)
 [v2] Tue, 6 Oct 2020 15:15:38 UTC (6,691 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 2004
 Change to browse by:
 cs

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Sho Yokoi
 Kentaro Inui
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Attention Module is Not Only a Weight: Analyzing Transformers with Vector Norms
Goro Kobayashi 
Tohoku University
Tatsuki Kuribayashi kuribayashi@ecei.tohoku.ac.jp 
Tohoku University
Langsmith Inc. 3 RIKEN
Sho Yokoi yokoi@ecei.tohoku.ac.jp 
Tohoku University
Kentaro Inui inui@ecei.tohoku.ac.jp 
Tohoku University
Attention Module is Not Only a Weight: Analyzing Transformers with Vector Norms
Because attention modules are core components of Transformer-based models that have recently achieved considerable success in natural language processing, the community has a great deal of interest in why attention modules are successful and what kind of linguistic information they capture. In particular, previous studies have mainly analyzed attention weights to see how much information the attention modules gather from each input to produce an output. In this study, we point out that attention weights alone are only one of the two factors determining the output of selfattention modules, and we propose to incorporate the other factor as well, namely, the transformed input vectors into the analysis. That is, we measure the norm of the weighted vectors as the contribution of each input to an output. Our analysis of self-attention modules in BERT and the Transformer-based neural machine translation system shows that the attention modules behave very intuitively, contrary to previous findings. That is, our analysis reveals that (1) BERT's attention modules do not pay so much attention to special tokens, and (2) Transformer's attention modules capture word alignment quite well.
Introduction
Transformer-based models have improved the state-of-the-art in a wide range of natural language processing (NLP) tasks (Vaswani et al., 2017;Devlin et al., 2019;Yang et al., 2019;Lan et al., 2020). As the success of these models has not yet been sufficiently explained, a substantial body of research has focused on their linguistic capabilities (Rogers et al., 2020). Since attention modules (Section 2) are the core components of Transformer-based models, one prominent line of research has been the analysis of correlations of attention weights with various linguistic phenomena (Clark et al., 2019;Kovaleva et al., 2019;Reif et al., 2019;Lin et al., 2019;Mareček and Rosa, 2019;Htut et al., 2019;Raganato and Tiedemann, 2018;Tang et al., 2018). However, the community has not reached a consensus about how and why attention modules work well.
To go a step further in this line of research, in this paper, we propose a novel method of analyzing the behavior of an attention module. The previous approach that tracks attention weights is based on the assumption that if an input vector gains considerable attention weight than the other input vectors, that input vector has a greater contribution to the output. In contrast, we start with the insight that the computation of an output of the attention module can be decomposed into a weighted sum of the transformed input vectors, and point out that analyzing only attention weights is insufficient to investigate the behavior of an entire attention module (Section 2). We then propose an analysis method with the norm (length) of the weighted vectors that considers the previously ignored factor as well, namely, the transformed vectors, to estimate the contribution of each input vector in the computation of the output vector (Section 3).
We first analyze the self-attention modules in BERT (Devlin et al., 2019) in our experiments (Section 4). We empirically show that the figures of the contributions of input tokens to their output hidden vectors are considerably different from what was reported in previous work. In particular, some previous reports showed several non-intuitive observations; for example, specific tokens such as commas, periods, and special tokens (e.g., separator token; [SEP]) tend to gain markedly large attention weights (Clark et al., 2019, etc.). Our experiments demonstrate that, unlike the previous attention weight-tracking approach, our norm-based method produces more intuitive observations. Finally, we analyze the source-target attention modules in the Transformer-based neural machine translation (NMT) system (Vaswani et al., 2017) (Section 5). The properties of the NMT systems observed by using our norm-based method are markedly different from previous results -their attention modules implicitly learn word alignment in contrast to previous findings from the attention weights analysis.
The contributions of this study are as follows:
• We propose a method of analyzing an attention module based on vector norms, that considers the previously ignored factor as well as attention weights. • We analyzed the self-attention modules in BERT using our norm-based method and found that (1) the dispersion of norms of the previously ignored vectors is large, (2) the attention module does not pay much attention to special tokens, and (3) the attention module tends to discount frequent words. • We analyzed the source-target attention modules in a Transformer-based NMT system and found that, contrary to previous findings, the attention module implicitly performs word alignment well.
Background
Attention module
We briefly review the attention module 1 implemented in Transformer (Vaswani et al., 2017). An overview of the attention module is shown in Figure 1. In the attention module, the output vector is calculated using the weighted sum of the input vectors, the weights, and the bias. Intuitively, when the weight assigned to a certain vector (the blue circles in Figure 1) is large, the module pays much attention to the input vector, whereas a vector with a smaller weight is paid less attention.
Here and subsequently, boldface letters such as x denote row vectors. Formally, the attention module computes each output vector y i ∈ R d from the corresponding pre-update vector y i ∈ R d and a set of input vectors X = {x 1 , . . . x n } ⊆ R d 1 The term attention module in this paper corresponds to a submodule that divided the concept called multi-head attention module in Vaswani et al. (2017)   to be incorporated:
y i =   n j=1 α i,j v(x j )   W O + b O ∈ R d (1) α i,j := softmax x j ∈X q( y i )k(x j ) √ d ∈ R,(2)
where q(·), k(·), and v(·) are the Query, Key, and Value vectors, respectively.
q( y i ) := y i W Q + b Q W Q ∈ R d×d , b Q ∈ R d k(xj) := xjW K + b K W K ∈ R d×d , b K ∈ R d v(xj) := xjW V + b V W V ∈ R d×d , b V ∈ R d .
That is, the attention weight α i,j is firstly computed, then the Value vector v(x j ) is weighted by α i,j , the weighted Value vectors are summed, and finally, the affine transformation by W O ∈ R d ×d and b O ∈ R d is applied (see Figure 1). In the case of y i ∈ X , this module is called the Self-attention module, which is implemented in, e.g., BERT. In the attention module between the encoder and the decoder of the Transformer sequence-to-sequence model, X corresponds to the representations from the encoder stack, and vectors y i and y i correspond to the i-th token in the decoder side.
Attention module sums weighted vectors
With a simple reformulation, we can see that the attention module computes a weighted sum of input vectors. Using the linearity of the matrix product, we can rewrite Equation 1 as 
! " # $ % ! " # $ % !,# # !,$ $ !,% % !,& & !,' ' &,% &," &,# &,y i = n j=1 α i,j f (x j ) + b O (3) f (x) := xW V + b V W O .(4)
Equation 3 shows that the attention module firstly transforms each input vector x to generate f (x) and computes attention weights α , then sums αf (x) with a bias b O (see Figure 2).
Problems with attention weight analysis
The attention module is designed to update representations by gathering relevant information from the input sequence. Therefore, in prior studies, it has been investigated how much information from each input is used to generate an output by analyzing the magnitude of attention weights (Clark et al., 2019;Kovaleva et al., 2019;Reif et al., 2019;Lin et al., 2019;Mareček and Rosa, 2019;Htut et al., 2019;Raganato and Tiedemann, 2018;Tang et al., 2018). Analyses in this line of research are based on the assumption that if an input vector is assigned a larger attention weight than other input vectors, then that input vector contributes more to the output vector than the others. However, this assumption disregards the magnitude of the vectors to be weighted. Intuitively, with attention weights being equal, a larger vector will contribute more to the output vector than a smaller vector.
The problem of neglecting the effect of f (x j ) is illustrated in Figure 2. Here, the transformed vec-tor f (x 1 ) for input x 1 is assumed to be very small ( f (x 1 ) ≈ 0) while it gains considerably greater attention weight α i,1 than other input vectors. In such a case, even if the contribution of input x 1 to the output vector, say, y i is very small, looking at the attention weight alone might lead to a wrong observation that this input vector x 1 had the greatest contribution to output y i .
In fact, in previous studies, analysis based on attention weights has produced some non-intuitive observations, probably due to this flaw. For example, Clark et al. (2019) reported that input vectors for specific tokens such as commas, periods, and separator tokens [SEP] tend to gain remarkably large attention weights, whereas intuitively, their contributions to the pre-training tasks (i.e., masked word prediction and next sentence prediction) are expected to be limited compared with more informative content words.
3 Proposal: norm as attention degree
As described in Section 2.3, in prior work, only α i,j has been considered in Equation 3 but the effect of f (x j ) is neglected. Our main proposal in this paper is to estimate the contribution of the input vector x j to the output vector y i by α i,j f (x j ) instead. As we will show, this difference does matter in the exploration of the behavior of attention modules.
To address the aforementioned issue, we propose to use αf (x) , which is the standard Euclidean norm (length) of the weighted, transformed vector in Equation 3, as a novel method for analyzing the behavior of the attention module. Unlike in previous studies in which attention weights alone were observed, we analyze the behavior of αf (x) and f (x) as well as α to gain a more in-depth view of how the attention modules work. We call this proposed way of analyzing the attention module the norm-based analysis in contrast to the previous weight-based analysis.
In the experiments section (Section 4 and 5), we empirically show that by using norm-based analysis instead of weight-based analysis, one can gain more intuitive and meaningful observations on the behavior of the attention modules. It is worth noting that our norm-based analysis can be naturally extended to the multi-head attention module implemented in the Transformer, BERT, and others because the multi-head attention module is a natural extension of the attention module and also it  is linearly decomposable; i.e., attention heads (attention modules) are arranged in parallel and the output of the multi-head attention module is the sum of the output vectors of all the heads.
Experiments1: BERT
Overview: First, we preliminary show that previously ignored effect, namely, the norm of the transformed vectors, could affect the results in analyzing the attention modules in BERT (Section 4.1). Second, by using our norm-based analysis, we re-examine the reports on BERT in the previous weight-based analysis (Section 4.2). Finally, we introduce the previously overlooked property of BERT (Section 4.3).
General settings: Following previous studies (Clark et 
Does f (x) have an impact?
As a preliminary demonstration of how much αf (x) differs from α, we analyze the coefficient of variation 4 of f (x) . Based on Equation 3, if f (x) is almost constant, the attention weights α can be a good approximation of 2 We used PyTorch implementation of BERT-base released at https://github.com/huggingface/ transformers. 3 https://github.com/clarkkev/ attention-analysis 4 Coefficient of variation (CV) is a standardized (scaleinvariant) measure of dispersion, which is defined by the ratio of the standard deviation σ to the mean µ: CV := σ/µ.
Avg.
Layer (a) Weight-based analysis.
Avg.
Layer
(b) Norm-based analysis. Figure 3: Each point corresponds to averaged α or αf (x) on a token category in a given layer. Therefore, in each layer, the sum of α on all categories is 1. The x-axis denotes the index of the layers.
αf (x) . That is, our norm-based analysis is not worth applying if f (x) is constant.
All the data were first fed into the model; then, we computed the coefficient of variation of f (x) . Table 1 shows that the coefficient of variation is 0.22 on average; the value of f (x) often varies from 0.78 times to 1.22 times the mean value. Thus, there is a difference between α and αf (x) owing to this dispersion of f (x) , which motivates to take into account f (x) in analyzing the attention modules. Appendix A shows the effect of x and the affine transformation f .
Re-examining previously observed phenomena
In this section, by using our norm-based analysis, we reinvestigate the observations of BERT previously reported with the weight-based analysis (Clark et al., 2019).
Settings: All the data were fed into BERT; then, α and αf (x) were collected from each head.   shows that the vectors for specific tokens ([CLS], [SEP], and punctuation) gain remarkably large attention weights. However, this observation is not intuitive because those tokens are expected not to have so linguistic information and not to be so useful for pre-training tasks (i.e., masked word prediction and next sentence prediction). However, this trend is indeed consistent with the report by the weight-based analysis (Clark et al., 2019). In contrast, our norm-based method produces an intuitive observation; the contributions of those vectors for specific tokens are generally small (Figure 3b). Clark et al. (2019) hypothesized that if there is no necessary information in the input vectors, BERT assigns large weights to [SEP] tokens that guaranteed to appear in any input sequence, not to incorporate any additional information via attention modules. Note that the attention module has the constraint that the sum of the attention weights becomes 1.0 (see Equation 2). Clark et al. (2019) called this operation "no-op" (no operation). This hypothesis is convincing; however, it remains unclear how the strong attention weights assigned to such special tokens could collect nothing. Our norm-based analysis suggests that the amount is actually small (Figure 3b), which supports the interpretation of BERT conducting "no-op".
Analysis -relationship between α and f (x) :
How is "no-op" realized? Here, we show an interesting trend that α and f (x) cancel each other out on specific tokens. Table 2 shows the Spearman rank correlation coefficient between α and f (x) assigned to the vectors in each category. α and f (x) corresponding to [CLS], [SEP], periods, and commas are highly inversely correlated. This achieves the "no-op" function of collecting no information from input tokens. More specifically, Figure 4 illustrates the contrast between α and f (x) corresponding to [SEP]    periods, and commas. On the other hand, there was no strong trend on the other tokens (see Appendix C).
More generally, Figure 5 shows 1% randomly selected pairs of α and f (x) in each token category. Even when the same attention weights α are assigned, the values of f (x) can vary, which suggests that they play a different role in the modules.
Relationship between token frequency and f (x)
In the previous section, we demonstrated that f (x) corresponding to the specific tokens (e.g., [SEP]) is small. Based on the presumption that such special tokens have relatively high frequency, we hypothesize that BERT controls the amount of contribution of highly frequent, less informative tokens by adjusting the norm of f (x).
Settings: First, all the data were fed into the model. Then, for each input vector x (r,o) in the data, assigned α (r,o) and f (x (r,o) ) were collected. Here, the vector denoted by x (r,o) is the vectors corresponding to the o-th occurrence of the r-th frequent (rank r) type (subword) 5 . Each α (r,o)6 and f (x (r,o) ) denote the averaged score of the values obtained from all the 144 heads. We analyzed the relationships between the rank of frequency r and α (r,o) , and that between r and f (x (r,o) ) .
Results: Figure 6 shows the results of the experiments. Each plot corresponds to (r, α (r,o) ) or (r, f (x (r,o) ) ). The Spearman rank correlation coefficient showed no correlation (ρ = −0.08) between r and α (r,o) (Figure 6a). On the other hand, the Spearman rank correlation coefficient between r and f (x (r,o) ) was 0.75, indicating a strong positive correlation (Figure 6b). These results demonstrate that the self-attention modules in BERT reduce the information from highly frequent tokens by adjusting f (x) but not α. This frequency-based effect is consistent with the intuition that highly frequent tokens such as stop words are unlikely to play an important role in solving the pre-training tasks (masked token prediction and next sentence prediction). Note that the remarkable behavior in detail might be ignored since α and f (x) were averaged for all the heads in our experiments. Thus, a more detailed head-level or layer-level analysis is needed in future work.
Experiments2: Transformer-based NMT system
We additionally analyze the source-target attention modules in Transformer-based NMT systems. One of the community's interests in NMT is whether the NMT systems capture word alignment between the source text and the target text, and if so, how to extract it from the black box NMT systems. This is an important issue for interpreting the internal on-line translation process of the systems, conducting detailed error analysis, and showing rich information (which words are translated to which words) for the users of NMT systems (Ding et al., 2017). 5 We used the token ids used in BERT's tokenizer to acquire each token's frequency rank in the training data of BERT. Special tokens and single characters were excluded from them as we cannot get their exact frequency rank. We considered commas, periods, [SEP], and [CLS] as the top four frequent tokens because they are considered to appear very frequently. 6 First, attention weights assigned to x (r,o) for generating each output vector in a sequence are averaged in each head. Then, we calculated the averaged score over the results of 144 heads.  Previously, several studies showed that the attention weights in the Transformer's source-target attention module do not induce word alignment so well (Zenkel et al., 2019;Li et al., 2019;Ding et al., 2019). In this section, we demonstrate that much better word alignments can be extracted from a Transformer-based NMT system simply by applying our norm-based method, i.e., taking into account the norm of the transformed vectors as well. Soft alignments were extracted from the attention module by the following methods:
Settings
• Attention-weights for each layer were computed by averaging α of all heads following Li et al. (2019). • For our norm-based method, we merged αf (x) from all attention heads in each layer by the following strategy: adding all the vec-tors αf (x) from every head, then calculating the norm of the summed vector (Vector-norms). Adding all αf (x) from every head is the same as the procedure that combines the results from every head into the results of the multi-head attention module. Next, soft Byte-Pair Encoding (Sennrich et al., 2016) alignments extracted from the methods above converted to hard word alignment following the procedure in Li et al. (2019). Their alignments were evaluated using the alignment error rate (AER) (Och and Ney, 2000) on the gold alignment data provided by Vilar et al. (2006) 8 .
We also include the AER results reported by Ding et al. (2019) and Zenkel et al. (2019), obtained using the gradient-based methods and word aligners: fast align (Dyer et al., 2013) and GIZA++ (Och and Ney, 2003). In the gradientbased methods, SmoothGrad (Smilkov et al., 2017) is applied to the NMT systems (Zenkel et al., 2019;Li et al., 2016).
Results: We obtained the following findings: (1) the results obtained using our norm-based method considerably differ from the observation through attention weights (Figure 7 and 8), (2) the normbased result is much better than existing worksour result is competitive with fast align, one of the de facto word aligners (Table 3), and (3) These results overturn the previous finding that the attention modules in Transformer do not perform word alignment (Li et al., 2019).
On the difference between norm-based and weight-based results: Figure 7 shows that, especially in the lower layer, the AER scores with our norm-based method significantly improve. The major reason for the difference between the scores obtained using the weight-based and the norm-based methods is probably the same mechanism described in Section 4.2 -while the attention weights assigned to certain tokens (e.g., periods) are large, their vector norms are adjusted to become smaller (Figure 8).
On the comparison of the extracted alignment quality: Our norm-based result is also better than the gradient-based results. While the gradient-based methods introduce some additional hyperparameters, such as the number of augmented samples n, our norm-based method does  not, which suggests that we succeed in extracting better alignments from the NMT model with more simple way. It is also worth noting that the extracted alignments using our norm-based method, which does not use any external modules, are as clean as those extracted by the existing aligners, fast align. This result suggests that the Transformer-based NMT system implicitly learned word alignment through the attention modules during training. Therefore, it overturns the previous finding that the attention modules in Transformer fail to capture word alignment (Li et al., 2019).
To sum up, these results suggest that our normbased method may change the view of our understanding of not only BERT but also other Transformer-based systems.  Table 3: AER scores with different alignment extraction methods on German-English translation. The results of the gradient-based approach are the scores reported by Ding et al. (2019). The results of fast align and GIZA++ are the scores reported by Zenkel et al. (2019) 6 Related work
Methods
AER
Transformer -Attention-based Approach
Existing studies investigating Transformer-based models have mainly analyzed the internal representations, outputs (e.g., the scores for predicting masked words in BERT), or attention weights. Studies focusing on internal representations or outputs showed that BERT captures, more or less, syntactic structures and semantic information (Goldberg, 2019;Hewitt and Manning, 2019;Reif et al., 2019;Tenney et al., 2019;Jawahar et al., 2019). The weight-based analyses have shown that BERT or Transformer has attention heads whose attention weight assignments correspond to specific linguistic relations (Clark et al., 2019;Kovaleva et al., 2019;Reif et al., 2019;Lin et al., 2019;Mareček and Rosa, 2019;Htut et al., 2019;Raganato and Tiedemann, 2018;Tang et al., 2018) and that the encoder-decoder attention modules in Transformer do not capture word alignment quite well (Zenkel et al., 2019;Li et al., 2019;Ding et al., 2019). Our study improved the method of analyzing the attention modules considering the effects of conventionally overlooked elements in addition to the attention weights. Clark et al. (2019) and Brunner et al. (2020) analyzed BERT by observing gradients. To estimate the contribution of each input vector (integral), both the gradient (derivative, sensitivity) and the range of integration are complementary and essential features. While the gradient-based analysis solely focuses on the former factor, our normbased analysis can consider both of them. Consid-ering both the gradient and the range of integration is similar to multiplying α (coefficient) by f (x) to compute the actual summed up vectors in our norm-based analysis.
As a similar study, Brunner et al. (2020) introduced "effective attention", which ignores certain elements in the attention weights matrix that are canceled out by the transformations in the attention module or the input vectors. While the effective attention approach considers the noninfluential elements as zero, our norm-based analysis takes into account the degree of the effects for the output vectors.
Conclusions and future work
In this paper, we have shown that attention weights alone are only one of two factors determining the output of self-attention modules, and we have proposed to incorporate the other factor as well, namely, the transformed input vectors, into the analysis. Through analysis of self-attention modules in BERT, we have shown that the proposed norm-based method produces insights that better agree with linguistic intuitions than an analysis based on attention weights alone. Our analysis has further revealed that BERT controls the amount of the contribution from highly frequent tokens, not by attention weights but via vector norms. Finally, our norm-based analysis showed that the Transformer-based NMT system implicitly learns word alignment much better than previously estimated by tracking the attention weights.
Future work: We believe that our norm-based analysis opens up many new directions for analyzing attention-based models. Our norm-based analysis is directly applicable to analyzing other attention-based models. For future work, we plan to apply it to the attention modules in, such as fine-tuned BERT and RoBERTa. Furthermore, we will extend the scope of analysis from the attention module to an entire architecture to better understand the inner workings and linguistic capabilities of the current powerful black-box NLP systems. Table 4: Mean (μ), standard deviation (σ), coefficient of variance (CV), and maximum and minimum values of x ; the former three are averaged on all the layers. Figure 9: Singular values of f at randomly selected heads in each layer. We use layer -head number to denote a particular attention head.
C Relationship between α and f (x)
corresponding to the other token categories
As described in Section 4.2, for vectors corresponding to [SEP], α and f (x) were canceled out in almost all heads. In this section, we show the trends of other vectors. Figures 10 and 11 show that the vectors corresponding to [CLS], periods and commas have the same tendency as the vectors corresponding to [SEP]. On the other hand, Figure 12 shows that the vectors corresponding to the other tokens do not strongly have the same tendency. Note that in the heatmaps of f (x) , the color scale is determined by the maximum value for all heads.
D Hyperparameters of NMT system
We used Transformer (Vaswani et al., 2017) NMT model implemented in fairseq  for the experiments in Section 5. Table 5 shows the hyperparameters of the model. We used the same data and word segmentation method as Li et al.     (a) Averaged α assigned to vectors corresponding to periods and commas.
(b) Averaged f (x) of vectors corresponding to periods and commas. Figure 11: α and f (x) corresponding to periods and commas, averaged on all the input text.
(a) Averaged α assigned to vectors corresponding to the other tokens.
(b) Averaged f (x) of vectors corresponding to the other tokens.      
Figure 1 :
1Overview of attention module based on Equation 1. The sizes of the colored circles illustrate the value of the scalar or the norm of the vector.
Following Clark et al. (2019), we report the results of the following categories: (i) [CLS], (ii) [SEP], (iii) periods and commas, and (iv) the other tokens.Results: The values of α and αf (x) showed completely different trends(Figure 3).Figure 3a
in each head. For almost all heads, α and f (x) clearly negate the magnitude of each other. A similar trend was observed on [CLS], (a) α.(b) f (x) .
Figure 4 :
4The darkness of each cell corresponds to the value of averaged α or f (x) on a [SEP] category in a given head.
Figure 5 :
5Relationship between α and f (x) . Each plot corresponds to a pair of α i,j and f (x j ) for output vector y i in either attention head. Each plot is colored by the token category corresponding to x j .
between frequency rank r and α (r,o) . !,# (b) Relationship between frequency rank r and f (x (r,o) ) .
Figure 6 :
6Relationship between frequency rank r and α(r,o)  or f (x (r,o) ) of each token in data. Each value is averaged of all the 144 heads.
:
Following Li et al. (2019), we trained the Transformer-based NMT system with the data used in the WMT2016 news translation task (DE-EN) 7 . The hyperparameters of the model are shown in Appendix D.
Figure 7 :
7AER scores of each layer in the Transformerbased system.(a) Reference.(b) Attention-weights.(c) Vector-norms (ours).
Figure 8 :
8Examples of soft alignment extracted from the attention modules in layer 2 of the system and reference of word alignment. Word pairs with a green frame are extracted as the alignments. Examples in the other layers are shown in Appendix E.
In
Section 5, alignment examples extracted from the attention modules in layer 2 were shown. Here, examples from the other layers are shown in Figure 13 to Figure 17).
Averaged f (x) of vectors corresponding to [CLS].
Figure 10 :
10α and f (x) corresponding to [CLS] token, averaged on all the input text.
Figure 12 :
12Comparison between assigned α and f (x) for vectors corresponding to the other tokens.
Figure 13 :
13Examples of soft alignment extracted from the attention modules in layer 1 of the model and reference of word alignment. Word pairs with a green frame are extracted as the alignments.(a) Attention-weights.(b) Vector-norms (ours).
Figure 14 :
14Examples of soft alignments extracted from layer 3. Word pairs with a green frame are extracted as the alignments.(a) Attention-weights.(b) Vector-norms (ours).
Figure 15 :
15Examples of soft alignments extracted from layer 4. Word pairs with a green frame are extracted as the alignments.(a) Attention-weights.(b) Vector-norms (ours).
Figure 16 :
16Examples of soft alignments extracted from layer 5. Word pairs with a green frame are extracted as the alignments.(a) Attention-weights.(b) Vector-norms (ours).
Figure 17 :
17Examples of soft alignments extracted from layer 6. Word pairs with a green frame are extracted as the alignments.
into the number of heads.!,# 
# 
!,$ 
$ 
!,% 
% 
!,& 
& 
!,' 
' 
Bias 
!,# 
!,$ 
!,% 
!,& 
Weighted 
Value vectors 
Attention 
weights 
Input vectors 
!,' 
! 
! 
! 
Pre-update 
vector 
Output vector 
Weights 
& 
$ 
% 
' 
# 
Value vectors 
" ! 
$ 
% 
& 
' 
( 
!Weighted 
vectors 
Attention 
weights 
Transformed 
vectors 
Input vectors 
&,$ 
Σ 
eq. (2) 
& 
# & 
Bias 
! 
Output vector 
Pre-update 
vector 
Summed vector 
!,' 
' 
Figure 2: Overview of attention module based on Equa-
tion 3. The module computes the output vector by 
summing weighted vectors, where vectors with larger 
norms can provide a more dominant contribution. The 
arrows in the circles represent the corresponding vec-
tors. The sizes of the colored circles illustrate the value 
of the scalar or the norm of the corresponding vector. 
Table 1 :
1Mean (µ), standard deviation (σ), coefficient 
of variance (CV), and maximum and minimum values 
of f (x) ; the former three are averaged on all the 
heads. 
al., 2019;Kovaleva et al., 2019;Reif et al., 2019;Lin et al., 2019;Htut et al., 2019), we analyzed the self-attention modules in the pretrained BERT-base (uncased), 2 which has 12 layers with 12 attention heads. We used the data provided byClark et al. (2019). 3 This data consists of 992 input sequences extracted from Wikipedia, each sequence consisting of two continuous paragraphs, in the form of: [CLS] paragraph1 [SEP] paragraph2[SEP]. Each sequence consists of up to 128 tokens, with an average of 122 tokens.
Table 2 :
2Spearman rank correlation coefficient between α and f (x) in each token category.
Table 5 :
5Hyperparameters of the NMT model.
http://www.statmt.org/wmt16/
https://www-i6.informatik.rwth-aachen. de/goldAlignment/
AcknowledgementThis work was supported by JSPS KAKENHI Grant Number JP19H04162.A Where the dispersion of f (x) comes fromAs described in Section 4.1, f (x) has dispersion, but where does it come from? To understand this, we checked the dispersion of x and scaling effects of transformation f .Dispersion of xWe also checked the coefficient of variation (CV) of x .Table 4shows that the CV is 0.12 on average; the value of x often varies from 0.88 times to 1.12 times the mean value. This is less than the dispersion of f (x) .Scaling effects of f We investigate how much transformation f can change x . Since the affine transformation f : R d → R d can be regarded as a linear transformation R d+1 → R d+1 (Appendix B), its scaling effect can be analyzed by checking the singular values of the linear transformation.Figure 9shows the results of randomly selected heads in BERT. The singular values of the transformation f for each head are in descending order. In each head, there is a difference of at least 1.8 times or more between the maximum and minimum singular values. That is, the value of f (x) varies greatly due to the transformation f .B Affine transformation as linear transformationAffine transformation f : R d → R d in Equation 4 can be regarded as a linear transformation R d+1 → R d+1 . If 1 is added to the end of each input vector x ∈ R d and x := x 1 ∈ R d+1 is given, the affine transformation f can be identified with the linear transformation f as
On Identifiability in Transformers. Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Massimiliano Ciaramita, Roger Wattenhofer, 8th International Conference on Learning Representations. ICLRGino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Watten- hofer. 2020. On Identifiability in Transformers. In 8th International Conference on Learning Represen- tations (ICLR).
What Does BERT Look At? An Analysis of BERT's Attention. Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D Manning, 10.18653/v1/W19-4828Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLPKevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What Does BERT Look At? An Analysis of BERT's Attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 10.18653/v1/N19-1423Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language TechnologiesLong and Short Papers1NAACL-HLT)Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers) (NAACL-HLT), pages 4171-4186.
Saliency-driven Word Alignment Interpretation for Neural Machine Translation. Shuoyang Ding, Hainan Xu, Philipp Koehn, 10.18653/v1/W19-5201Proceedings of the Fourth Conference on Machine Translation. the Fourth Conference on Machine Translation1Research Papers) (WMT)Shuoyang Ding, Hainan Xu, and Philipp Koehn. 2019. Saliency-driven Word Alignment Interpretation for Neural Machine Translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers) (WMT), pages 1-12.
Visualizing and Understanding Neural Machine Translation. Yanzhuo Ding, Yang Liu, Huanbo Luan, Maosong Sun, 10.18653/v1/P17-1106Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. the 55th Annual Meeting of the Association for Computational Linguistics1Long Papers) (ACL)Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and Understanding Neural Machine Translation. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), pages 1150-1159.
A Simple, Fast, and Effective Reparameterization of IBM Model 2. Chris Dyer, Victor Chahuneau, Noah A Smith, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACCL-HLT). the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACCL-HLT)Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A Simple, Fast, and Effective Reparameter- ization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACCL-HLT), pages 644- 648.
Yoav Goldberg, arXiv:1901.05287Assessing BERT's Syntactic Abilities. arXiv preprintYoav Goldberg. 2019. Assessing BERT's Syntactic Abilities. arXiv preprint arXiv:1901.05287.
A Structural Probe for Finding Syntax in Word Representations. John Hewitt, D Christopher, Manning, 10.18653/v1/N19-1419Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language TechnologiesLong and Short Papers1NAACL-HLT)John Hewitt and Christopher D Manning. 2019. A Structural Probe for Finding Syntax in Word Rep- resentations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers) (NAACL-HLT), pages 4129-4138.
Jason Phu Mon Htut, Shikha Phang, Samuel R Bordia, Bowman, arXiv:1911.12246Do Attention Heads in BERT Track Syntactic Dependencies? arXiv preprint. Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do Attention Heads in BERT Track Syntactic Dependencies? arXiv preprint arXiv:1911.12246.
What Does BERT Learn about the Structure of Language?. Ganesh Jawahar, Benoît Sagot, Djamé Seddah, 10.18653/v1/P19-1356Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). the 57th Annual Meeting of the Association for Computational Linguistics (ACL)Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics (ACL), pages 3651-3657.
Revealing the Dark Secrets of BERT. Olga Kovaleva, Alexey Romanov, Anna Rogers, Anna Rumshisky, 10.18653/v1/D19-1445Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the Dark Secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 4364-4373.
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut, 8th International Conference on Learning Representations. ICLRZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In 8th Inter- national Conference on Learning Representations (ICLR).
Visualizing and Understanding Neural Models in NLP. Jiwei Li, Xinlei Chen, Eduard Hovy, Dan Jurafsky, 10.18653/v1/N16-1082Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Juraf- sky. 2016. Visualizing and Understanding Neural Models in NLP. In Proceedings of the 2016 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies (NAACL-HLT), pages 681-691.
On the Word Alignment from Neural Machine Translation. Xintong Li, Guanlin Li, Lemao Liu, Max Meng, Shuming Shi, 10.18653/v1/P19-1124Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). the 57th Annual Meeting of the Association for Computational Linguistics (ACL)Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the Word Alignment from Neural Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics (ACL), pages 1293-1303.
Open Sesame: Getting Inside BERT's Linguistic Knowledge. Yongjie Lin, Yi Chern Tan, Robert Frank, 10.18653/v1/W19-4825Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLPYongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open Sesame: Getting Inside BERT's Linguis- tic Knowledge. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpret- ing Neural Networks for NLP, pages 241-253.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, arXiv:1907.11692RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprintYinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv preprint arXiv:1907.11692.
From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions. David Mareček, Rudolf Rosa, 10.18653/v1/W19-4827Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLPDavid Mareček and Rudolf Rosa. 2019. From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 263- 275.
Improved Statistical Alignment Models. Josef Franz, Hermann Och, Ney, 10.3115/1075218.1075274Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL). the 38th Annual Meeting of the Association for Computational Linguistics (ACL)Franz Josef Och and Hermann Ney. 2000. Improved Statistical Alignment Models. In Proceedings of the 38th Annual Meeting of the Association for Compu- tational Linguistics (ACL), pages 440-447.
A Systematic Comparison of Various Statistical Alignment Models. Josef Franz, Hermann Och, Ney, 10.1162/089120103321337421Computational Linguistics. 291Franz Josef Och and Hermann Ney. 2003. A System- atic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19-51.
fairseq: A Fast, Extensible Toolkit for Sequence Modeling. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli, 10.18653/v1/N19-4009Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics (Demonstrations), pages 48-53.
An Analysis of Encoder Representations in Transformer-Based Machine Translation. Alessandro Raganato, Jörg Tiedemann, 10.18653/v1/W18-5431Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLPAlessandro Raganato and Jörg Tiedemann. 2018. An Analysis of Encoder Representations in Transformer-Based Machine Translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297.
Visualizing and Measuring the Geometry of BERT. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, Been Kim, Advances in Neural Information Processing Systems. 32Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and Measuring the Geometry of BERT. Advances in Neural Information Processing Systems 32 (NIPS), pages 8594-8603.
Anna Rogers, Olga Kovaleva, Anna Rumshisky, arXiv:2002.123272020. A Primer in BERTology: What we know about how BERT works. arXiv preprintAnna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What we know about how BERT works. arXiv preprint arXiv:2002.12327.
Neural Machine Translation of Rare Words with Subword Units. Rico Sennrich, Barry Haddow, Alexandra Birch, 10.18653/v1/P16-1162Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. the 54th Annual Meeting of the Association for Computational Linguistics1Long Papers) (ACL)Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), pages 1715-1725.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B Viégas, Martin Wattenberg, arXiv:1706.03825SmoothGrad: removing noise by adding noise. arXiv preprintDaniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B Viégas, and Martin Wattenberg. 2017. SmoothGrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825.
An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation. Gongbo Tang, Rico Sennrich, Joakim Nivre, 10.18653/v1/W18-6304Proceedings of the Third Conference on Machine Translation (WMT): Research Papers. the Third Conference on Machine Translation (WMT): Research PapersGongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation (WMT): Research Papers, pages 26-35.
What do you learn from context? Probing for sentence structure in contextualized word representations. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, Thomas Mccoy, Najoung Kim, Benjamin Van Durme, Dipanjan Samuel R Bowman, Ellie Das, Pavlick, 7th International Conference on Learning Representations. ICLRIan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipan- jan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th Inter- national Conference on Learning Representations (ICLR).
Attention is All you Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin, Advances in Neural Information Processing Systems 30 (NIPS). Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Pro- cessing Systems 30 (NIPS), pages 5998-6008.
AER: Do we need to "improve" our alignments?. David Vilar, Maja Popović, Hermann Ney, International Workshop on Spoken Language Translation (IWSLT) 2006. David Vilar, Maja Popović, and Hermann Ney. 2006. AER: Do we need to "improve" our alignments? In International Workshop on Spoken Language Translation (IWSLT) 2006, pages 205-212.
XLNet: Generalized Autoregressive Pretraining for Language Understanding. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, V Quoc, Le, Advances in Neural Information Processing Systems 32 (NIPS). Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretrain- ing for Language Understanding. In Advances in Neural Information Processing Systems 32 (NIPS), pages 1-18.
Thomas Zenkel, Joern Wuebker, John Denero, arXiv:1901.11359Adding Interpretable Attention to Neural Translation Models Improves Word Alignment. arXiv preprintThomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding Interpretable Attention to Neu- ral Translation Models Improves Word Alignment. arXiv preprint arXiv:1901.11359.
Attention Module is Not Only a Weight: Analyzing Transformers with Vector Norms
https://arxiv.org/abs/2004.10102
https://proceedings.mlr.press/v139/touvron21a
Training data-efficient image transformers & distillation through attention
2021-07-01
Hugo Touvron; Matthieu Cord; Matthijs Douze; Francisco Massa; Alexandre Sablayrolles; Herve Jegou
<div><article>
<p><i>Proceedings of the 38th International Conference on Machine Learning</i>, PMLR 139:10347-10357, 2021.
 </p>
<h4>Abstract</h4>
<p>
 Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.
 </p>
<h4>Cite this Paper</h4>
<hr/>
<div><p> BibTeX</p> <p><code>
@InProceedings{pmlr-v139-touvron21a,
 title = 	 {Training data-efficient image transformers &amp; distillation through attention},
 author = {Touvron, Hugo and Cord, Matthieu and Douze, Matthijs and Massa, Francisco and Sablayrolles, Alexandre and Jegou, Herve},
 booktitle = 	 {Proceedings of the 38th International Conference on Machine Learning},
 pages = 	 {10347--10357},
 year = 	 {2021},
 editor = 	 {Meila, Marina and Zhang, Tong},
 volume = 	 {139},
 series = 	 {Proceedings of Machine Learning Research},
 month = 	 {18--24 Jul},
 publisher = {PMLR},
 pdf = 	 {http://proceedings.mlr.press/v139/touvron21a/touvron21a.pdf},
 url = 	 {https://proceedings.mlr.press/v139/touvron21a.html},
 abstract = 	 {Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.}
}
</code></p>
</div>
<div><p> Endnote</p> <p><code>%0 Conference Paper
%T Training data-efficient image transformers &amp; distillation through attention
%A Hugo Touvron
%A Matthieu Cord
%A Matthijs Douze
%A Francisco Massa
%A Alexandre Sablayrolles
%A Herve Jegou
%B Proceedings of the 38th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2021
%E Marina Meila
%E Tong Zhang	
%F pmlr-v139-touvron21a
%I PMLR
%P 10347--10357
%U https://proceedings.mlr.press/v139/touvron21a.html
%V 139
%X Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.
</code></p>
</div>
<div><p> APA</p> <p><code>
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A. &amp; Jegou, H.. (2021). Training data-efficient image transformers &amp; distillation through attention. <i>Proceedings of the 38th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 139:10347-10357 Available from https://proceedings.mlr.press/v139/touvron21a.html.
</code></p>
</div>
<hr/>
<h4>Related Material</h4>
</article></div>||||I|||| Proceedings of Machine Learning Research
 Volume 139JMLR MLOSS FAQ Submission Format

 [edit]

 Training data-efficient image transformers & distillation through attention

 Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou
 Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10347-10357, 2021.

 Abstract

 Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.

 Cite this Paper

 BibTeX
 @InProceedings{pmlr-v139-touvron21a, title = {Training data-efficient image transformers & distillation through attention}, author = {Touvron, Hugo and Cord, Matthieu and Douze, Matthijs and Massa, Francisco and Sablayrolles, Alexandre and Jegou, Herve}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {10347--10357}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/touvron21a/touvron21a.pdf}, url = {https://proceedings.mlr.press/v139/touvron21a.html}, abstract = {Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.} }
 Copy to ClipboardDownload
 Endnote
 %0 Conference Paper %T Training data-efficient image transformers & distillation through attention %A Hugo Touvron %A Matthieu Cord %A Matthijs Douze %A Francisco Massa %A Alexandre Sablayrolles %A Herve Jegou %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-touvron21a %I PMLR %P 10347--10357 %U https://proceedings.mlr.press/v139/touvron21a.html %V 139 %X Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.
 Copy to ClipboardDownload
 APA
 Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A. & Jegou, H.. (2021). Training data-efficient image transformers & distillation through attention. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:10347-10357 Available from https://proceedings.mlr.press/v139/touvron21a.html.
 Copy to ClipboardDownload

 Related Material

 * Download PDF
 * Supplementary PDF

 This site last compiled Thu, 21 Apr 2022 18:23:36 +0000

 Github Account
 Copyright © The authors and PMLR 2022. MLResearchPress
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10347-10357, 2021.
 
Abstract
 Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.
 
Cite this Paper
  BibTeX  
@InProceedings{pmlr-v139-touvron21a,
 title = 	 {Training data-efficient image transformers &amp; distillation through attention},
 author = {Touvron, Hugo and Cord, Matthieu and Douze, Matthijs and Massa, Francisco and Sablayrolles, Alexandre and Jegou, Herve},
 booktitle = 	 {Proceedings of the 38th International Conference on Machine Learning},
 pages = 	 {10347--10357},
 year = 	 {2021},
 editor = 	 {Meila, Marina and Zhang, Tong},
 volume = 	 {139},
 series = 	 {Proceedings of Machine Learning Research},
 month = 	 {18--24 Jul},
 publisher = {PMLR},
 pdf = 	 {http://proceedings.mlr.press/v139/touvron21a/touvron21a.pdf},
 url = 	 {https://proceedings.mlr.press/v139/touvron21a.html},
 abstract = 	 {Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.}
}
 
  Endnote  %0 Conference Paper
%T Training data-efficient image transformers &amp; distillation through attention
%A Hugo Touvron
%A Matthieu Cord
%A Matthijs Douze
%A Francisco Massa
%A Alexandre Sablayrolles
%A Herve Jegou
%B Proceedings of the 38th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2021
%E Marina Meila
%E Tong Zhang	
%F pmlr-v139-touvron21a
%I PMLR
%P 10347--10357
%U https://proceedings.mlr.press/v139/touvron21a.html
%V 139
%X Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolution-free transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.
 
  APA  
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A. &amp; Jegou, H.. (2021). Training data-efficient image transformers &amp; distillation through attention. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:10347-10357 Available from https://proceedings.mlr.press/v139/touvron21a.html.
 
Related Material
Training data-efficient image transformers & distillation through attention
https://proceedings.mlr.press/v139/touvron21a
https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html
Rethinking Graph Transformers with Spectral Attention
2021-12-06
Kreuzer; Devin; Beaini; Dominique; Hamilton; Will; Létourneau; Vincent; Tossou; Prudencio
<div><div>
<h4>Rethinking Graph Transformers with Spectral Attention</h4>
<p>
 Part of
 <a href="https://proceedings.neurips.cc/paper_files/paper/2021">Advances in Neural Information Processing Systems 34 (NeurIPS 2021)
 </a>
</p>
<br/>
<h4>Authors</h4>
<p><i>Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, Prudencio Tossou</i></p>
<h4>Abstract</h4>
<p>In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the \textit{Spectral Attention Network} (SAN), which uses a learned positional encoding (LPE) that can take advantage of the full Laplacian spectrum to learn the position of each node in a given graph.This LPE is then added to the node features of the graph and passed to a fully-connected Transformer.By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance.Further, by fully connecting the graph, the Transformer does not suffer from over-squashing, an information bottleneck of most GNNs, and enables better modeling of physical phenomenons such as heat transfer and electric interaction.When tested empirically on a set of 4 standard datasets, our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin, becoming the first fully-connected architecture to perform well on graph benchmarks.</p>
</div><div>
<p>Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.</p>
<p>Use the "Report an Issue" link to request a name change.</p>
</div><p>Do not remove: This comment is monitored to verify that the site is working properly</p></div>||||I|||| NeurIPS Proceedings
 * 
 * 
 Search


 Rethinking Graph Transformers with Spectral Attention

 Part of Advances in Neural Information Processing Systems 34 (NeurIPS 2021)

 Bibtex Paper Reviews And Public Comment » Supplemental


 Authors

 Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, Prudencio Tossou

 Abstract

 In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the \textit{Spectral Attention Network} (SAN), which uses a learned positional encoding (LPE) that can take advantage of the full Laplacian spectrum to learn the position of each node in a given graph.This LPE is then added to the node features of the graph and passed to a fully-connected Transformer.By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance.Further, by fully connecting the graph, the Transformer does not suffer from over-squashing, an information bottleneck of most GNNs, and enables better modeling of physical phenomenons such as heat transfer and electric interaction.When tested empirically on a set of 4 standard datasets, our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin, becoming the first fully-connected architecture to perform well on graph benchmarks.


 Name Change Policy

 ×

 Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.

 Use the "Report an Issue" link to request a name change.

 Report an Issue | Name Change Policy
 Do not remove: This comment is monitored to verify that the site is working properly
Rethinking Graph Transformers with Spectral Attention
 Part of
 Advances in Neural Information Processing Systems 34 (NeurIPS 2021)
 
Authors
 Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, Prudencio Tossou 
Abstract
In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the \textit{Spectral Attention Network} (SAN), which uses a learned positional encoding (LPE) that can take advantage of the full Laplacian spectrum to learn the position of each node in a given graph.This LPE is then added to the node features of the graph and passed to a fully-connected Transformer.By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance.Further, by fully connecting the graph, the Transformer does not suffer from over-squashing, an information bottleneck of most GNNs, and enables better modeling of physical phenomenons such as heat transfer and electric interaction.When tested empirically on a set of 4 standard datasets, our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin, becoming the first fully-connected architecture to perform well on graph benchmarks.
 
Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.
Use the "Report an Issue" link to request a name change.
 Do not remove: This comment is monitored to verify that the site is working properly
Rethinking Graph Transformers with Spectral Attention
https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html
https://arxiv.org/abs/1905.04226
Language Modeling with Deep Transformers
2019-05-10
Irie; Kazuki; Zeyer; Albert; Schlüter; Ralf; Ney; Hermann
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/1905.04226">Download PDF</a></p><blockquote>
 Abstract: We explore deep autoregressive Transformer models in language modeling for
speech recognition. We focus on two aspects. First, we revisit Transformer
model configurations specifically for language modeling. We show that well
configured Transformer models outperform our baseline models based on the
shallow stack of LSTM recurrent neural network layers. We carry out experiments
on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level
and 10K byte-pair encoding subword-level language modeling. We apply our
word-level models to conventional hybrid speech recognition by lattice
rescoring, and the subword-level models to attention based encoder-decoder
models by shallow fusion. Second, we show that deep Transformer language models
do not require positional encoding. The positional encoding is an essential
augmentation for the self-attention mechanism which is invariant to sequence
ordering. However, in autoregressive setup, as is the case for language
modeling, the amount of information increases along the position dimension,
which is a positional signal by its own. The analysis of attention weights
shows that deep autoregressive self-attention models can automatically make use
of such positional information. We find that removing the positional encoding
even slightly improves the performance of these models.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Kazuki Irie [<a href="/show-email/930c46a2/1905.04226">view email</a>]
 <br />
 <strong><a href="/abs/1905.04226v1">[v1]</a></strong>
 Fri, 10 May 2019 15:50:00 UTC (1,072 KB)<br /><strong>[v2]</strong>
Thu, 11 Jul 2019 15:45:32 UTC (941 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:1905.04226

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:1905.04226 (cs)
 [Submitted on 10 May 2019 (v1), last revised 11 Jul 2019 (this version, v2)]

 Title: Language Modeling with Deep Transformers

 Authors: Kazuki Irie, Albert Zeyer, Ralf Schlüter, Hermann Ney
 Download PDF
 Abstract: We explore deep autoregressive Transformer models in language modeling for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for language modeling. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level language modeling. We apply our word-level models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer language models do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for language modeling, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.
 Comments: To appear in the proceedings of INTERSPEECH 2019 
 Subjects: Computation and Language (cs.CL) ; Machine Learning (cs.LG)
 Cite as: arXiv:1905.04226 [cs.CL] 
 (or arXiv:1905.04226v2 [cs.CL] for this version) 
 https://doi.org/10.48550/arXiv.1905.04226 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 https://doi.org/10.21437/Interspeech.2019-2225 
 Related DOI: Focus to learn more 
 DOI(s) linking to related resources 
 

 Submission history

 From: Kazuki Irie [view email]
 [v1] Fri, 10 May 2019 15:50:00 UTC (1,072 KB)
 [v2] Thu, 11 Jul 2019 15:45:32 UTC (941 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 1905
 Change to browse by:
 cs
 cs.LG

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Kazuki Irie
 Albert Zeyer
 Ralf Schlüter
 Hermann Ney
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs and how to get involved.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Language Modeling with Deep Transformers
Kazuki Irie irie@cs.rwth-aachen.de 
Computer Science Department
Human Language Technology and Pattern Recognition Group
RWTH Aachen University
52074AachenGermany
Albert Zeyer zeyer@cs.rwth-aachen.de 
Computer Science Department
Human Language Technology and Pattern Recognition Group
RWTH Aachen University
52074AachenGermany
AppTek GmbH
52062AachenGermany
Ralf Schlüter schlueter@cs.rwth-aachen.de 
Computer Science Department
Human Language Technology and Pattern Recognition Group
RWTH Aachen University
52074AachenGermany
Hermann Ney ney@cs.rwth-aachen.de 
Computer Science Department
Human Language Technology and Pattern Recognition Group
RWTH Aachen University
52074AachenGermany
AppTek GmbH
52062AachenGermany
Language Modeling with Deep Transformers
Index Terms: language modelingself-attentionTransformerspeech recognition
We explore multi-layer autoregressive Transformer models in language modeling for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for language modeling. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level language modeling. We apply our wordlevel models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer language models do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for language modeling, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive selfattention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.
Introduction
Transformer encoder-decoder models [1] have become popular in natural language processing. The Transformer architecture allows to successfully train a deep stack of self-attention layers [2][3][4] via residual connections [5] and layer normalization [6]. The positional encodings [1,7], typically based on sinusoidal functions, are used to provide the self-attention with the sequence order information. Across various applications, systematic improvements have been reported over the standard, multi-layer long short-term memory (LSTM) [8] recurrent neural network based models. While originally designed as an encoder-decoder architecture in machine translation, the encoder (e.g., [9]) and the decoder (e.g., [10]) components are also separately used in corresponding problems depending on whether the problem disposes the whole sequence for prediction or not.
A number of recent works have also shown impressive performance in language modeling using the Transformer decoder component [10][11][12][13][14][15]. The earliest example can be found in [10] where such models are investigated for text generation. Recent works on training larger and deeper models [12,14,15] have shown further potential of the Transformer in language modeling. On the other hand, an obvious limitation of the Transformers is that their memory requirement linearly increases in terms of number of tokens in the sequence, which requires to work with a limited context window (basically a n-gram model where the typical number for n is 512) for tasks dealing with long sequences such as character-level language modeling [12]. Dai et al. [11] has introduced a segment-level recurrence and relative positional encoding in the Transformer language model to be able to potentially handle unlimited context.
In this work, we investigate deep autoregressive Transformers for language modeling in speech recognition. To be specific, we focus on two aspects. First, we revisit the parameter configurations of Transformers, originally engineered for the sequenceto-sequence problem [1], specifically for language modeling. We conduct experiments on the LibriSpeech automatic speech recognition (ASR) task [16] for both word-level conventional speech recognition and byte-pair encoding (BPE) [17] level end-to-end speech recognition [18,19]. We apply our word-level models to hybrid speech recognition by lattice rescoring [20], and the BPElevel models to end-to-end models by shallow fusion [21,22]. We show that well configured Transformer language models outperform models based on the simple stack of LSTM RNN layers in terms of both perplexity and word error rate (WER).
Second, we experimentally show that the positional encoding is not needed for multi-layer autoregressive self-attention models. The visualization of the attention weights shows that when the sinusoidal positional encoding is provided with the input, the first layer of the Transformers learns to extract ngram features (therefore making use of positional information). However, in the autoregressive problem where a new token is provided to the model at each time step, the amount of information the model has access to strictly increases from left to right at the lowest level of the network, which should provide some positional information by its own. We observe that deep Transformer language models without positional encoding automatically make use of such information, and even give slight improvements over models with positional encodings.
Related Work
The first part of our work follows the spirits of Al-Rfou et al.'s work [12] and Radford et al.'s work [14,15] in investigating larger and deeper Transformers for language modeling. We show that deep Transformer language models can be successfully applied to speech recognition and give good performance. The second part of this work concerns the positional encoding, which is a crucial component in the original Transformer. There are active investigations on positional encoding variants to improve self-attention (e.g., [11,23]). Previous works in Transformer language models systematically use positional encoding, either jointly learned one or the sinusoidal one (both cases are reported to give similar performance in [12]). We show that the deep autoregressive self-attention models do not require any explicit model for encoding positions to give the best performance.
Autoregressive Self-Attention
The language model we consider is based on the decoder component of the Transformer architecture [1]. Similar to previous work [10-12, 14, 15], we define layer as a stack of two modules:
self-attention and feed-forward 1 modules.
The autoregressive self-attention module in the l-th layer transforms the input z (l−1) t at position t as follows:
x (l) t = LayerNorm(z (l−1) t ) q (l) t , k (l) t , v (l) t = Qx (l) t , Kx (l) t , V x (l) t h (l) t = h (l) t−1 , (k (l) t , v (l) t ) y (l) t = z (l−1) t + W0 SelfAttention(h (l) t , q (l) t )
where Q, K, V , respectively denote query, key, value projection matrices, LayerNorm denotes layer normalization [6], SelfAttention denotes the scaled multi-head dot product selfattention [1], and W0 denotes the projection matrix for the residual connection [5].
The output y (l) t is then fed to the feed-forward module:
m (l) t = LayerNorm(y (l) t ) z (l) t = y (l) t + W2 Activation(W1m (l) t )
where Activation is rectifier [24], Gaussian error linear unit (GELU) [15,25], or gated linear unit (GLU) [26] in this work. The final model is build by stacking these layers multiple times.
The input of the network consists of the sum of the token embedding (word or BPE in this work) and the sinusoidal positional encoding as specified in [1]. The output softmax layer gives the probability distribution for the next token. As shown in the equations above, h (l) t can be seen as states of the Transformer model 2 (whose size, as opposed to the standard RNN states, linearly grows as we progress along the position dimension). During inference, these states are stored to avoid redundant computation. During training, the computation along the position dimension is parallelized for speed-up.
LibriSpeech Dataset
Language Modeling Data Descriptions
The LibriSpeech datasets [16] for language modeling consists of 800M-word text only data and 960hr of audio transcriptions which corresponds to 10M-word text data. Based on analysis of count model perplexities, we observe that the audio transcription part does not contain special domain signal which matches the development set. Therefore, we simply merge the two datasets to form one training dataset for language model training. The average sentence length in the resulting training data is 21 words with the maximum length of 600 words. The development and test sets respectively have two parts: dev-clean, dev-other, testclean, and test-other. This separation is based on the audio-level characteristics, therefore it has no special meaning for language modeling. In the experimental section, we denote by "Dev" and "Test" the concatenation of clean and other parts of the respective data. Both datasets consist of about 110K running words with average of 20 words per sentence. The word-level vocabulary contains 200K words.
4-gram count and LSTM-RNN Baselines
We use the official 4-gram count language model provided with the LibriSpeech dataset [16]. No improvement in perplexity is observed when going up to 5-grams. For LSTM-RNN language models [27], we first train our base configuration; the model has 2 LSTM-RNN layers with dimension 2048 and the input projection layer of 128, where the dropout with a rate of 0.2 is applied between each layer. Since we observe that this model underfits the LibriSpeech training set, we further train two models; the same model without dropout, and the one with 4 LSTM layers stacked without dropout. Both these changes effectively give better perplexity, though our current experiments did not show improvements from simply stacking more LSTM layers. The perplexities of these models are summarized in Table 1. We observe that a good relative improvements of greater than 57% is obtained by LSTM language models over the 4-gram model. 
Text based Experiments
We carry out experiments for both word-level and BPE-level language modeling. We first focus on the word-level one.
Hyper-parameters in Transformers
The Transformer architecture presents a new search space Odyssey [28]. The exhaustive model hyper-parameters for Transformer language models specified by the equations in Sec. 3 are the number of layers and the dimension of the residual connection, and for each layer the number of attention heads, the dimension of the key and query, the dimension of the value, and the dimension of the feed-forward layer.
In our experiments, we use the same dimension for key, query and value, as well as the residual connection. We use the same dimensionality across all layers. Therefore, our models can be fully specified by the tuple (number of layers L, feedforward dimension d f f , residual dimension dres, number of heads H). We do not apply any regularization method including dropout. We train all models using the plain stochastic gradient descent and new-bob learning rate tuning on a single GPU. We define our training sub-epoch (for new-bob) as the 10th of the full training data. All our implementations are based on the Tensorflow [29] based open-source toolkit RETURNN [30] 3 .
Hyper-parameter Tuning
Given the amount of LibriSpeech training data (810M words), it is unreasonable to train all model variants until full convergence. The earlier stage of the training already consistently indicates the performance of the models. Therefore, we carry out comparisons between models with different configuration at the equal, large enough, but reasonable number of updates.
The first set of comparison investigates the effect of depth and width. The perplexity results can be found in Table 2 Table 2 shows the effect of number of layers; we observe that increasing number of layers (therefore the number of parameters) from 1 to 42 gradually improves the perplexity. In the lower part of Table 2 , we vary both the number of layers, feed-forward dimension, and the residual dimension. First of all, the 12-layer (12,4096,512,8) model outperforms the 6-layer (6,8192,512,8) model, while having similar number of parameters, which seems to indicate that the depth effectively benefits Transformer language models. We also train an extreme model which has only 2 layers with wide dimensions (2,8192,2048,8). The number of parameters in fact blows up because of the large value of dres which results in a large matrix in the output softmax layer with 200K vocabulary 4 . We observe that such wide but shallow models do not perform well. However, the softmax bottleneck dimension for language modeling typically needs to be large for the best performance [31]. In Transformers, the bottleneck dimension corresponds to the residual connection dimension which is typically kept rather small (typically 512 or 1024). As a control experiment, we also train a model in which we insert an additional projection layer with a large dimension before the softmax layer to give larger bottleneck capacity. Table 3 shows the comparison conducted on the (12,2048,512,8) model. We observe that simply enlarging the bottleneck dimension does not improve Transformer models.  Table 4 shows the effect of number of attention heads. 16 heads which is the largest number we try in this setup give the best performance. In addition, we examine the type of activation function ( Table 5). As opposed to previous work on feed-forward language models using GLUs [26,32], we do not observe faster convergence. As we observe that the impact of choice of activation functions on the perplexity is overall limited, all our other models use the standard ReLU. As reported in the original Transformer, we confirm that both layer normalization and residual connections are needed for these models for stable training 5 .
Finally, we train models with the best configurations until convergence. Table 6 shows the perplexities which are better than those obtained by our LSTM based models (Table 1).
Parameter Tying
Dehghani et al. [34] reports Universal Transformers to perform particularly well for language modeling. This motivates us to experiment with Transformer models which share the parameters across layers. For a Universal Transformer to have comparable number of parameters with the standard deep Transformers, the 4 We note that this is also the reason why the number of parameters of our baseline LSTM language models in Table 1 is relatively high. 5 We tried to train multiple models without either residual connections or layer normalization. Also, following [15], we tried reorganizing the feed-forward module to insert one additional pre-activation layer normalization [33] and one more activation function. However, we did not observe any improvement. The original Transformers anyway do not have any activation on the residual path throughout the whole network.   (24,2048,512,8).
Activation Perplexity Train Dev ReLU [1,24] 76.4 72.5 GLU [26] 76.5 72.8 GELU [15,25] 75.7 72.2 dimensions in each layer must be increased, which results in slower training; here we simply investigate the effect of number of recurrence. Table 7 shows the perplexity results. First of all, we observe that the model performance is behind that of the standard Transformer 6 . However, we clearly observe that increasing the number of layers from 3 to 6 consistently improves the perplexity. This improvement without additional parameters motivates future work to investigate further parameter sharing strategies for Transformers. 
ASR Experiments
Lattice Rescoring Results
We apply our word-level Transformer language models to conventional hybrid speech recognition by lattice rescoring. The standard push-forward lattice rescoring algorithm [20] for longspan language models can be directly applied to self-attention based models. The only modifications from the RNN version is to define the "state" as all hidden states (h  Table 8 shows the WERs and perplexities (PPL). Our baseline acoustic model is based on multi-layer bi-directional LSTM [35]. Further descriptions of our baseline acoustic model can be found in [36]. We obtain consistent improvements in WER over the LSTM baselines.
End-to-End ASR Shallow Fusion Results
We train 10K BPE-level Transformer language models to be combined with attention-based encoder-decoder speech model by shallow fusion [21,22]. The 10K BPE level training data has a longer average length of 24 tokens per sentence with the longest sentence length of 1343, which is still manageable without any truncation for self-attention. We use the Transformer architecture of (24,2048,512,8). The two-layer LSTM architecture is also 6 We note that here the direct comparison is not as straightforward as between the standard Transformers. In fact, we observe that the training hyperparameters tuned for the standard Transformers can not be directly applied to Universal Transformers; specifically, we find it crucial to reduce the gradient norm clipping threshold from 1 to 0.1, which is potentially slowing down the convergence .     so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping  the  shore   so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping  the  shore     (a) First layer with PE     so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping  the  shore   so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened to the sea lapping the shore  (b) First layer without PE     so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping  the  shore   so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping the shore  (c) "Blur" layer     so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping  the  shore   so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened to the sea lapping the shore  (d) " Window" layer     so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened  to  the  sea  lapping  the  shore   so  they  went  on  to  the  verandah  and  looked  down  upon  the  lights  of  the  prison  and  listened to the sea lapping the shore  (e) "Structured" layer Figure 1: Layer Categories in word-level 24-layer Transformer language models. The x-axis corresponds to the input words. The y-axis shows the target words; each target word position has 8 sub-rows corresponding to 8 heads. "PE" denotes positional encoding. the same as described in 4.2 without dropout. We refer to our previous work [19] for the description of the baseline attention model; the better baseline WERs than our previous work [19] are obtained by improved curriculum learning and longer training. Table 9 shows both perplexities and WERs 7 . Again, we obtain consistent improvements over the LSTM baseline. These results are slightly better than previously reported best WERs [37][38][39] for end-to-end models without data augmentation. 
Analysis
Compared with hidden states in RNNs, attention weights are easier to be visualized, which gives opportunity for analysis.
In particular, we focus on the comparison of the Transformer language models with and without positional encoding.
Transformer LM without positional encoding
In the autoregressive problem where a new token is provided to the model at each time step, the amount of information the model has access to strictly increases from left to right at the lowest level of the network; the deeper layers should be able to recognize this structure which should provide the model with some positional information by its own. To check this hypothesis, we train models without any positional encoding. First, we observe that they give better perplexities than the models with sinusoidal positional encoding (Table 10).
First layer
The attention in the first layer is the most straightforward for interpretation because the feature at each position exactly cor- 7 Following [37], we introduced an end-of-sentence penalty and a larger beam size for shallow fusion which gave direct improvements in WERs; we expect further improvements from tuning of these parameters. responds to the word at the position (while deeper layers can potentially shuffle the feature content). The attention weights in the first layer of 24-layer Transformer language models with and without positional encodings are visualized in Figure 1. We observe that the first layer of the model with positional encoding (Figure 1(a)) learns to create n-gram features (roughly 2 or 3gram), which indicates that the positional information is directly used. In contrast, the first layer of the model without positional encoding learns to focus on the new input token as can be seen as the diagonal in Figure 1(b) (interestingly, we also see that it ignores some functional words such as "the", "and", "to" which might be modeled by some off-set values, therefore attending to the beginning of sentence token instead), which demonstrates that the model is aware of the position of the new input.
Other layers
We observe that the behavior of other layers are rather similar for both Transformer models with and without positional encoding. We find 3 categories of layers in the other 23 layers; the second and third layers are "blur" layers as shown in Figure 1(c), which seems to roughly average over all positions (while we can also see that some heads focus on difficult words, here "verandah"). Layer 4 to 9 are window layers which focus on the local n-gram. A representative example is show in Figure 1(d). Finally, we find the top layers 10 to 23 to be more structured, attending to some specific patterns; an example is shown in Figure 1(e).
Conclusion
We apply deep Transformer language models for speech recognition. We show that such models outperform the shallow stack of LSTM-RNNs on both word-level and BPE-level modeling. Future work investigates application of crucial components of deep Transformers (such as layer normalization) to deeper LSTM models; e.g., the RNMT+ decoder architecture [40] for language modeling. Furthermore, we do not apply any regularization on models for the large LibriSpeech task, as no overfitting is observed in the range of model sizes we experimented with (for the word-level models). We can possibly still improve our models simply by scaling up their size and using regularization.
Sec.3) in all layers from all predecessor positions and the current position (t; for position encoding).
Table 1 :
1Perplexities of the baseline models.Model 
Num. Dropout 
Dev 
Test 
Params 
layers 
rate 
in M 
4-gram 
-
-
146.2 151.8 
230 
LSTM 
2 
0.2 
71.3 
74.8 
487 
0.0 
66.6 
69.9 
4 
61.9 
64.9 
554 
. All models in the table use 8 attention heads. Other parameters are specified in the table. The table is organized in two parts. The upper part of
Table 2 :
2Perplexity after 2.5 epoch (25 sub-epochs in our setup; 6.5M updates). The number of heads H is 8 for all models below.L d f f dres 
Params. Perplexity 
in M 
Train Dev 
1 
2048 512 
208 
108.3 104.9 
6 
224 
75.7 74.3 
12 
243 
67.6 67.1 
24 
280 
62.2 62.3 
32 
306 
60.1 60.6 
42 
338 
59.5 60.3 
2 8192 
2048 536 
73.1 73.8 
6 
512 
262 
66.7 66.7 
12 4096 
268 
63.5 63.8 
4 16384 
277 
67.6 67.4 
Table 3 :
3Effect of larger softmax bottleneck dimension. Perplexity after 1 epoch (10 sub-epochs in our setup; 2.6M updates) for(12, 2048, 512,8).Bottleneck Params. 
Perplexity 
dimension 
in M 
Train Dev 
512 
243 
76.4 72.5 
2048 
512 
74.7 73.8 
Table 4 :
4Effect of number of heads. Perplexity after 2.5 epoch for(12, 2048, 512, H).H 
Params. 
Perplexity 
in M 
Train Dev 
1 
243 
71.9 70.8 
4 
69.1 68.6 
8 
67.6 67.1 
16 
66.9 66.6 
Table 5 :
5Effectof activation 
functions. Perplexity after 1 
epoch (10 sub-epochs in our 
setup) for 
Table 6 :
6Perplexity of the best models. dres is 512 for all models.L 
d f f 
Para. Max 
Perplexity 
in M Ep. 
Train Dev Test 
12 4096 268 
55 
57.3 59.9 62.3 
24 
2048 
281 
50 
55.6 58.0 60.7 
32 
306 
53.4 56.6 59.5 
42 
338 
51.2 55.0 57.7 
Table 7 :
7Effect of number of recurrence in Universal Transformers. Perplexity after 1.5 epoch (15 sub-epochs in our setup) for (L, 8192, dres, 8) models.Param L dres 
Params. 
Perplexity 
Tying 
in M 
Train Dev 
Yes 
3 1024 
329 
95.3 90.5 
6 
87.9 83.5 
No 
512 
262 
74.7 73.3 
Table 8 :
8WERs (%) for hybrid systems on the LibriSpeech 960hr. 4-gram model is used in the first pass, and it generates lattices for rescoring. "Lattice" shows oracle WERs of the lattice.LM 
L 
Para. 
dev 
test 
in M 
clean 
other 
clean 
other 
PPL WER PPL WER PPL WER PPL WER 
4-gram -230 151.7 4.3 140.6 10.1 158.1 4.8 145.7 10.6 
Lattice --
-1.4 
-3.2 
-1.6 
-3.9 
LSTM 
2 487 66.9 3.1 66.2 7.6 71.7 3.5 68.2 8.2 
4 554 62.0 3.0 61.7 7.3 66.4 3.4 63.4 8.2 
Transfo. 
24 281 57.8 2.9 58.3 7.2 62.2 3.3 59.4 7.8 
42 338 54.5 2.9 55.5 7.1 59.1 3.3 56.4 7.7 
Table 9 :
9WERs (%) for attention-based models on LibriSpeech 960hr dataset. Perplexities are on the 10K BPE level.LM 
Beam 
dev 
test 
clean 
other 
clean 
other 
PPL WER PPL WER PPL WER PPL WER 
None 
12 
-
4.3 
-12.9 
-
4.4 
-13.5 
LSTM 64 
45.7 
3.0 48.3 
9.1 48.8 
3.5 48.9 10.0 
Transfo. 
40.2 
2.9 43.2 
8.8 43.0 
3.1 43.5 
9.8 
Table 10 :
10Effect of sinusoidal positional encoding. Perplexity after 5 epochs (13M updates) for (L, 2048, 512, 8) models.L 
Position. 
Params. 
Perplexity 
encoding 
in M. 
Train Dev Test 
12 
Sinusoidal 
243 
61.8 63.1 66.1 
None 
58.0 60.5 63.4 
24 
Sinusoidal 
280 
55.6 58.0 60.8 
None 
52.7 56.6 59.2 
Typically called position-wise feed-forward module[1]. Here we omit position-wise as it is obvious for autoregressive models.2 In principle, we could also consider an autoregressive self-attention model which updates states at all predecessor positions for each new input, which would be then much more computationally inefficient.
We will make training config files and models available at https: //github.com/rwth-i6/returnn-experiments/tree/master/2019-lm-transformers.
AcknowledgementsThis work has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 694537, project "SEQCLAS") and from a Google Focused Award. The work reflects only the authors' views and none of the funding parties is responsible for any use that may be made of the information it contains. We thanks Liuhui Deng for contributing to our lattice rescoring code based on Tensorflow C++ API, Arne Nix and Julian Schamper for sharing their base Transformer configs, and Eugen Beck and Christoph Lüscher for help with generating lattices.
Attention is all you need. A Vaswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A N Gomez, Ł Kaiser, I Polosukhin, Advances in Neural Information Processing Systems (NIPS). Long Beach, CA, USAA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, USA, Dec. 2017, pp. 5998-6008.
Long short-term memorynetworks for machine reading. J Cheng, L Dong, M Lapata, Proc. Conf. on Empirical Methods in Nat. Lang. Processing (EMNLP). Conf. on Empirical Methods in Nat. Lang. essing (EMNLP)Austin, TX, USAJ. Cheng, L. Dong, and M. Lapata, "Long short-term memory- networks for machine reading," in Proc. Conf. on Empirical Meth- ods in Nat. Lang. Processing (EMNLP), Austin, TX, USA, Nov. 2016, pp. 551-561.
A structured self-attentive sentence embedding. Z Lin, M Feng, C N Santos, M Yu, B Xiang, B Zhou, Y Bengio, Int. Conf. on Learning Representations (ICLR). Z. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, "A structured self-attentive sentence embedding," Int. Conf. on Learning Representations (ICLR), Apr. 2017.
A decomposable attention model for natural language inference. A P Parikh, O Täckström, D Das, J Uszkoreit, Proc. Conf. on Empirical Methods in Nat. Lang. Processing (EMNLP). Conf. on Empirical Methods in Nat. Lang. essing (EMNLP)Austin, TX, USAA. P. Parikh, O. Täckström, D. Das, and J. Uszkoreit, "A decom- posable attention model for natural language inference," in Proc. Conf. on Empirical Methods in Nat. Lang. Processing (EMNLP), Austin, TX, USA, Nov. 2016, pp. 2249-2255.
Deep residual learning for image recognition. K He, X Zhang, S Ren, J Sun, IEEE Conf. on Computer Vision and Patt. Recog. (CVPR). Las Vegas, NV, USAK. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in IEEE Conf. on Computer Vision and Patt. Recog. (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 770-778.
Layer normalization. J L Ba, J R Kiros, G E Hinton, arXiv:1607.06450arXiv preprintJ. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint arXiv:1607.06450, 2016.
Convolutional sequence to sequence learning. J Gehring, M Auli, D Grangier, D Yarats, Y N Dauphin, Proc. ICML. ICMLSydney, AustraliaJ. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, "Convolutional sequence to sequence learning," in Proc. ICML, Sydney, Australia, Aug. 2017, pp. 1243-1252.
Long short-term memory. S Hochreiter, J Schmidhuber, Neural computation. 98S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
BERT: Pretraining of deep bidirectional transformers for language understanding. J Devlin, M.-W Chang, K Lee, K Toutanova, Proc. North American Chap. of the Assoc. for Comput. Ling. on Human Lang. Tech. (NAACL-HLT). North American Chap. of the Assoc. for Comput. Ling. on Human Lang. Tech. (NAACL-HLT)Minneapolis, USAJ. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre- training of deep bidirectional transformers for language understand- ing," in Proc. North American Chap. of the Assoc. for Comput. Ling. on Human Lang. Tech. (NAACL-HLT), Minneapolis, USA, Jun. 2019.
Generating wikipedia by summarizing long sequences. P J Liu, M Saleh, E Pot, B Goodrich, R Sepassi, Ł Kaiser, N Shazeer, Int. Conf. on Learning Representations (ICLR). Vancouver, CanadaP. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Ł. Kaiser, and N. Shazeer, "Generating wikipedia by summarizing long se- quences," in Int. Conf. on Learning Representations (ICLR), Van- couver, Canada, Apr. 2018.
Transformer-XL: Attentive language models beyond a fixed-length context. Z Dai, Z Yang, Y Yang, J Carbonell, Q V Le, R Salakhutdinov, arXiv:1901.02860arXiv preprintZ. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhut- dinov, "Transformer-XL: Attentive language models beyond a fixed-length context," arXiv preprint arXiv:1901.02860, 2019.
Character-level language modeling with deeper self-attention. R Al-Rfou, D Choe, N Constant, M Guo, L Jones, Proc. AAAI Conf. on Artif. Int. AAAI Conf. on Artif. IntHonolulu, HI, USAR. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones, "Character-level language modeling with deeper self-attention," in Proc. AAAI Conf. on Artif. Int., Honolulu, HI, USA, Jan. 2019.
Adaptive input representations for neural language modeling. A Baevski, M Auli, ICLR. New Orleans, LA, USAA. Baevski and M. Auli, "Adaptive input representations for neural language modeling," in ICLR, New Orleans, LA, USA, May 2019.
Improving language understanding by generative pre-training. A Radford, K Narasimhan, T Salimans, I Sutskever, OnlineA. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, "Im- proving language understanding by generative pre-training," [On- line]. : https://blog.openai.com/language-unsupervised/, 2018.
Language models are unsupervised multitask learners. A Radford, J Wu, R Child, D Luan, D Amodei, I Sutskever, A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," [Online]. : https://blog.openai.com/better-language-models/, 2019.
LibriSpeech: an ASR corpus based on public domain audio books. V Panayotov, G Chen, D Povey, S Khudanpur, Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP). IEEE Int. Conf. on Acoustics, Speech and Signal essing (ICASSP)South Brisbane, Queensland, AustraliaV. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "LibriSpeech: an ASR corpus based on public domain audio books," in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Queensland, Australia, Apr. 2015, pp. 5206-5210.
Neural machine translation of rare words with subword units. R Sennrich, B Haddow, A Birch, Proc. ACL. ACLBerlin, GermanyR. Sennrich, B. Haddow, and A. Birch, "Neural machine translation of rare words with subword units," in Proc. ACL, Berlin, Germany, August 2016, pp. 1715-1725.
Listen, attend and spell: a neural network for large vocabulary conversational speech recognition. W Chan, N Jaitly, Q Le, O Vinyals, Proc. ICASSP. ICASSPShanghai, ChinaW. Chan, N. Jaitly, Q. Le, and O. Vinyals, "Listen, attend and spell: a neural network for large vocabulary conversational speech recognition," in Proc. ICASSP, Shanghai, China, Mar. 2016, pp. 4960-4964.
Improved training of end-to-end attention models for speech recognition. A Zeyer, K Irie, R Schlüter, H Ney, Proc. Interspeech. InterspeechHyderabad, IndiaA. Zeyer, K. Irie, R. Schlüter, and H. Ney, "Improved training of end-to-end attention models for speech recognition," in Proc. Interspeech, Hyderabad, India, Sep. 2018, pp. 7-11.
Lattice decoding and rescoring with long-span neural network language models. M Sundermeyer, Z Tüske, R Schlüter, H Ney, Interspeech. M. Sundermeyer, Z. Tüske, R. Schlüter, and H. Ney, "Lattice decoding and rescoring with long-span neural network language models," in Interspeech, Singapore, Sep. 2014, pp. 661-665.
On using monolingual corpora in neural machine translation. Ç Gülçehre, O Firat, K Xu, K Cho, L Barrault, H.-C Lin, F Bougares, H Schwenk, Y Bengio, Computer Speech & Language. 45Ç . Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H.-C. Lin, F. Bougares, H. Schwenk, and Y. Bengio, "On using monolin- gual corpora in neural machine translation," Computer Speech & Language, vol. 45, pp. 137-148, Sep. 2017.
A comparison of techniques for language model integration in encoder-decoder speech recognition. S Toshniwal, A Kannan, C.-C Chiu, Y Wu, T N Sainath, K Livescu, Proc. SLT. SLTAthens, GreeceS. Toshniwal, A. Kannan, C.-C. Chiu, Y. Wu, T. N. Sainath, and K. Livescu, "A comparison of techniques for language model integration in encoder-decoder speech recognition," in Proc. SLT, Athens, Greece, Dec. 2018.
Self-attention with relative position representations. P Shaw, J Uszkoreit, A Vaswani, Proc. NAACL. NAACLNew Orleans, LA, USAP. Shaw, J. Uszkoreit, and A. Vaswani, "Self-attention with relative position representations," in Proc. NAACL, New Orleans, LA, USA, Jun. 2018, pp. 464-468.
Rectified linear units improve restricted Boltzmann machines. V Nair, G E Hinton, Proc. Int. Conf. on Machine Learning (ICML). Int. Conf. on Machine Learning (ICML)Haifa, IsraelV. Nair and G. E. Hinton, "Rectified linear units improve restricted Boltzmann machines," in Proc. Int. Conf. on Machine Learning (ICML), Haifa, Israel, Jun. 2010, pp. 807-814.
Gaussian error linear units (GELUs). D Hendrycks, K Gimpel, arXiv:1606.08415arXiv preprintD. Hendrycks and K. Gimpel, "Gaussian error linear units (GELUs)," arXiv preprint arXiv:1606.08415, 2018.
Language modeling with gated convolutional networks. Y N Dauphin, A Fan, M Auli, D Grangier, Proc. ICML. ICMLSydney, AustraliaY. N. Dauphin, A. Fan, M. Auli, and D. Grangier, "Language modeling with gated convolutional networks," in Proc. ICML, Sydney, Australia, Aug. 2017, pp. 933-941.
LSTM neural networks for language modeling. M Sundermeyer, R Schlüter, H Ney, Proc. Interspeech. InterspeechPortland, OR, USAM. Sundermeyer, R. Schlüter, and H. Ney, "LSTM neural networks for language modeling." in Proc. Interspeech, Portland, OR, USA, Sep. 2012, pp. 194-197.
LSTM: A search space odyssey. K Greff, R K Srivastava, J Koutník, B R Steunebrink, J Schmidhuber, IEEE Trans. Neural Netw. Learn. Syst. 2810K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, "LSTM: A search space odyssey," IEEE Trans. Neural Netw. Learn. Syst., vol. 28, no. 10, pp. 2222-2232, 2017.
Tensorflow: A system for large-scale machine learning. M Abadi, P Barham, J Chen, Z Chen, A Davis, J Dean, M D , Proc. USENIX Sympo. on Operating Systems Design and Impl. (OSDI 16). USENIX Sympo. on Operating Systems Design and Impl. (OSDI 16)Savannah, GA, USAM. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, and M. D. et al., "Tensorflow: A system for large-scale machine learn- ing," in Proc. USENIX Sympo. on Operating Systems Design and Impl. (OSDI 16), Savannah, GA, USA, Nov. 2016, pp. 265-283.
RETURNN as a generic flexible neural toolkit with application to translation and speech recognition. A Zeyer, T Alkhouli, H Ney, Proc. Assoc. for Computational Linguistics (ACL). Assoc. for Computational Linguistics (ACL)Melbourne, AustraliaA. Zeyer, T. Alkhouli, and H. Ney, "RETURNN as a generic flexible neural toolkit with application to translation and speech recognition," in Proc. Assoc. for Computational Linguistics (ACL), Melbourne, Australia, Jul. 2018.
Breaking the softmax bottleneck: A high-rank RNN language model. Z Yang, Z Dai, R Salakhutdinov, W W Cohen, ICLR. Vancouver, CanadaZ. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen, "Breaking the softmax bottleneck: A high-rank RNN language model," in ICLR, Vancouver, Canada, Apr. 2018.
Prediction of LSTM-RNN full context states as a subtask for N-gram feedforward language models. K Irie, Z Lei, R Schlüter, H Ney, ICASSP. K. Irie, Z. Lei, R. Schlüter, and H. Ney, "Prediction of LSTM-RNN full context states as a subtask for N-gram feedforward language models," in ICASSP, Calgary, Canada, Apr. 2018, pp. 6104-6108.
Identity mappings in deep residual networks. K He, X Zhang, S Ren, J Sun, Proc. European Conf. on Computer Vision (ECCV). European Conf. on Computer Vision (ECCV)Amsterdam, NetherlandsK. He, X. Zhang, S. Ren, and J. Sun, "Identity mappings in deep residual networks," in Proc. European Conf. on Computer Vision (ECCV), Amsterdam, Netherlands, Oct. 2016, pp. 630-645.
Universal Transformers. M Dehghani, S Gouws, O Vinyals, J Uszkoreit, Ł Kaiser, Int. Conf. on Learning Representations (ICLR). New Orleans, LA, USAM. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser, "Universal Transformers," in Int. Conf. on Learning Representa- tions (ICLR), New Orleans, LA, USA, May 2019.
A comprehensive study of deep bidirectional lstm rnns for acoustic modeling in speech recognition. A Zeyer, P Doetsch, P Voigtlaender, R Schlüter, H Ney, Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP). IEEE Int. Conf. on Acoustics, Speech and Signal essing (ICASSP)New Orleans, LA, USAA. Zeyer, P. Doetsch, P. Voigtlaender, R. Schlüter, and H. Ney, "A comprehensive study of deep bidirectional lstm rnns for acoustic modeling in speech recognition," in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, Mar. 2017, pp. 2462-2466.
RWTH ASR systems for LibriSpeech: Hybrid vs Attention. C Lüscher, E Beck, K Irie, M Kitza, W Michel, A Zeyer, R Schlüter, H Ney, Graz, AustriaSubmitted to InterspeechC. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney, "RWTH ASR systems for LibriSpeech: Hybrid vs Attention," in Submitted to Interspeech 2019, Graz, Austria, Sep. 2019.
Sequence-tosequence speech recognition with time-depth separable convolutions. A Hannun, A Lee, Q Xu, R Collobert, arXiv:1904.02619arXiv preprintA. Hannun, A. Lee, Q. Xu, and R. Collobert, "Sequence-to- sequence speech recognition with time-depth separable convo- lutions," arXiv preprint arXiv:1904.02619, 2019.
Fully convolutional speech recognition. N Zeghidour, Q Xu, V Liptchinsky, N Usunier, G Synnaeve, R Collobert, arXiv:1812.06864arXiv preprintN. Zeghidour, Q. Xu, V. Liptchinsky, N. Usunier, G. Synnaeve, and R. Collobert, "Fully convolutional speech recognition," arXiv preprint arXiv:1812.06864, 2018.
Model unit exploration for sequence-to-sequence speech recognition. K Irie, R Prabhavalkar, A Kannan, A Bruguier, D Rybach, P Nguyen, arXiv:1902.01955preprintK. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. Nguyen, "Model unit exploration for sequence-to-sequence speech recognition," preprint arXiv:1902.01955, 2019.
The best of both worlds: Combining recent advances in neural machine translation. M X Chen, O Firat, A Bapna, M Johnson, W Macherey, G Foster, L Jones, M Schuster, N Shazeer, N Parmar, A Vaswani, J Uszkoreit, L Kaiser, Z Chen, Y Wu, M Hughes, ACL. M. X. Chen, O. Firat, A. Bapna, M. Johnson, W. Macherey, G. Fos- ter, L. Jones, M. Schuster, N. Shazeer, N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, Z. Chen, Y. Wu, and M. Hughes, "The best of both worlds: Combining recent advances in neural machine translation," in ACL, Melbourne, Australia, Jul. 2018, pp. 76-86.
Language Modeling with Deep Transformers
https://arxiv.org/abs/1905.04226
https://ieeexplore.ieee.org/abstract/document/9772392
MCWS-Transformers: Towards an Efficient Modeling of Protein Sequences via Multi Context-Window Based Scaled Self-Attention
2024-04-07
None
<div><div>
<div>
</div>
<div>
<div>
<div>
<h3>IEEE Account</h3>
<ul>
<li><a href="https://www.ieee.org/profile/changeusrpwd/showChangeUsrPwdPage.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Change Username/Password</a></li>
<li><a href="https://www.ieee.org/profile/address/getAddrInfoPage.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Update Address</a></li>
</ul>
</div>
<div>
<h3>Purchase Details</h3>
<ul>
<li><a href="https://www.ieee.org/profile/payment/showPaymentHome.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Payment Options</a></li>
<li><a href="https://www.ieee.org/profile/vieworder/showOrderHistory.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Order History</a></li>
<li><a href="https://ieeexplore.ieee.org/articleSale/purchaseHistory.jsp">View Purchased Documents</a></li>
</ul>
</div>
<div>
<h3>Profile Information</h3>
<ul>
<li><a href="https://www.ieee.org/ieee-privacyportal/app/ibp?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Communications Preferences</a></li>
<li><a href="https://www.ieee.org/profile/profedu/getProfEduInformation.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Profession and Education</a></li>
<li><a href="https://www.ieee.org/profile/tips/getTipsInfo.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Technical Interests</a></li>
</ul>
</div>
<div>
<h3>Need Help?</h3>
<ul>
<li><strong>US &amp; Canada:</strong> +1 800 678 4333</li>
<li><strong>Worldwide: </strong> +1 732 981 0060<br/>
</li>
<li><a href="https://ieeexplore.ieee.org/xpl/contact">Contact &amp; Support</a></li>
</ul>
</div>
</div>
<div>
<ul>
<li><a href="https://ieeexplore.ieee.org/Xplorehelp/about-ieee-xplore.html">About IEEE <em>Xplore</em></a></li>
<li><a href="https://ieeexplore.ieee.org/xpl/contact">Contact Us</a></li>
<li><a href="https://ieeexplore.ieee.org/Xplorehelp/Help_start.html">Help</a></li>
<li><a href="https://ieeexplore.ieee.org/Xplorehelp/accessibility-statement.html">Accessibility</a></li>
<li><a href="https://ieeexplore.ieee.org/Xplorehelp/Help_Terms_of_Use.html">Terms of Use</a></li>
<li><a href="http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html">Nondiscrimination Policy</a></li>
<li><a href="https://ieeexplore.ieee.org/xpl/sitemap.jsp">Sitemap</a></li>
<li><a href="http://www.ieee.org/about/help/security_privacy.html">Privacy &amp; Opting Out of Cookies</a></li>
</ul>
<p>
	A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.<br/>© Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.
	</p>
</div>
</div>
</div></div>||||I|||| IEEE Account

 * Change Username/Password
 * Update Address

 Purchase Details

 * Payment Options
 * Order History
 * View Purchased Documents

 Profile Information

 * Communications Preferences
 * Profession and Education
 * Technical Interests

 Need Help?

 * US & Canada: +1 800 678 4333
 * Worldwide: +1 732 981 0060
 * Contact & Support
 * About IEEE Xplore
 * Contact Us
 * Help
 * Accessibility
 * Terms of Use
 * Nondiscrimination Policy
 * Sitemap
 * Privacy & Opting Out of Cookies

 A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.
 © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

IEEE Account
 Change Username/Password 
 Update Address 
Purchase Details
 Payment Options 
 Order History 
 View Purchased Documents 
Profile Information
 Communications Preferences 
 Profession and Education 
 Technical Interests 
Need Help?
 US &amp; Canada: +1 800 678 4333
 Worldwide:  +1 732 981 0060
 Contact &amp; Support 
 About IEEE Xplore  
 Contact Us 
 Help 
 Accessibility 
 Terms of Use 
 Nondiscrimination Policy 
 Sitemap 
 Privacy &amp; Opting Out of Cookies 
	A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.
MCWS-Transformers: Towards an Efficient Modeling of Protein Sequences via Multi Context-Window Based Scaled Self-Attention
https://ieeexplore.ieee.org/abstract/document/9772392