arxiv paper about machine learning
https://arxiv.org/abs/1409.0473
Neural Machine Translation by Jointly Learning to Align and Translate
2014-09-01
Bahdanau; Dzmitry; Cho; Kyunghyun; Bengio; Yoshua
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/1409.0473">Download PDF</a></p><blockquote>
 Abstract: Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Dzmitry Bahdanau [<a href="/show-email/517fd701/1409.0473">view email</a>]
 <br />
 <strong><a href="/abs/1409.0473v1">[v1]</a></strong>
 Mon, 1 Sep 2014 16:33:02 UTC (83 KB)<br />
 <strong><a href="/abs/1409.0473v2">[v2]</a></strong>
 Thu, 4 Sep 2014 18:32:00 UTC (83 KB)<br />
 <strong><a href="/abs/1409.0473v3">[v3]</a></strong>
 Tue, 7 Oct 2014 18:10:39 UTC (84 KB)<br />
 <strong><a href="/abs/1409.0473v4">[v4]</a></strong>
 Fri, 19 Dec 2014 21:39:11 UTC (107 KB)<br />
 <strong><a href="/abs/1409.0473v5">[v5]</a></strong>
 Sun, 22 Mar 2015 17:08:39 UTC (107 KB)<br />
 <strong><a href="/abs/1409.0473v6">[v6]</a></strong>
 Fri, 24 Apr 2015 13:25:33 UTC (144 KB)<br /><strong>[v7]</strong>
Thu, 19 May 2016 21:53:22 UTC (144 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:1409.0473

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:1409.0473 (cs)
 [Submitted on 1 Sep 2014 (v1), last revised 19 May 2016 (this version, v7)]

 Title: Neural Machine Translation by Jointly Learning to Align and Translate

 Authors: Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
 Download PDF
 Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
 Comments: Accepted at ICLR 2015 as oral presentation  
 Subjects: Computation and Language (cs.CL) ; Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)
 Cite as: arXiv:1409.0473 [cs.CL]  
 (or arXiv:1409.0473v7 [cs.CL] for this version)  
 https://doi.org/10.48550/arXiv.1409.0473  
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: Dzmitry Bahdanau [view email]
 [v1] Mon, 1 Sep 2014 16:33:02 UTC (83 KB)
 [v2] Thu, 4 Sep 2014 18:32:00 UTC (83 KB)
 [v3] Tue, 7 Oct 2014 18:10:39 UTC (84 KB)
 [v4] Fri, 19 Dec 2014 21:39:11 UTC (107 KB)
 [v5] Sun, 22 Mar 2015 17:08:39 UTC (107 KB)
 [v6] Fri, 24 Apr 2015 13:25:33 UTC (144 KB)
 [v7] Thu, 19 May 2016 21:53:22 UTC (144 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 1409
 Change to browse by:
 cs
 cs.LG
 cs.NE
 stat
 stat.ML

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 28 blog links

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Dzmitry Bahdanau
 Kyunghyun Cho
 Yoshua Bengio
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs and how to get involved.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE
Date Published: 2016-05-19
Authors:
Dzmitry  Bahdanau
Kyunghyun  Cho
Yoshua  Bengio
Abstract
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and encode a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
1 INTRODUCTION
Neural machine translation is a newly emerging approach to machine translation, recently proposed by Kalchbrenner and Blunsom (2013) , Sutskever et al. (2014) and Cho et al. (2014b) . Unlike the traditional phrase-based translation system (see, e.g., Koehn et al., 2003) which consists of many small sub-components that are tuned separately, neural machine translation attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation.
Most of the proposed neural machine translation models belong to a family of encoderdecoders (Sutskever et al., 2014; Cho et al., 2014a) , with an encoder and a decoder for each language, or involve a language-specific encoder applied to each sentence whose outputs are then compared (Hermann and Blunsom, 2014 ). An encoder neural network reads and encodes a source sentence into a fixed-length vector. A decoder then outputs a translation from the encoded vector. The whole encoder-decoder system, which consists of the encoder and the decoder for a language pair, is jointly trained to maximize the probability of a correct translation given a source sentence.
A potential issue with this encoder-decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector. This may make it difficult for the neural network to cope with long sentences, especially those that are longer than the sentences in the training corpus. Cho et al. (2014b) showed that indeed the performance of a basic encoder-decoder deteriorates rapidly as the length of an input sentence increases.
In order to address this issue, we introduce an extension to the encoder-decoder model which learns to align and translate jointly. Each time the proposed model generates a word in a translation, it (soft-)searches for a set of positions in a source sentence where the most relevant information is concentrated. The model then predicts a target word based on the context vectors associated with these source positions and all the previous generated target words.
The most important distinguishing feature of this approach from the basic encoder-decoder is that it does not attempt to encode a whole input sentence into a single fixed-length vector. Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively while decoding the translation. This frees a neural translation model from having to squash all the information of a source sentence, regardless of its length, into a fixed-length vector. We show this allows a model to cope better with long sentences.
In this paper, we show that the proposed approach of jointly learning to align and translate achieves significantly improved translation performance over the basic encoder-decoder approach. The improvement is more apparent with longer sentences, but can be observed with sentences of any length. On the task of English-to-French translation, the proposed approach achieves, with a single model, a translation performance comparable, or close, to the conventional phrase-based system. Furthermore, qualitative analysis reveals that the proposed model finds a linguistically plausible (soft-)alignment between a source sentence and the corresponding target sentence.
2 BACKGROUND: NEURAL MACHINE TRANSLATION
From a probabilistic perspective, translation is equivalent to finding a target sentence y that maximizes the conditional probability of y given a source sentence x, i.e., arg max y p(y | x). In neural machine translation, we fit a parameterized model to maximize the conditional probability of sentence pairs using a parallel training corpus. Once the conditional distribution is learned by a translation model, given a source sentence a corresponding translation can be generated by searching for the sentence that maximizes the conditional probability.
Recently, a number of papers have proposed the use of neural networks to directly learn this conditional distribution (see, e.g., Kalchbrenner and Blunsom, 2013; Cho et al., 2014a; Sutskever et al., 2014; Cho et al., 2014b; Forcada and Ñeco, 1997) . This neural machine translation approach typically consists of two components, the first of which encodes a source sentence x and the second decodes to a target sentence y. For instance, two recurrent neural networks (RNN) were used by (Cho et al., 2014a) and (Sutskever et al., 2014) to encode a variable-length source sentence into a fixed-length vector and to decode the vector into a variable-length target sentence.
Despite being a quite new approach, neural machine translation has already shown promising results. Sutskever et al. (2014) reported that the neural machine translation based on RNNs with long shortterm memory (LSTM) units achieves close to the state-of-the-art performance of the conventional phrase-based machine translation system on an English-to-French translation task. 1 Adding neural components to existing translation systems, for instance, to score the phrase pairs in the phrase table (Cho et al., 2014a) or to re-rank candidate translations (Sutskever et al., 2014) , has allowed to surpass the previous state-of-the-art performance level.
2.1 RNN ENCODER-DECODER
Here, we describe briefly the underlying framework, called RNN Encoder-Decoder, proposed by Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns to align and translate simultaneously.
In the Encoder-Decoder framework, an encoder reads the input sentence, a sequence of vectors x = (x 1 , • • • , x Tx ), into a vector c. 2 The most common approach is to use an RNN such that
h t = f (x t , h t-1 ) (1) and c = q ({h 1 , • • • , h Tx })
, where h t ∈ R n is a hidden state at time t, and c is a vector generated from the sequence of the hidden states. f and q are some nonlinear functions. Sutskever et al. (2014) used an LSTM as f and
q ({h 1 , • • • , h T }) = h T , for instance.
The decoder is often trained to predict the next word y t given the context vector c and all the previously predicted words {y 1 , • • • , y t -1 }. In other words, the decoder defines a probability over the translation y by decomposing the joint probability into the ordered conditionals:
p(y) = T t=1 p(y t | {y 1 , • • • , y t-1 } , c),where y = y 1 , • • • , y Ty . With an RNN, each conditional probability is modeled as
p(y t | {y 1 , • • • , y t-1 } , c) = g(y t-1 , s t , c),where g is a nonlinear, potentially multi-layered, function that outputs the probability of y t , and s t is the hidden state of the RNN. It should be noted that other architectures such as a hybrid of an RNN and a de-convolutional neural network can be used (Kalchbrenner and Blunsom, 2013) .
3 LEARNING TO ALIGN AND TRANSLATE
In this section, we propose a novel architecture for neural machine translation. The new architecture consists of a bidirectional RNN as an encoder (Sec. 3.2) and a decoder that emulates searching through a source sentence during decoding a translation (Sec. 3.1).
3.1 DECODER: GENERAL DESCRIPTION In a new model architecture, we define each conditional probability in Eq. (2) as:
p(y i |y 1 , . . . , y i-1 , x) = g(y i-1 , s i , c i ),where s i is an RNN hidden state for time i, computed by
s i = f (s i-1 , y i-1 , c i ).
It should be noted that unlike the existing encoder-decoder approach (see Eq. ( 2)), here the probability is conditioned on a distinct context vector c i for each target word y i .
The context vector c i depends on a sequence of annotations (h 1 , • • • , h Tx ) to which an encoder maps the input sentence. Each annotation h i contains information about the whole input sequence with a strong focus on the parts surrounding the i-th word of the input sequence. We explain in detail how the annotations are computed in the next section.
The context vector c i is, then, computed as a weighted sum of these annotations h i :
c i = Tx j=1 α ij h j .The weight α ij of each annotation h j is computed by
α ij = exp (e ij ) Tx k=1 exp (e ik ) ,where e ij = a(s i-1 , h j ) is an alignment model which scores how well the inputs around position j and the output at position i match. The score is based on the RNN hidden state s i-1 (just before emitting y i , Eq. ( 4)) and the j-th annotation h j of the input sentence.
We parametrize the alignment model a as a feedforward neural network which is jointly trained with all the other components of the proposed system. Note that unlike in traditional machine translation, the alignment is not considered to be a latent variable. Instead, the alignment model directly computes a soft alignment, which allows the gradient of the cost function to be backpropagated through. This gradient can be used to train the alignment model as well as the whole translation model jointly.
We can understand the approach of taking a weighted sum of all the annotations as computing an expected annotation, where the expectation is over possible alignments. Let α ij be a probability that the target word y i is aligned to, or translated from, a source word x j . Then, the i-th context vector c i is the expected annotation over all the annotations with probabilities α ij .
The probability α ij , or its associated energy e ij , reflects the importance of the annotation h j with respect to the previous hidden state s i-1 in deciding the next state s i and generating y i . Intuitively, this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the encoder from the burden of having to encode all information in the source sentence into a fixedlength vector. With this new approach the information can be spread throughout the sequence of annotations, which can be selectively retrieved by the decoder accordingly.
3.2 ENCODER: BIDIRECTIONAL RNN FOR ANNOTATING SEQUENCES
The usual RNN, described in Eq. ( 1), reads an input sequence x in order starting from the first symbol x 1 to the last one x Tx . However, in the proposed scheme, we would like the annotation of each word to summarize not only the preceding words, but also the following words. Hence, we propose to use a bidirectional RNN (BiRNN, Schuster and Paliwal, 1997) , which has been successfully used recently in speech recognition (see, e.g., Graves et al., 2013) .
A BiRNN consists of forward and backward RNN's. The forward RNN -→ f reads the input sequence as it is ordered (from x 1 to x Tx ) and calculates a sequence of forward hidden states (
- → h 1 , • • • , - → h Tx ).
The backward RNN
←f reads the sequence in the reverse order (from x Tx to x 1 ), resulting in a sequence of backward hidden states ( ← -
h 1 , • • • , ← - h Tx ).
We obtain an annotation for each word x j by concatenating the forward hidden state -→ h j and the backward one ←h j , i.e., h j = -→ h j ; ←h j . In this way, the annotation h j contains the summaries of both the preceding words and the following words. Due to the tendency of RNNs to better represent recent inputs, the annotation h j will be focused on the words around x j . This sequence of annotations is used by the decoder and the alignment model later to compute the context vector (Eqs. ( 5)-( 6)).
See Fig. 1 for the graphical illustration of the proposed model.
4 EXPERIMENT SETTINGS
We evaluate the proposed approach on the task of English-to-French translation. We use the bilingual, parallel corpora provided by ACL WMT '14. 3 As a comparison, we also report the performance of an RNN Encoder-Decoder which was proposed recently by Cho et al. (2014a) . We use the same training procedures and the same dataset for both models.4 4.1 DATASET WMT '14 contains the following English-French parallel corpora: Europarl (61M words), news commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively, totaling 850M words. Following the procedure described in Cho et al. (2014a) , we reduce the size of the combined corpus to have 348M words using the data selection method by Axelrod et al. (2011) . 5We do not use any monolingual data other than the mentioned parallel corpora, although it may be possible to use a much larger monolingual corpus to pretrain an encoder. We concatenate news-test- After a usual tokenization6 , we use a shortlist of 30,000 most frequent words in each language to train our models. Any word not included in the shortlist is mapped to a special token ([UNK]). We do not apply any other special preprocessing, such as lowercasing or stemming, to the data.
4.2 MODELS
We train two types of models. The first one is an RNN Encoder-Decoder (RNNencdec, Cho et al., 2014a) , and the other is the proposed model, to which we refer as RNNsearch. We train each model twice: first with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then with the sentences of length up to 50 word (RNNencdec-50, RNNsearch-50).
The encoder and decoder of the RNNencdec have 1000 hidden units each. 7 The encoder of the RNNsearch consists of forward and backward recurrent neural networks (RNN) each having 1000 hidden units. Its decoder has 1000 hidden units. In both cases, we use a multilayer network with a single maxout (Goodfellow et al., 2013) hidden layer to compute the conditional probability of each target word (Pascanu et al., 2014) .
We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train each model. Each SGD update direction is computed using a minibatch of 80 sentences. We trained each model for approximately 5 days.
Once a model is trained, we use a beam search to find a translation that approximately maximizes the conditional probability (see, e.g., Graves, 2012; Boulanger-Lewandowski et al., 2013) . Sutskever et al. (2014) used this approach to generate translations from their neural machine translation model.
For more details on the architectures of the models and training procedure used in the experiments, see Appendices A and B.
5.1 QUANTITATIVE RESULTS
In Table 1 , we list the translation performances measured in BLEU score. It is clear from the table that in all the cases, the proposed RNNsearch outperforms the conventional RNNencdec. More importantly, the performance of the RNNsearch is as high as that of the conventional phrase-based translation system (Moses), when only the sentences consisting of known words are considered. This is a significant achievement, considering that Moses uses a separate monolingual corpus (418M words) in addition to the parallel corpora we used to train the RNNsearch and RNNencdec. One of the motivations behind the proposed approach was the use of a fixed-length context vector in the basic encoder-decoder approach. We conjectured that this limitation may make the basic encoder-decoder approach to underperform with long sentences. In Fig. 2 , we see that the performance of RNNencdec dramatically drops as the length of the sentences increases. On the other hand, both RNNsearch-30 and RNNsearch-50 are more robust to the length of the sentences. RNNsearch-50, especially, shows no performance deterioration even with sentences of length 50 or more. This superiority of the proposed model over the basic encoder-decoder is further confirmed by the fact that the RNNsearch-30 even outperforms RNNencdec-50 (see tokens when only the sentences having no unknown words were evaluated (last column).
5.2.1 ALIGNMENT
The proposed approach provides an intuitive way to inspect the (soft-)alignment between the words in a generated translation and those in a source sentence. This is done by visualizing the annotation weights α ij from Eq. ( 6), as in Fig. 3 . Each row of a matrix in each plot indicates the weights associated with the annotations. From this we see which positions in the source sentence were considered more important when generating the target word.
We can see from the alignments in Fig. 3 that the alignment of words between English and French is largely monotonic. We see strong weights along the diagonal of each matrix. However, we also observe a number of non-trivial, non-monotonic alignments. Adjectives and nouns are typically ordered differently between French and English, and we see an example in Fig. 3 We observe similar behaviors in all the presented cases in Fig. 3 . An additional benefit of the soft alignment is that it naturally deals with source and target phrases of different lengths, without requiring a counter-intuitive way of mapping some words to or from nowhere ([NULL]) (see, e.g., Chapters 4 and 5 of Koehn, 2010).
5.2.2 LONG SENTENCES
As clearly visible from Fig. 2 the proposed model (RNNsearch) is much better than the conventional model (RNNencdec) at translating long sentences. This is likely due to the fact that the RNNsearch does not require encoding a long sentence into a fixed-length vector perfectly, but only accurately encoding the parts of the input sentence that surround a particular word.
As an example, consider this source sentence from the test set:
An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
The RNNencdec-50 translated this sentence into:
Un privilège d'admission est le droit d'un médecin de reconnaître un patient à l'hôpital ou un centre médical d'un diagnostic ou de prendre un diagnostic en fonction de son état de santé.
The RNNencdec-50 correctly translated the source sentence until [a medical center]. However, from there on (underlined), it deviated from the original meaning of the source sentence. For instance, it replaced [based on his status as a health care worker at a hospital] in the source sentence with [en fonction de son état de santé] ("based on his state of health").
On the other hand, the RNNsearch-50 generated the following correct translation, preserving the whole meaning of the input sentence without omitting any details:
Un privilège d'admission est le droit d'un médecin d'admettre un patient à un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, selon son statut de travailleur des soins de santé à l'hôpital.
Let us consider another sentence from the test set:
This kind of experience is part of Disney's efforts to "extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important," he added.
The translation by the RNNencdec-50 is
Ce type d'expérience fait partie des initiatives du Disney pour "prolonger la durée de vie de ses nouvelles et de développer des liens avec les lecteurs numériques qui deviennent plus complexes. As with the previous example, the RNNencdec began deviating from the actual meaning of the source sentence after generating approximately 30 words (see the underlined phrase). After that point, the quality of the translation deteriorates, with basic mistakes such as the lack of a closing quotation mark.
Again, the RNNsearch-50 was able to translate this long sentence correctly:
Ce genre d'expérience fait partie des efforts de Disney pour "prolonger la durée de vie de ses séries et créer de nouvelles relations avec des publics via des plateformes numériques de plus en plus importantes", a-t-il ajouté.
In conjunction with the quantitative results presented already, these qualitative observations confirm our hypotheses that the RNNsearch architecture enables far more reliable translation of long sentences than the standard RNNencdec model.
In Appendix C, we provide a few more sample translations of long source sentences generated by the RNNencdec-50, RNNsearch-50 and Google Translate along with the reference translations.
6.1 LEARNING TO ALIGN
A similar approach of aligning an output symbol with an input symbol was proposed recently by Graves (2013) in the context of handwriting synthesis. Handwriting synthesis is a task where the model is asked to generate handwriting of a given sequence of characters. In his work, he used a mixture of Gaussian kernels to compute the weights of the annotations, where the location, width and mixture coefficient of each kernel was predicted from an alignment model. More specifically, his alignment was restricted to predict the location such that the location increases monotonically.
The main difference from our approach is that, in (Graves, 2013) , the modes of the weights of the annotations only move in one direction. In the context of machine translation, this is a severe limitation, as (long-distance) reordering is often needed to generate a grammatically correct translation (for instance, English-to-German).
Our approach, on the other hand, requires computing the annotation weight of every word in the source sentence for each word in the translation. This drawback is not severe with the task of translation in which most of input and output sentences are only 15-40 words. However, this may limit the applicability of the proposed scheme to other tasks.
6.2 NEURAL NETWORKS FOR MACHINE TRANSLATION
Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a fixed number of the preceding words, neural networks have widely been used in machine translation. However, the role of neural networks has been largely limited to simply providing a single feature to an existing statistical machine translation system or to re-rank a list of candidate translations provided by an existing system.
For instance, Schwenk (2012) proposed using a feedforward neural network to compute the score of a pair of source and target phrases and to use the score as an additional feature in the phrase-based statistical machine translation system. More recently, Kalchbrenner and Blunsom (2013) and Devlin et al. (2014) reported the successful use of the neural networks as a sub-component of the existing translation system. Traditionally, a neural network trained as a target-side language model has been used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al., 2006) .
Although the above approaches were shown to improve the translation performance over the stateof-the-art machine translation systems, we are more interested in a more ambitious objective of designing a completely new translation system based on neural networks. The neural machine translation approach we consider in this paper is therefore a radical departure from these earlier works. Rather than using a neural network as a part of the existing system, our model works on its own and generates a translation from a source sentence directly.
7 CONCLUSION
The conventional approach to neural machine translation, called an encoder-decoder approach, encodes a whole input sentence into a fixed-length vector from which a translation will be decoded. We conjectured that the use of a fixed-length context vector is problematic for translating long sentences, based on a recent empirical study reported by Cho et al. (2014b) and Pouget-Abadie et al. (2014) .
In this paper, we proposed a novel architecture that addresses this issue. We extended the basic encoder-decoder by letting a model (soft-)search for a set of input words, or their annotations computed by an encoder, when generating each target word. This frees the model from having to encode a whole source sentence into a fixed-length vector, and also lets the model focus only on information relevant to the generation of the next target word. This has a major positive impact on the ability of the neural machine translation system to yield good results on longer sentences. Unlike with the traditional machine translation systems, all of the pieces of the translation system, including the alignment mechanism, are jointly trained towards a better log-probability of producing correct translations.
We tested the proposed model, called RNNsearch, on the task of English-to-French translation. The experiment revealed that the proposed RNNsearch outperforms the conventional encoder-decoder model (RNNencdec) significantly, regardless of the sentence length and that it is much more robust to the length of a source sentence. From the qualitative analysis where we investigated the (soft-)alignment generated by the RNNsearch, we were able to conclude that the model can correctly align each target word with the relevant words, or their annotations, in the source sentence as it generated a correct translation.
Perhaps more importantly, the proposed approach achieved a translation performance comparable to the existing phrase-based statistical machine translation. It is a striking result, considering that the proposed architecture, or the whole family of neural machine translation, has only been proposed as recently as this year. We believe the architecture proposed here is a promising step toward better machine translation and a better understanding of natural languages in general.
One of challenges left for the future is to better handle unknown, or rare words. This will be required for the model to be more widely used and to match the performance of current state-of-the-art machine translation systems in all contexts.
A MODEL ARCHITECTURE
A.1 ARCHITECTURAL CHOICES The proposed scheme in Section 3 is a general framework where one can freely define, for instance, the activation functions f of recurrent neural networks (RNN) and the alignment model a. Here, we describe the choices we made for the experiments in this paper.
A.1.1 RECURRENT NEURAL NETWORK For the activation function f of an RNN, we use the gated hidden unit recently proposed by Cho et al. (2014a) . The gated hidden unit is an alternative to the conventional simple units such as an element-wise tanh. This gated unit is similar to a long short-term memory (LSTM) unit proposed earlier by Hochreiter and Schmidhuber (1997) , sharing with it the ability to better model and learn long-term dependencies. This is made possible by having computation paths in the unfolded RNN for which the product of derivatives is close to 1. These paths allow gradients to flow backward easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013a) . It is therefore possible to use LSTM units instead of the gated hidden unit described here, as was done in a similar context by Sutskever et al. (2014) .
The new state s i of the RNN employing n gated hidden units8 is computed by
s i = f (s i-1 , y i-1 , c i ) = (1 -z i ) • s i-1 + z i • si ,
where • is an element-wise multiplication, and z i is the output of the update gates (see below). The proposed updated state si is computed by
si = tanh (W e(y i-1 ) + U [r i • s i-1 ] + Cc i ) ,
where e(y i-1 ) ∈ R m is an m-dimensional embedding of a word y i-1 , and r i is the output of the reset gates (see below). When y i is represented as a 1-of-K vector, e(y i ) is simply a column of an embedding matrix E ∈ R m×K . Whenever possible, we omit bias terms to make the equations less cluttered.
The update gates z i allow each hidden unit to maintain its previous activation, and the reset gates r i control how much and what information from the previous state should be reset. We compute them by
z i = σ (W z e(y i-1 ) + U z s i-1 + C z c i ) , r i = σ (W r e(y i-1 ) + U r s i-1 + C r c i ) ,
where σ (•) is a logistic sigmoid function.
At each step of the decoder, we compute the output probability (Eq. ( 4)) as a multi-layered function (Pascanu et al., 2014) . We use a single hidden layer of maxout units (Goodfellow et al., 2013) and normalize the output probabilities (one for each word) with a softmax function (see Eq. ( 6)).
A.1.2 ALIGNMENT MODEL
The alignment model should be designed considering that the model needs to be evaluated T x × T y times for each sentence pair of lengths T x and T y . In order to reduce computation, we use a singlelayer multilayer perceptron such that
a(s i-1 , h j ) = v a tanh (W a s i-1 + U a h j ) ,
where W a ∈ R n×n , U a ∈ R n×2n and v a ∈ R n are the weight matrices. Since U a h j does not depend on i, we can pre-compute it in advance to minimize the computational cost.
A.2.1 ENCODER
In this section, we describe in detail the architecture of the proposed model (RNNsearch) used in the experiments (see . From here on, we omit all bias terms in order to increase readability.
The model takes a source sentence of 1-of-K coded word vectors as input x = (x 1 , . . . , x Tx ), x i ∈ R Kx and outputs a translated sentence of 1-of-K coded word vectors y = (y 1 , . . . , y Ty ), y i ∈ R Ky , where K x and K y are the vocabulary sizes of source and target languages, respectively. T x and T y respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
- → h i = (1 -- → z i ) • - → h i-1 + - → z i • - → h i , if i > 0 0 , if i = 0 where - → h i = tanh -→ W Ex i + - → U - → r i • - → h i-1 - → z i =σ -→ W z Ex i + - → U z - → h i-1 - → r i =σ -→ W r Ex i + - → U r - → h i-1 . E ∈ R m×Kx is the word embedding matrix. -→ W , -→ W z , -→ W r ∈ R n×m , - → U , - → U z , - → U r ∈ R n×n
are weight matrices. m and n are the word embedding dimensionality and the number of hidden units, respectively. σ(•) is as usual a logistic sigmoid function.
The backward states (
← - h 1 , • • • , ← -Tx ) are computed similarly. We share the word embedding matrix E between the forward and backward RNNs, unlike the weight matrices.
We concatenate the forward and backward states to to obtain the annotations (h 1 , h 2 , • • • , h Tx ), where
h i = - → h i ← - h iA.2.2 DECODER The hidden state s i of the decoder given the annotations from the encoder is computed by
s i =(1 -z i ) • s i-1 + z i • si , where si = tanh (W Ey i-1 + U [r i • s i-1 ] + Cc i ) z i =σ (W z Ey i-1 + U z s i-1 + C z c i ) r i =σ (W r Ey i-1 + U r s i-1 + C r c i )
E is the word embedding matrix for the target language. W, W z , W r ∈ R n×m , U, U z , U r ∈ R n×n , and C, C z , C r ∈ R n×2n are weights. Again, m and n are the word embedding dimensionality and the number of hidden units, respectively. The initial hidden state s 0 is computed by s
0 = tanh W s ← - h 1 , where W s ∈ R n×n .
The context vector c i are recomputed at each step by the alignment model:
c i = Tx j=1 α ij h j ,
normalized the L 2 -norm of the gradient of the cost function each time to be at most a predefined threshold of 1, when the norm was larger than the threshold (Pascanu et al., 2013b) . Each SGD update direction was computed with a minibatch of 80 sentences.
At each update our implementation requires time proportional to the length of the longest sentence in a minibatch. Hence, to minimize the waste of computation, before every 20-th update, we retrieved 1600 sentence pairs, sorted them according to the lengths and split them into 20 minibatches. The training data was shuffled once before training and was traversed sequentially in this manner.
In Tables 2 we present the statistics related to training all the models used in the experiments. Un privilège admettre est le droit d'un médecin d'admettre un patient dans un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, fondée sur sa situation en tant que travailleur de soins de santé dans un hôpital.
Source
This kind of experience is part of Disney's efforts to "extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important," he added.
Reference
Ce type d'expérience entre dans le cadre des efforts de Disney pour "étendre la durée de vie de ses séries et construire de nouvelles relations avec son public grâce à des plateformes numériques qui sont de plus en plus importantes", a-t-il ajouté.
RNNenc-50
Ce type d'expérience fait partie des initiatives du Disney pour "prolonger la durée de vie de ses nouvelles et de développer des liens avec les lecteurs numériques qui deviennent plus complexes. RNNsearch-50 Ce genre d'expérience fait partie des efforts de Disney pour "prolonger la durée de vie de ses séries et créer de nouvelles relations avec des publics via des plateformes numériques de plus en plus importantes", a-t-il ajouté. Google Translate Ce genre d'expérience fait partie des efforts de Disney à "étendre la durée de vie de sa série et construire de nouvelles relations avec le public par le biais des plates-formes numériques qui deviennent de plus en plus important", at-il ajouté.
Source
In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might constitute a "reasonable motive" that could lead to criminal charges being brought against the mayor.
Reference
En conférence de presse, jeudi, M. Blair a affirmé qu'il n'y avait rien dans cette vidéo qui puisse constituer des "motifs raisonnables" pouvant mener au dépôt d'une accusation criminelle contre le maire.
RNNenc-50
Lors de la conférence de presse de jeudi, M. Blair a dit qu'il n'y avait rien dans cette vidéo qui pourrait constituer une "motivation raisonnable" pouvant entraîner des accusations criminelles portées contre le maire. RNNsearch-50 Lors d'une conférence de presse jeudi, M. Blair a déclaré qu'il n'y avait rien dans cette vidéo qui pourrait constituer un "motif raisonnable" qui pourrait conduire à des accusations criminelles contre le maire. Google Translate Lors d'une conférence de presse jeudi, M. Blair a déclaré qu'il n'y avait rien dans cette vido qui pourrait constituer un "motif raisonnable" qui pourrait mener à des accusations criminelles portes contre le maire.
Table 3 : The translations generated by RNNenc-50 and RNNsearch-50 from long source sentences (30 words or more) selected from the test set. For each source sentence, we also show the goldstandard translation. The translations by Google Translate were made on 27 August 2014.
We mean by the state-of-the-art performance, the performance of the conventional phrase-based system without using any neural network-based component.
Although most of the previous works (see, e.g.,Cho et al., 2014a;Sutskever et al., 2014;Kalchbrenner and Blunsom, 2013) used to encode a variable-length input sentence into a fixed-length vector, it is not necessary, and even it may be beneficial to have a variable-length vector, as we will show later.
http://www.statmt.org/wmt14/translation-task.html
Implementations are available at https://github.com/lisa-groundhog/GroundHog.
Available online at http://www-lium.univ-lemans.fr/ ˜schwenk/cslm_joint_paper/.
We used the tokenization script from the open-source machine translation package, Moses.
In this paper, by a 'hidden unit', we always mean the gated hidden unit (see Appendix A.1.1).
Here, we show the formula of the decoder. The same formula can be used in the encoder by simply ignoring the context vector ci and the related terms.
References
[1]Axelrod A, He X, Gao J. Domain adaptation via pseudo in-domain data selection. Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) 2011;:355--362.
[2]Bastien F, Lamblin P, Pascanu R, Bergstra J, Goodfellow IJ, Bergeron A, et al.. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.  2012;:.
[3]Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 1994;5:157--166.
[4]Bengio Y, Ducharme R, Vincent P, Janvin C. A neural probabilistic language model. J. Mach. Learn. Res 2003;3:1137--1155.
[5]Bergstra J, Breuleux O, Bastien F, Lamblin P, Pascanu R, Desjardins G, et al.. Theano: a CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy) 2010;:.
[6]Boulanger-Lewandowski N, Bengio Y, Vincent P. Audio chord recognition with recurrent neural networks.  2013;:.
[7]Cho K, Van Merrienboer B, Gulcehre C, Bougares F, Schwenk H, Bengio Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Proceedings of the Empiricial Methods in Natural Language Processing 2014;:.
[8]Cho K, Van Merriënboer B, Bahdanau D, Bengio Y. On the properties of neural machine translation: Encoder-Decoder approaches. Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation 2014;:.
[9]Devlin J, Zbib R, Huang Z, Lamar T, Schwartz R, Makhoul J. Fast and robust neural network joint models for statistical machine translation.  2014;:.
[10]Forcada ML, Ñeco RP. Recursive hetero-associative memories for translation. Biological and Artificial Computation: From Neuroscience to Technology 1997;1240:453--462.
[11]Goodfellow I, Warde-Farley D, Mirza M, Courville A, Bengio Y. Maxout networks. Proceedings of The 30th International Conference on Machine Learning 2013;:1319--1327.
[12]Graves A. Sequence transduction with recurrent neural networks. Proceedings of the 29th International Conference on Machine Learning 2012;:.
[13]Graves A. Generating sequences with recurrent neural networks.  2013;:.
[14]Graves A, Jaitly N,  Mohamed,  A.-R. Hybrid speech recognition with deep bidirectional LSTM. Automatic Speech Recognition and Understanding (ASRU) 2013;:273--278.
[15]Hermann K, Blunsom P. Multilingual distributed representations without word alignment. Proceedings of the Second International Conference on Learning Representations 2014;:.
[16]Hochreiter S. Untersuchungen zu dynamischen neuronalen Netzen.  1991;:.
[17]Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation 1997;9:1735--1780.
[18]Kalchbrenner N, Blunsom P. Recurrent continuous translation models. Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) 2013;:1700--1709.
[19]Koehn P. Statistical Machine Translation.  2010;:.
[20]Koehn P, Och FJ, Marcu D. Statistical phrase-based translation. Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology 2003;1:48--54.
[21]Pascanu R, Mikolov T, Bengio Y. On the difficulty of training recurrent neural networks.  2013;:.
[22]Pascanu R, Mikolov T, Bengio Y. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning 2013;:.
[23]Pascanu R, Gulcehre C, Cho K, Bengio Y. How to construct deep recurrent neural networks. Proceedings of the Second International Conference on Learning Representations 2014;:.
[24]Pouget-Abadie J, Bahdanau D, Van Merriënboer B, Cho K, Bengio Y. Overcoming the curse of sentence length for neural machine translation using automatic segmentation. Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation 2014;:.
[25]Schuster M, Paliwal KK. Bidirectional recurrent neural networks. Signal Processing. IEEE Transactions on 1997;45:2673--2681.
[26]Schwenk H. Continuous space translation models for phrase-based statistical machine translation. Proceedings of the 24th International Conference on Computational Linguistics (COLIN) 2012;:1071--1080.
[27]Schwenk H, Dchelotte D, Gauvain J.-L. Continuous space language models for statistical machine translation. Proceedings of the COLING/ACL on Main conference poster sessions 2006;:723--730.
[28]Sutskever I, Vinyals O, Le Q. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems 2014;:.
[29]Zeiler MD. ADADELTA: An adaptive learning rate method.  2012;:.
NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE
https://arxiv.org/abs/1409.0473
https://arxiv.org/abs/2104.07857
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
2021-04-16
Rajbhandari; Samyam; Ruwase; Olatunji; Rasley; Jeff; Smith; Shaden; He; Yuxiong
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2104.07857">Download PDF</a></p><blockquote>
 Abstract: In the last three years, the largest dense deep learning models have grown
over 1000x to reach hundreds of billions of parameters, while the GPU memory
has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has
been supported primarily though system innovations that allow large models to
fit in the aggregate GPU memory of multiple GPUs. However, we are getting close
to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion
parameter model for training, and such clusters are simply out of reach for
most data scientists. In addition, training models at that scale requires
complex combinations of parallelism techniques that puts a big burden on the
data scientists to refactor their model.
<br />In this paper we present ZeRO-Infinity, a novel heterogeneous system
technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented
model scale on limited resources without requiring model code refactoring. At
the same time it achieves excellent training throughput and scalability,
unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models
with tens and even hundreds of trillions of parameters for training on current
generation GPU clusters. It can be used to fine-tune trillion parameter models
on a single NVIDIA DGX-2 node, making large models more accessible. In terms of
training throughput and scalability, it sustains over 25 petaflops on 512
NVIDIA V100 GPUs(40% of peak), while also demonstrating super linear
scalability. An open source implementation of ZeRO-Infinity is available
through DeepSpeed, a deep learning optimization library that makes distributed
training easy, efficient, and effective.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Samyam Rajbhandari [<a href="/show-email/31c2d88f/2104.07857">view email</a>]
 <br /><strong>[v1]</strong>
Fri, 16 Apr 2021 02:22:12 UTC (5,397 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2104.07857

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Distributed, Parallel, and Cluster Computing

 arXiv:2104.07857 (cs)
 [Submitted on 16 Apr 2021]

 Title: ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

 Authors: Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
 Download PDF
 Abstract: In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model.
 In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaflops on 512 NVIDIA V100 GPUs(40% of peak), while also demonstrating super linear scalability. An open source implementation of ZeRO-Infinity is available through DeepSpeed, a deep learning optimization library that makes distributed training easy, efficient, and effective.
 Subjects: Distributed, Parallel, and Cluster Computing (cs.DC) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Performance (cs.PF)
 Cite as: arXiv:2104.07857 [cs.DC]  
 (or arXiv:2104.07857v1 [cs.DC] for this version)  
 https://doi.org/10.48550/arXiv.2104.07857  
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: Samyam Rajbhandari [view email]
 [v1] Fri, 16 Apr 2021 02:22:12 UTC (5,397 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 Current browse context:
 cs.DC
 < prev | next >
 new | recent | 2104
 Change to browse by:
 cs
 cs.AI
 cs.LG
 cs.PF

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Samyam Rajbhandari
 Shaden Smith
 Yuxiong He
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs and how to get involved.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
16 Apr 2021
Samyam Rajbhandari samyamr@microsoft.com 
Olatunji Ruwase olruwase@microsoft.com 
Jeff Rasley jerasley@microsoft.com 
Shaden Smith shsmit@microsoft.com 
Yuxiong He 
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
16 Apr 2021
In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model.In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaflops on 512 NVIDIA V100 GPUs (40% of peak), while also demonstrating super linear scalability. An open source implementation of ZeRO-Infinity is available through DeepSpeed 1 .EXTENDED INTRODUCTIONDeep learning (DL) has made tremendous advances in recent years, allowing it to become an integral part of our lives from powering our search engines to our smart home virtual assistants. Increased model size is at the center of these advancements [1-3], and multiple studies have shown that this trend will continue[4,5]. As a result, there has been significant investment in training huge models.In the last three years, the largest trained dense model in deep learning has grown over 1000x, from one hundred million parameters (ELMo [6]) to over one hundred billion parameters ). In comparison, the single GPU memory has increased by a meager 5x (16 GB to 80 GB). Therefore, the growth in model size has been made possible mainly through advances in system technology for training large DL models, with parallelism technologies such as 1 DeepSpeed (https://www.deepspeed.ai/) is a deep learning optimization library designed to make distributed training easy, efficient, and effective. DeepSpeed has been extensively adopted by the DL community.Figure 1: ZeRO-Infinity can train a model with 32 trillion parameters on 32 NVIDIA V100 DGX-2 nodes (512 GPUs), 50x larger than 3D parallelism, the existing state-of-the-art.model parallelism[7], pipeline parallelism [8-10], and ZeRO[11,12]creating a path to training larger and more powerful models.The current state-of-the-art in large model training technology is three-dimensional parallelism (3D parallelism [13,14]), which combines model (tensor-slicing) and pipeline parallelism with data parallelism to efficiently scale DL training to trillions of parameters on hundreds or thousands of GPUs. For example, the DeepSpeed implementation of 3D parallelism can scale to over a trillion parameters on 800 NVIDIA V100 GPUs by fully leveraging the aggregate GPU memory of a cluster[15].Despite the capabilities of 3D parallelism for large model training, we are now arriving at the GPU memory wall[16]. The aggregate GPU memory is simply not large enough to support the growth in model size. Even with the newest NVIDIA A100 GPUs with 80 GB of memory, 3D parallelism requires 320 GPUs just to fit a trillionparameter model for training, and scaling to a hundred trillion parameter model of the future would require over 6K GPUs even if we assume a 5x increase in GPU memory in the next few years. We can no longer sustain the continuous growth in the model scale with GPU memory as the bottleneck.The GPU memory wall also limits data scientists from accessing even the large models of today, especially for fine tuning. Large models are first pretrained on large amounts of generic data, and through fine tuning the same model can be specialized for a wide variety of applications. While pretraining a model with hundreds of billions of parameters can require millions of GPU compute hours, fine-tuning it is much cheaper, requiring significantly fewer GPU compute hours, and could be done on a single compute node with a handful of GPUs. While such compute resources are accessible to many businesses and users, they are unfortunately restricted by the memory available on these compute nodes, which in turn limits the size of the model that can be fine tuned. It makes large model fine
In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model.
In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaflops on 512 NVIDIA V100 GPUs (40% of peak), while also demonstrating super linear scalability. An open source implementation of ZeRO-Infinity is available through DeepSpeed 1 .
EXTENDED INTRODUCTION
Deep learning (DL) has made tremendous advances in recent years, allowing it to become an integral part of our lives from powering our search engines to our smart home virtual assistants. Increased model size is at the center of these advancements [1][2][3], and multiple studies have shown that this trend will continue [4,5]. As a result, there has been significant investment in training huge models.
In the last three years, the largest trained dense model in deep learning has grown over 1000x, from one hundred million parameters (ELMo [6]) to over one hundred billion parameters (GPT-3 [4]). In comparison, the single GPU memory has increased by a meager 5x (16 GB to 80 GB). Therefore, the growth in model size has been made possible mainly through advances in system technology for training large DL models, with parallelism technologies such as Figure 1: ZeRO-Infinity can train a model with 32 trillion parameters on 32 NVIDIA V100 DGX-2 nodes (512 GPUs), 50x larger than 3D parallelism, the existing state-of-the-art.
model parallelism [7], pipeline parallelism [8][9][10], and ZeRO [11,12] creating a path to training larger and more powerful models.
The current state-of-the-art in large model training technology is three-dimensional parallelism (3D parallelism [13,14]), which combines model (tensor-slicing) and pipeline parallelism with data parallelism to efficiently scale DL training to trillions of parameters on hundreds or thousands of GPUs. For example, the DeepSpeed implementation of 3D parallelism can scale to over a trillion parameters on 800 NVIDIA V100 GPUs by fully leveraging the aggregate GPU memory of a cluster [15].
Despite the capabilities of 3D parallelism for large model training, we are now arriving at the GPU memory wall [16]. The aggregate GPU memory is simply not large enough to support the growth in model size. Even with the newest NVIDIA A100 GPUs with 80 GB of memory, 3D parallelism requires 320 GPUs just to fit a trillionparameter model for training, and scaling to a hundred trillion parameter model of the future would require over 6K GPUs even if we assume a 5x increase in GPU memory in the next few years. We can no longer sustain the continuous growth in the model scale with GPU memory as the bottleneck.
The GPU memory wall also limits data scientists from accessing even the large models of today, especially for fine tuning. Large models are first pretrained on large amounts of generic data, and through fine tuning the same model can be specialized for a wide variety of applications. While pretraining a model with hundreds of billions of parameters can require millions of GPU compute hours, fine-tuning it is much cheaper, requiring significantly fewer GPU compute hours, and could be done on a single compute node with a handful of GPUs. While such compute resources are accessible to many businesses and users, they are unfortunately restricted by the memory available on these compute nodes, which in turn limits the size of the model that can be fine tuned. It makes large model fine tuning inaccessible to most researchers and companies that do not have access to massive GPU clusters. For example fine-tuning GPT-3 would require over 8 DGX-2 nodes(128 GPUs) with 3D parallelism to just fit the model for training, even though a single DGX-2 node (16-GPUs) has enough compute to fine-tune it in a reasonable time.
In addition to the GPU memory wall, state-of-the-art for training massive models is also limited in terms of usability and flexibility. As discussed above, 3D parallelism requires combining data, model, and pipeline parallelism in sophisticated ways to get to hundreds of billions or trillions of parameters. While such a system can be very efficient, it requires data scientists to perform major model code refactoring, replacing single GPU operators with tensor-sliced versions, and splitting the model into load-balanced pipeline stages. This also makes 3D parallelism inflexible in the types of models that it can support. Models with complex dependencies cannot be easily converted into a load-balanced pipeline.
Given the landscape of large model training, 3 questions arise:
• Looking ahead, how do we support the next 1000x growth in model size, going from models like GPT-3 with 175 billion parameters to models with hundreds of trillions of parameters? • How can we make large models of today accessible to more data scientists who don't have access to hundreds to GPUs? • Can we make large model training easier by eliminating the need for model refactoring and multiple forms of parallelism?
In this paper, we take a leap forward from 3D parallelism and present ZeRO-Infinity, a novel system capable of addressing all the aforementioned challenges of large model training.
Unprecedented Model Scale ZeRO-Infinity extends the ZeRO family of technology [11,12] with new innovations in heterogeneous memory access called the infinity offload engine. This allows ZeRO-Infinity to support massive model sizes on limited GPU resources by exploiting CPU and NVMe memory simultaneously. In addition, ZeRO-Infinity also introduces a novel GPU memory optimization technique called memory-centric tiling to support extremely large individual layers that would otherwise not fit in GPU memory even one layer at a time. With the infinity offload engine and memory-centric tiling, ZeRO-Infinity not only supports the next 1000x growth in model size, but also makes large models accessible to data scientists with limited GPU resources.
Excellent Training Efficiency ZeRO-Infinity introduces a novel data partitioning strategy for leveraging aggregate memory bandwidth across all devices, which we refer to as bandwidth-centric partitioning, and combines it with powerful communication overlapcentric design, as well as optimizations for high performance NVMe access in the infinity offload engine. Together, ZeRO-Infinity offers excellent training efficiency, despite offloading data to CPU or NVMe, unencumbered by their limited bandwidth.
Ease of Use With ZeRO-Infinity, data scientists no longer have to adapt their model to multiple forms of parallelism like in 3D parallelism. This is possible due to memory-centric tiling in ZeRO-Infinity discussed above aimed at reducing GPU memory requirements of large individual layers that would otherwise require model parallelism (tensor-slicing) to fit the layers in GPU memory. In addition, ZeRO-Infinity eliminates the need for manual model code refactoring, even when scaling to trillions of parameters via an ease inspired implementation that automates all of the communication and data partitioning required for training arbitrary model architectures.
The main contributions of this paper are as follows:  [7,17,18] and pipeline parallelism (PP) [7][8][9] can split the model among processes, vertically and horizontally, respectively. 3D parallelism [14,15] combines data, model, and pipeline parallelism to leverage the merits of each, allowing it to scale to trillions of parameters efficiently. While 3D parallelism can be highly efficient, it requires i) significant model code refactoring to split the model into model and pipeline parallel components, ii) models with complex dependency graphs are difficult to be expressed into load-balanced pipeline stages and iii) the model size is limited by the total available GPU memory. We refer the reader to Ben-Nun and Hoefler [19] for a thorough survey on parallelism in DL.
•
ZeRO: Zero Redundancy Optimizer ZeRO [11] removes the memory redundancies across data-parallel processes by partitioning the three model states (i.e., optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing so, it boosts memory efficiency compared to classic data parallelism while retaining its computational granularity and communication efficiency. There are three stages in ZeRO corresponding to three model states: the first stage (ZeRO-1) partitions only the optimizer states, the second stage (ZeRO-2) partitions both the optimizer states and the gradients, and the final stage (ZeRO-3) partitions all three model states. In ZeRO-3, the parameters in each layer of the model are owned by a unique data parallel process. During the training, ZeRO-3 ensures that the parameters required for the forward or backward pass of an operator are available right before its execution by issuing broadcast communication collectives from the owner process. After the execution of the operator, ZeRO-3 also removes the parameters as they are no longer needed until the next forward or backward pass of the operator. Additionally, during the parameter update phase of training, ZeRO-3 ensures that each data-parallel process only updates the optimizer states corresponding to the parameters that it owns. Thus, ZeRO-3 can keep all the model states partitioned throughout the training except for the parameters that are required by the immediate computation.
Heterogeneous Training Approaches Out of several heterogeneous CPU memory based training approaches [20][21][22][23][24][25][26], ZeRO-Offload [12] is the state-of-the-art (SOTA) for large model training on multi-GPUs. ZeRO-Offload is built on top of ZeRO-2 and stores the gradients and the optimizer states in CPU memory. ZeRO-Offload leverages CPU memory in the absence of enough GPU devices to store the optimizer states and gradients. However, it still requires the parameters to be stored in GPU memory and replicated across all devices. Thus, the model scale with ZeRO-Offload is limited to the total number of parameters that the memory on a single GPU device can host. ZeRO-Offload also requires a large batch size to remain efficient due to suboptimal data partitioning and limited PCIe bandwidth. We address these limitations of ZeRO-Offload with ZeRO-Infinity. In terms of NVMe based approaches, Zhao et al. [27] use a hierarchical parameter server-based design to offload sparse parameters to SSD for creating a massive scale DL Ads System. In contrast, ZeRO-Infinity is designed to be a generic DL system for training massive dense models.
Reducing Activation Memory Activations are the intermediate results produced during the forward propagation that need to be retained to compute the gradients during backward propagation. Multiple efforts have focused on reducing the memory required by activations through compression [28], activation checkpointing [29,30], or live analysis [31]. ZeRO-Infinity works together with activation checkpointing to reduce activation memory.
Adam Optimizer and Mixed Precision Training Adaptive optimization methods [32][33][34][35] are crucial to achieving SOTA performance and accuracy for effective model training of large models. Compared to SGD, by maintaining fine-grained first-order and second-order statistics for each model parameter and gradient at the cost of significant memory footprint. Adam [33] is the optimizer used most prominently in large model training.
Large model training is generally trained in mixed precision, where the forward and backward propagation are done in FP16 and the parameter updates in FP32 [36]. This leverages the performance acceleration of the tensor core units available on modern GPUs [37].
MEMORY REQUIREMENTS
This section characterizes the memory requirements for DL training. While our methodology is generic, we focus the concrete analysis on Transformer [38] based architectures since all of the SOTA models with over a billion parameters follow that. Our analysis assumes mixed precision training with the Adam optimizer since this recipe is the de facto standard for training Transformer based models.
The memory required for training can be categorized into two components: i) Model states including optimizer states, gradients, and model parameters, ii) Residual states primarily referring to activation memory. To study training on heterogeneous resources, we also characterize the GPU working memory, describing the minimum amount of memory that must be available on the GPU to support training, assuming the model and residual states can be successfully offloaded from GPU memory.
Memory for Model States: The model states are comprised of optimizer states, gradients, and parameters. For mixed precision training with Adam optimizer, the parameters and gradients are stored in FP16 while the optimizer states consist of FP32 momentum, variance, parameters, and gradients. In total, each parameter requires 20 bytes of memory. The total number of parameters in a Transformer based model primarily depends on the hidden dimension (ℎ ) and the number of Transformer layers ( ). Nearly all the parameters in a Transformer block come from four linear layers within each block with sizes: (ℎ , 3ℎ ), (ℎ , ℎ ), (ℎ , 4ℎ ) and (4ℎ , ℎ ), respectively. Thus, the total parameters in a Transformer based model and can be approximated as
12 × × ℎ 2 (1)
requiring a total memory 240 × × ℎ 2 (2) in bytes to store the model states. Figure 2a column 5 shows the memory required to store the model states of a GPT-3 like Transformer based model with 100 billion to a 100 trillion parameters created by varying hidden dimension and number of layers. To put the memory requirements in context, Figure 2b column 3 shows the aggregate GPU memory available on a single NVIDIA V100 DGX-2 box as well as a DGX-2 SuperPOD cluster. Note that it requires 64 GPUs to just fit the model states for a 100B parameter model. Fitting a trillion parameter model requires over 512 GPUs, while a 10 trillion parameter model is beyond the scope of even a massive 1536 GPU cluster.
Memory for Residual States: The residual states primarily consist of the activation memory, which depends on the model architecture, batch size ( ) and sequence length ( ), and it can be quite large. On the positive side, the memory required for activation can be significantly reduced via activation checkpointing [29], which trades off activation memory at the expense of 0.33x additional recomputation. Large models such as Turing-NLG 17.2B and GPT-3 175B were all trained using activation checkpointing. The memory required to store activation checkpoints is estimated as bytes where is the number of Transformer blocks between two activation checkpoints, and × × ℎ is the size of the input to each Transformer block. Figure 2a column 7 shows the memory required to store activation checkpoints for batch size of 32 and sequence length of 1024 assuming we store one activation per Transformer block. Many modern GPU clusters have 8-16 GPUs per node, and so we chose a batch size of 2-4 per GPU, resulting in a batch size of 32 as a conservative estimate of activation within each node. While the resulting activation checkpoints are orders of magnitude smaller than the full set of activations (column 6) , beyond a trillion parameters they still get too large to fit in GPU memory for the batch size and sequence length under consideration.
2 × × × ℎ × /(3)
Model State Working Memory (MSWM) is the minimum amount of GPU memory required to perform forward or backward propagation on the largest single operator in the model after all the model states have been offload to CPU or NVMe. This is approximately given by the size of the parameters and gradients of that operator in the model, since there must be at least enough memory to hold the parameter and its gradient for backward propagation.
For a Transformer based model, the largest operator is a linear layer that transforms hidden states from ℎ to 4ℎ . The size of the parameter and gradients of this linear layer in bytes is
4 × ℎ × 4ℎ(4)
Note that MSWM (Figure 2a Column 8) grows significantly beyond a 100 billion parameters, requiring multiple gigabytes in contiguous memory, which can result in running out of memory during training due to lack of enough contiguous memory to satisfy these requirements. State-of-art approaches like 3D Parallelism, addresses this issue via model parallelism, by splitting individual operator across multiple GPUs. In Sec. 5.1.3, we discuss a novel approach for addressing these massive model state working memory without requiring model parallelism. Activation Working Memory (AWM) is the memory required in the backward propagation for recomputing the activations before performing the actual backward propagation. This is the size of the activations between two consecutive activation checkpoints. For example, if we create one activation checkpoint per Transformer block, the memory is given by the size of the total activation per Transformer block. This is given in bytes by approximately
× × × (16 × ℎ + 2 × _ℎ × ).(5)
Figure 2a column 8 shows that AWM gets large beyond 10 trillion parameters, even with = 1. Unlike MSWM that is only composed of a single parameter and gradient, AWM is composed of dozens of activations, and does not cause memory issues due to lack of contiguous memory as long as the total AWM can fit in GPU memory.
BANDWIDTH REQUIREMENTS
A critical question of offloading to CPU and NVMe memory is whether their limited bandwidth will hurt training efficiency. This section characterizes the impact of bandwidth on training efficiency.
We start from defining an efficiency metric. Assuming a workload execution without any compute and communication overlap, we can use the peak computational throughput ( ), data movement bandwidth ( ) and its arithmetic intensity ( ) to estimate the training efficiency.
The arithmetic intensity (AIT) of a workload is the ratio between the total computation and the data required by the computation. It describes the amount of computation per data movement. Higher AIT means a lower requirement on the data movement bandwidth, since for each data loaded the accelerator can do more computations. The efficiency metric can derived as follows:
_ = _ = _ _ _ _ = _ _ = _ × = _ _ + _
The efficiency can be written as a function of , and :
= × × +(6)
We will use this simple efficiency equation to characterize the data movement bandwidth required for training massive models. But before that, we will first quantify for DL training workloads.
Quantifying AIT in DL training
Model states and activation checkpoints can have varying . We can quantify them by first identifying the total computation in each iteration of DL training, and then identifying the data movement volume for each of the model states and activations.
Total Computation per Iteration The total computation per iteration is dominated by the computation in the linear layers of the Transformer. For the forward propagation this can be approximated as a function of the number of parameters, sequence length, and batch size, given by 2 × × × . The cost of backward propagation is approximately twice that of forward propagation. Additionally, activation checkpointing requires an additional forward computation as part of recomputation during backward propagation. Therefore, the total computation per iteration is:
_ _ = 2 × 4 × × × (7) = 2 × 4 × 12 × × × × ℎ 2 (8)
AIT w.r.t. Parameters and Gradients During forward and back propagation, model parameters must be loaded from the source location to GPU registers at least twice, i) during forward, ii) during the actual backward, resulting in a data movement of 2× . In presence of activation checkpointing, the parameters may be loaded one additional time for re-computation during the backward pass, adding another 1 × . Furthermore, the gradients must be stored from the GPU registers to its final location at least once, adding a final 1 × in data movement. Therefore, assuming that parameters and gradients are stored at the same final location, the total data movement during the forward and backward pass would be 4 × , i.e. 2 × 4 × in bytes. The total computation per iteration is given by Sec. 4.1. Therefore the w.r.t parameter and gradients is × .
AIT w.r.t. Optimizer States During the optimizer step, the optimizer states must be read at least once, and the optimizer states must be written at least once. So the total data movement is 2 × _ , which is approximately 2×16× bytes. The total computation per iteration is given by Sec. 4.1. Therefore w.r.t optimizer states during a full training iteration is
× /4.(10)
AIT w.r.t. Activation Checkpoints During the forward propagation activation checkpoints must be saved to their final location, and must be retrieved during the backward propagation. Therefore, the total data movement w.r.t activation checkpoints in bytes is given by 2× _ _ ℎ _ _ which is given by 4 × / × ℎ × × from Eq. (3). The total computation per iteration is given by Sec. 4.1. So the w.r.t activation checkpoints is given by
24 × ℎ × .(11)
Bandwidth Requirements
Due to the variation in the AIT, model states and activation checkpoints have very different bandwidth requirements to achieve good efficiency. The former only depends on the batch size and sequence length, while the latter only depends on the frequency of activation checkpoints and hidden dimension size of the model. Besides AIT, the bandwidth requirement for efficiency also depends on , as shown in Eq. (6). Using , and we first show how efficiency varies with bandwidth w.r.t to different model and residual states, and then discuss the bandwidth requirements on these states for DL training to be efficient. Our methodology is generic and can be applied to understanding the bandwidth requirements on any current or future generation clusters. Here, we use NVIDIA V100 DGX-2 SuperPOD cluster as our example platform.
Using the expression from Sec. 4.1 and efficiency metric based on Eq. (6), Figure 3 shows the relationship between efficiency and available bandwidth w.r.t. parameter and gradients, optimizer states, and activation checkpoints. To produce these plots, we computed the based on expressions derived in Sec. 4.1, for varying batch sizes, sequence length and model configurations. More specifically, we use a sequence length of 1024, the same sequence length used for GPT-2 [2], Megatron-LM [7], and Turing-NLG [39]. We vary batch size range from 1 to 16 to capture large GPU and small GPU experiments, respectively. A small batch size per GPU is used when running on large number of GPUs, while a large batch size per GPU is used when training on relatively fewer GPUs to maintain a reasonable effective batch size for training. Our hidden size ranges from 8K-64K representing models with hundreds of billions of parameters, to tens of trillions of parameters as shown in Figure 2a.
To identify for this analysis, we use an empirical approach 4 . We ran models with aforementioned configurations on a single NVIDIA V100 DGX-2 box with all non-GPU communication turned off to simulate a virtually unlimited bandwidth scenario. The performance achieved ranged from 62-78 TFlops/GPU based on the hidden size of 8K-64K, respectively. We used the average of 70 TFlops/GPU to represent for the purpose of this analysis 5 .
Bandwidth w.r.t. Parameter and Gradients Figure 3a shows that with a bandwidth of over 70 GB/s for parameter and gradients, we can achieve over 50% efficiency for even the smallest batch size. At this bandwidth, the data movement in theory can be completely overlapped with the computation to achieve a 100% efficiency.
Bandwidth w.r.t. Optimizer States Figure 3b shows that optimizer states require nearly 4x higher bandwidth to achieve 50% efficiency compared to parameters and gradients. Furthermore, the optimizer states are updated at the end of the forward and backward propagation and cannot be overlapped with the computation. As a result they require significantly larger bandwidth to keep the overall DL workload efficient. For example achieving 90% efficiency with batch size of 2 per GPU requires nearly 1.5 TB/s of effective bandwidth, which is greater than even the GPU memory bandwidth.
Bandwidth w.r.t. activation memory Figure 3c also shows that with activation checkpointing enabled, a meager bandwidth of 2 GB/s is able to sustain over 50% efficiency even for a hidden size of 2 . The bandwidth requirement drops down to less than 1 GB/s once the hidden size grows over 8 .
ZERO-INFINITY DESIGN OVERVIEW
In this section we present an overview of the design choices in ZeRO-Infinity that enable it to achieve unprecedented model scale while offering excellent training efficiency and ease of use. A bird's eye view of ZeRO-Infinity is illustrated in Figure 4 and discussed below.
Design for Unprecedented Scale
Modern GPU clusters are highly heterogeneous in terms of memory storage. In addition to the GPU memory, they have CPU memory as well as massive NVMe storage that is over 50x larger than the GPU memory and nearly 20x larger than CPU memory (See Fig. 2b).
We developed ZeRO-Infinity, a parallel system for DL training that can transcend the GPU memory wall by exploiting these heterogeneous memory systems in modern GPU clusters. Figure 1 compares the maximum achieved model size of 3D parallelism and ZeRO-Infinity. ZeRO-Infinity supports one trillion parameters per NVIDIA V100 DGX-2 node, a 50x increase over 3D parallelism.
Infinity offload engine for model states.
ZeRO-Infinity is built on top of ZeRO-3 [11] which partitions all model states to remove memory redundancy as discussed in Sec. 2. Unlike any of the existing ZeRO family of technology, ZeRO-Infinity is designed with a powerful offload mechanism called the infinity offload engine which can offload all of the partitioned model states to CPU or NVMe memory, or keep them on the GPU based on the memory requirements. Note from Fig. 2a and Fig. 2b, even the model states required by a 100 trillion parameter model can fit in the aggregate NVMe memory of a DGX-2 cluster with 96 nodes (1536 GPUs). Therefore, the infinity offload engine allows ZeRO-Infinity to fit model states of models with hundreds of trillions of parameters. See Sec. 6 for more details.
5.1.2 CPU Offload for activations. In addition to model states, ZeRO-Infinity can offload activation memory to CPU memory, when necessary. Note that the activation checkpoints (0.76 TB) required by a 10 trillion parameter model can easily fit in the 1.5TB Figure 4: A snapshot of ZeRO-Infinity training a model with two layers on four data parallel (DP) ranks. Communication for the backward pass of the first layer is depicted. Partitioned parameters are moved from slow memory to GPU and then collected to form the full layer. After gradients are computed, they are aggregated, repartitoned, and then offloaded to slow memory. Layers are denoted with subscripts and DP ranks are denoted with superscripts. For example, (2) 0 is the portion of layer 0's parameters owned by (2) .
of CPU memory available on a DGX-2 system, while the 3 TBs of activation checkpoints required by a 100 trillion parameter is within reach of the CPU memory of the next generation hardware. Therefore, by offloading activation checkpoints to CPU memory, ZeRO-Infinity can fit the activation checkpoints of models with hundreds of trillions of parameters.
Memory-centric tiling for working memory.
To reduce the working memory requirements of DL training for large models, ZeRO-Infinity introduces a novel technique called memory-centric tiling that exploits the data fetch and release pattern of ZeRO-3 to reduce the working memory requirements by breaking down a large operator into smaller tiles that can be executed sequentially.
For example, to reduce the working memory for a large linear operator, ZeRO-Infinity represents the operator as a mathematically equivalent sequence of smaller linear operators consisting of tiles of parameters from the original operator, and executes them sequentially. When combined with ZeRO-3, the parameter and gradients of each tile can be fetched and released one at a time, reducing the working memory proportional to the number of tiles. Therefore, ZeRO-Infinity can support operators of arbitrary sizes, without relying on model parallelism to fit them in limited GPU memory.
Design for Excellent Training Efficiency
Offloading all model states and activations to CPU or NVMe is only practical if ZeRO-Infinity can achieve high efficiency despite the offload. In reality this is extremely challenging since CPU memory is an order of magnitude slower than GPU memory bandwidth, while the NVMe bandwidth is yet another order of magnitude slower than the CPU memory bandwidth. Furthermore, reading and writing to these memory from GPU is even slower (see Fig. 2b).
On a system like the DGX-2, the bandwidth must be greater than 70GB/s, 1.5TB/s, and 1-4 GB/s w.r.t. parameter and gradients, optimizer states, and activation checkpoints, respectively for DL training to be efficient, based on our analysis in Sec. 4. Here we discuss how ZeRO-Infinity achieves the necessary bandwidths to achieve excellent efficiency.
Efficiency w.r.t Parameter and
Gradients. The data movement bandwidth for parameters and gradients must be greater than 70GB/s, close to the GPU-GPU bandwidth available on DGX-2 clusters [40]. Therefore, a DL parallel training solution like ZeRO-3 [11] where parameters are broadcasted from the owner GPU to the rest before using them in forward or backward propagation can run efficiently as long as the communication is overlapped.
On the contrary, a meager 12 GB/s PCIe bandwidth from a single GPU to CPU memory or NVMe (see Fig. 2b) or vice-versa is simply not sufficient to support heterogeneous training at scale 6 . Therefore, existing heterogeneous solutions like ZeRO-Offload where the parameters must be first moved from CPU to owner GPU before broadcasting requires significantly large batch sizes per GPU to achieve enough necessary to be efficient under the limited bandwidth. This poses two problems: i) for massive models the activation memory will get too large to fit even in CPU memory, and ii) the effective batch size becomes too large when scaling to hundreds or thousands of GPUs for effective convergence.
ZeRO-Infinity addresses these challenges in two ways: i) bandwidthcentric partitioning: a novel data mapping and parallel data retrieval strategy for offloaded parameters and gradients that allows ZeRO-Infinity to achieve virtually unlimited heterogeneous memory bandwidth (details in Sec. 6.1), and ii) an overlap centric design that allows ZeRO-Infinity to overlap not only GPU-GPU communication with computation but also NVMe-CPU and CPU-GPU communications over the PCIe (details in Sec. 5.1.3).
Efficiency w.r.t Optimizer States.
Unlike parameters and gradients that are consumed and produced sequentially during the forward and backward propagation, optimizer states can be updated in parallel, all at once. This property is leveraged by both ZeRO-3 and ZeRO-Offload, that store and update the optimizer states in GPU and CPU memory, respectively, in parallel across all available GPUs and CPUs. As a result the aggregate GPU or CPU memory bandwidth can get much higher than the required 1.5TB/s with increase in GPU or CPU count.
Since ZeRO-Infinity is built upon ZeRO-3, it can also leverage the aggregate GPU and CPU memory bandwidth as well as the aggregate CPU compute for optimizer step, when offloading optimizer states to CPU memory. However, with NVMe offload, it is necessary to bring the data from NVMe to CPU memory and back in chunks that can fit in the CPU memory to perform the optimizer step, one chunk at a time. The optimizer step is therefore limited by the NVMe-CPU memory bandwidth: while ZeRO-Infinity can achieve aggregate NVMe bandwidth across multiple nodes, it is crucial to achieve near peak NVMe bandwidth per node, to allow supporting the necessary bandwidth of over 1.5 TB/s with as few nodes, and as small batch size as possible. Furthermore, the process of bringing data in and out of NVMe to CPU memory, or from CPU memory to GPU memory can cause CPU memory fragmentation 6 CPU and NVMe bandwidth are in the order of 100 GB/s and 25 GB/s, respectively, but reading data from CPU or NVMe to a single GPU is limited by the achievable PCIe bandwidth which is around 10-12 GB/s in both GPU and CPU that can result in out of memory even with plenty of memory still available.
The infinity offload engine can not only achieve near peak NVMe bandwidth, it can also allows ZeRO-Infinity to overlap NVMe to CPU reads with CPU to NVMe writes, as well as the CPU computation for the optimizer step at the same time to allow ZeRO-Infinity to remain efficient with a modest batch size on small number of GPUs and with small batch sizes on large numbers of GPUs. At the same time, it minimizes memory fragmentation by carefully reusing temporary buffers for data movement. We discuss the optimizations in infinity offload engine and in detail in Sec. 6.
Efficiency w.r.t Activations.
On a DGX-2 node, each GPU can read and write data at about 3 GB/s to CPU memory in parallel over the PCIe allowing activation checkpoints to be offloaded to CPU memory while retaining over 80% efficiency for hidden size larger 8 or larger. To also allow for high efficiency at smaller hidden sizes, ZeRO-Infinity can decrease the frequency of activation checkpoints as well as effectively overlap the communication of activation checkpoints both to and from CPU memory with the forward and backward computation on the GPU.
Design for Ease of Use
With ZeRO-Infinity, data scientists no longer have to adapt their model to multiple forms of parallelism like in 3D parallelism. This is possible due to memory-centric tiling in ZeRO-Infinity discussed in Sec. 5.1.3 aimed at reducing GPU memory requirements of large individual layers that would otherwise require model parallelism (tensor-slicing) to fit the layers in GPU memory.
In addition, ZeRO-Infinity is implemented in PyTorch in a way that eliminates the need for manual model code refactoring even when scaling to trillions of parameters. This is made possible through an ease-inspired implementation with two automated features: i) automated data movement to gather and partition parameters right before and after they are required during the training. ZeRO-Infinity does this by injecting i) pre forward/backward hooks into PyTorch submodules that trigger allgather collectives to collect the parameters required before its forward/backward pass and ii) post forward/backward hooks that trigger parameter/gradient partiitoning and optionally offloading them to CPU or NVMe (see Sec. 7.1 for details). ii) automated model partitioning during initialization such that models that can not fit within single GPU or CPU memory can still be initialized without requiring manual partitioning of the model across data parallel processes. ZeRO-Infinity achieves this by wrapping the constructor of all module classes so that parameters of each submodule are partitioned and offloaded immediately after they are created during initialization. The entire model is never fully instantiated on a single data parallel process (see Sec. 7.2 for details).
EFFICIENCY OPTIMIZATIONS
In this section, we deep dive into the optimizations introduced in Sec. 5 that allow ZeRO-Infinity to achieve excellent efficiency.
Bandwidth-Centric Partitioning
ZeRO-Infinity implements a novel data mapping and retrieval strategy to address the NVMe and CPU memory bandwidth limitations. Unlike ZeRO [11] and ZeRO-Offload [12], where parameters of each layer are owned by a single data parallel process, which broadcasts them to the rest when needed, ZeRO-Infinity partitions individual parameters across all the data parallel process, and uses an allgather instead of a broadcast when a parameter needs to be accessed. Note that both broadcast and allgather communication collectives have the same communication cost when it comes to data movement volume if the data is located on the GPU. Therefore, this makes no difference for a GPU-only training. However, this is a game changer when the data is located in NVMe or CPU.
In the broadcast-based approach, since each parameter is fully owned by one of the data parallel processes, the parameter must be first communicated from its source location to the GPU memory via the PCIe before the broadcast can happen. Note that only a single PCIe can be active for this process, while all the PCIe links connected to all the other GPUs are idle. On the contrary, with the partitioned parameter and allgather based approach in ZeRO-Infinity, all PCIe links are active in parallel, each bringing in 1/ ℎ portion of the parameter where is the data parallel degree. As a result, the effective communication bandwidth between NVMe or CPU to the GPU, increases linearly with the degree. For example, with broadcast-based approach, the CPU/NVMe to GPU bandwidth stays constant at about 12 GB/s with PCIe Gen 3, even with 16-way data parallelism on the DGX-2 box. However, with the all-gather-based approach, the effective achievable bandwidth increases to about 48/25 GB/s (3.0/1.6 GB/s per GPU), respectively (see Fig. 2b), limited only by the max aggregate PCIe bandwidth and max NVMe bandwidth per DGX-2 node. From here, the bandwidth grows linearly with more nodes. When training a massive model at massive scale, ZeRO-Infinity can therefore offer significantly more heterogeneous memory bandwidth than necessary (virtually unlimited) for the training to remain efficient. For example, on 64 DGX-2 nodes, ZeRO-Infinity has access to over 3TB/s of CPU memory bandwidth and over 1.5TB/s of NVMe bandwidth.
Overlap Centric Design
While ZeRO-Infinity can leverage sufficient heterogeneous memory bandwidth on a multi-node setup, the bandwidth can still be a bottleneck on a single GPU or single node setup. Even the GPU-GPU allgather communication has a big impact on efficiency when running with a small batch size (Fig. 3). Furthermore, accessing NVMe memory requires a three step process: i) read data from NVMe to CPU memory (nc-transfer), ii) copy the data from CPU memory to GPU memory (cg-transfer), iii) execute allgather to construct the full parameter on all GPUs (gg-transfer). The sequential nature of these data movements means that if done naively, the total communication time would be the sum of each of these three data movement cost, resulting in poor efficiency even if the bandwidth for data movement at each of these stages is individually sufficient.
To address these issues, ZeRO-Infinity has an overlap engine that not only overlaps GPU-GPU communication with GPU computation, but also overlaps the NVMe to CPU, and CPU to GPU communication, all at the same time. The overlap engine has two components: i) A dynamic prefetcher for overlapping the data movement required to reconstruct parameters before they are consumed in the forward or backward pass, and ii) a communication and offload overlapping mechanism for executing the data movement required by gradients in parallel with the backward computation.
The dynamic prefetcher in ZeRO-Infinity traces the forward and backward computation on that fly, constructing an internal map of the operator sequence for each iteration. During each iteration, the prefetcher keeps track of where it is in the operator sequence and prefetches the parameter requires by the future operators. The prefetcher is aware of the three step communication process, and therefore can overlap the nc-transfer for one parameter, with cgtransfer and gg-transfer of other parameters. For instance, before executing the ℎ operator, the prefetcher can invoke nc, cg, and gg-transfer for parameters required by + 3, + 2, and + 1 operators, respectively. Note that all of these data movement can happen in parallel with the execution of the ℎ operator. Furthermore, ZeRO-Infinity can update the operator sequence map in case of dynamic workflow, allowing for appropriate prefetching even when the forward and backward propagation changes across iterations.
Similarly, in the backward pass, ZeRO-Infinity can overlap the reduce-scatter for gradients of the parameters in ( + 1) ℎ operator with the computation of the ℎ operator, while simultaneous transferring the partitioned gradients from the reduce-scatter of the gradients of the ( + 2) ℎ operator to the CPU or NVMe.
With this powerful overlap centric design, ZeRO-Infinity hides significant portions of data movement even when training with a small number of GPUs and small batch size per GPU.
Infinity Offload Engine
The infinity offload engine is composed of two main components:
DeepNVMe, a powerful C++ NVMe read/write library in the infinity offload engine that supports bulk read/write requests for asynchronous completion, and explicit synchronization requests to flush ongoing read/writes. The support for asynchrony allows ZeRO-Infinity to overlap these requests with GPU/GPU or GPU/CPU communication or computation.
Most importantly, DeepNVMe is capable of achieving near peak sequential read and write bandwidths on the NVMe storage device. It achieves this high performance through a number of optimizations, including aggressive parallelization of I/O requests (whether from a single user thread or across multiple user threads), smart work scheduling, avoiding data copying, and memory pinning.
Pinned memory management layer To ensure high performance tensor reads (or writes) from (to) NVMe/CPU storage, the source (or destination) tensors must reside in pinned memory buffers. However, pinned memory buffers are scarce system resources, and their oversubscription by a single process can degrade overall system performance or cause system instability. This layer manages the limited supply of pinned memory by reusing a small amount (tens of GBs) for offloading the entire model states (up to tens of TBs) to CPU or NVMe. The reuse of memory buffer prevents memory fragmentation in CPU and GPU memory. This layer also provides PyTorch tensors with pinned memory data, allowing inplace computation of the tensors so that they can then be written to NVMe without any further copies to improve bandwidth.
ZeRO-Infinity is implemented on top of PyTorch, and is designed to be used without any model code refactoring similar to standard data-parallel training in PyTorch. This section details some of the challenges faced in implementing such a system.
Automating Data Movement
ZeRO-Infinity must coordinate the movement of tensors comprising the model parameters, gradients, and optimizer states. When a tensor is not in active use, it remains partitioned among workers and potentially offloaded to CPU or NVMe memory. The system must ensure that the tensors are resident in GPU memory in time for use and then later re-partitioned.
PyTorch models are expressed as a hierarchy of modules that represent the layers of a neural network. For example, Transformer architectures [38] contain submodules such as self-attention and feedforward networks. The self-attention submodules are further comprised of linear transformations and other submodules.
ZeRO-Infinity recursively injects hooks into the submodules of a model to automate the required data movement. At the start of a submodule's forward pass, these hooks ensure that the submodule's parameters are available for computations, otherwise it will execute the appropriate allgather collectives and block until the parmaeters become available. The overlap-centric design detailed in Sec. 6.2 is critical to minimizing stalls due to parameter communication. At the end of the submodule's forward pass, we partition the parameters again and optionally offload them. The backward pass is handled in a similar fashion.
Auto Registration of External Parameters.
In the ideal case, a submodule's parameters and gradients are only accessed within its own forward and backward passes, making it straight forward to identify and automate the data movement as discussed in section above. However, some model architectures are exceptions, where parameters defined and allocated in a submodule is used in forward and backward propagation of a different submodule. For example, language models such as GPT [41] share the weights of the embedding layer at both the beginning and the end of the network to map words to vectors and vice versa. We refer to the parameters that are used across module boundaries as external parameters. In presence of external parameters, it is difficult to know which parameters to gather at the beginning of a submodule's forward and backward pass.
One way to address this is to register external parameters with ZeRO-Infinity so that they are collected for the forward and backward passes of the submodule that access them. After registration, an external parameter is treated like all others and will be included in the prefetching system as described in Sec. 6.2. We provide APIs for manual registration of external parameters.
In order to improve user experience, we also provide mechanisms to detect these scenarios and automatically register external parameters so the the user does not have to make any code change:
Intercepting partitioned parameter accesses PyTorch modules store their tensor parameters in a hash table. At the initialization time, we replace the hash table with a subclassed type that overrides the tensor accesses. When a partitioned parameter is accessed, we do a blocking allgather on the parameter, register it as an external parameter, and then return the gathered parameter.
Activation introspection A submodule may return a parameter from its forward pass to be consumed by another submodule's forward and backward passes. For example, Megatron-LM returns bias vectors from the forward pass of linear layers and they are consumed by the parent Transformer layer modules. We inspect the activation outputs returned from each submodule's forward pass for partitioned parameters. If one is discovered, we collect and register it as an external parameter.
Automatic Model Partitioning during Initialization
If the model is large, then it may not be possible to fully initialize the model with traditional data parallel approach, replicating it on each data parallel process before it can be partitioned for ZeRO-Infinity. For example, a 500 billion parameter model will occupy 1 TB of memory in half precision, and thus a system with 8 GPUs per node requires 8 TB of aggregate CPU or GPU memory just for the initial data parallel allocation step. This is beyond the GPU or CPU memory available on a node. To address this limitation, the parameters corresponding to each layer of the model must be partitioned at the time of initialization, and not after the entire model is initialized. To do this, we provide a Python ZeRO-Infinity context which decorates the __init__ method of torch.nn.Module, so that parameters allocated under each module/sub-module are partitioned immediately after its initialization among the group of data parallel processes.
As a result, only individual sub-modules are fully initialized before they are partitioned, and the full model is never replicated on all the data parallel process. In the example above, the 500 billion parameter model can therefore be fully partitioned during its initialization requiring only 1 TB of aggregate CPU memory regardless of the total number of data parallel process.
EVALUATION
This section evaluates ZeRO-Infinity, demonstrating that it achieves excellent training efficiency and scalability for models with tens of trillion parameters. We also show the impact of various technologies within ZeRO-Infinity on model scale and performance.
Methodology
Hardware. We conducted our experiments on a cluster of up to 512 V100 SXM3 32 GB GPUs (32 DGX-2 nodes) with 800 Gbps internode communication bandwidth.
Baseline. For experiments without model parallelism (mp), we use torch's distributed data parallel (DDP [42]) as a baseline. For experiments with model parallelism, we use Megatron-LM [7]. As a baseline for each experiment we use the relevant state-of-the-art method among 3D Parallelism [13], ZeRO [11], or ZeRO-Offload [12].
Model Configurations. We use GPT-like Transformer based models. We fix the sequence length to 1024 and vary the hidden dimension and number of layers to obtain models with different number of parameters. Table 1   used throughout our evaluation, (see Appendix A) for additional configurations.
Model Size and Speed
Model Size ZeRO-Infinity trains models over 32 trillion parameters compared to about 650B parameters with 3D parallelism, state of the art, offering a leap of 50x in model scale ( Figure 1). Model Speed Figure 5a shows the performance of ZeRO-Infinity on up to 20 trillion parameter models on 512 GPUs. For the 500B model (close to the largest that 3D parallelism can run on these resources), ZeRO-Infinity and 3D parallelism achieve nearly identical throughput, indicating ZeRO-Infinity is on par with the training efficiency of the state-of-the-art. When increasing the model size further, 3D parallelism simply runs out of memory, while ZeRO-Infinity trains up to 20 trillion parameter models (40x larger) with excellent throughput of up to 49 TFlops/GPU. At the extreme-scale, Figure 5a shows a performance drop from 10T (43 TFlops/GPU), and 20T (34 TFlops/GPU). This drop is not due to NVMe bandwidth as both model sizes use NVMe offload, but instead due to an extremely small batch size per GPU (Table 1) at 20T scale as a result of limited CPU memory to store activation checkpoints. This can be improved by increasing the CPU memory or offloading activation checkpoints to NVMe in a future implementation.  Figure 5c shows performance of training 10B to 1T models on a single node (16 GPUs) with ZeRO-Infinity without any model parallelism. With models up to 100 billion parameters, ZeRO-Infinity achieves excellent performance of over 40 TFlops/GPU, making it possible to fine-tuning models such as GPT-3 with just a single DGX-2 box. In contrast, 3D parallelism is unable to scale to models with over 20 billion parameters. These results demonstrate two aspects of ZeRO-Infinity: i) Accessibility to fine-tuning large models with up to a trillion parameters on a single NVIDIA DGX-2 node, empowering users without access to large GPU clusters. ii) Ease-of-Use: Models at this scale can be trained using ZeRO-Infinity without combining model or pipeline parallelism, or requiring model code refactoring, making it easy for data scientists to scale up their models.
Superlinear Scalability
Democratizing Large Model Training
Impact of System Features on Model Scale
We show the impact of different device placement strategies on model scale and impact of memory-centric tiling (Sec. 5.1.3) on maximum hidden size using a single DGX-2 system (16 GPUs).
Maximum model size Figure 6a shows the effect of different device placement and partitioning strategies (see Table 2) on maximum model size. By using data parallelism alone we're limited to only 1.4B parameters, due to limited GPU memory and significant    model state redundancies. As we introduce optimizer/gradient partitioning and offloading to CPU with ZeRO-2 and ZeRO-Offload, we are able to scale up 9x to 13B parameters on a single node. Partitioning and offloading parameter states to CPU in ZeRO-Infinity allows us to almost reach 100B parameters. However, the final major jump in scale comes from offloading model states to NVMe which finally gets us to 1T parameters, resulting in a 700x increase in model size relative to data parallelism alone.
Maximum Hidden Size We evaluate the impact of memorycentric tiling in enabling large hidden sizes in the presence of memory fragmentation. We train a single layer transformer model with different hidden sizes and tiling factors to identify the largest hidden size that can be trained with and without tiling. To keep memory fragmentation consistent across all the experiments, we pre fragment the total GPU memory into 2 GB contiguous chunks so that all memory allocation requests larger than 2GB will fail. Figure 6b shows the largest hidden size that can be trained without memory-centric tiling is 8K, while we can even train a massive hidden size of 64K using memory-centric tiling factor of 16. With memory-centric tiling, ZeRO-Infinity greatly simplifies DL system stack by avoiding the need for model parallelism, making it easy for data scientists to train with large hidden sizes.
Impact of System Features on Performance
We evaluate the effects of the infinity offload engine (Sec. 5), bandwidthcentric partitioning (Sec. 6.1), overlap-centric design (Sec. 6.2), and activation checkpoint offloading (Sec. 4.1) on training speed.
ZeRO-Infinity vs ZeRO-Offload Figure 6c shows the impact of offloading gradients to CPU memory with ZeRO-Infinity vs ZeRO-Offload on the back propagation time of an 8B parameter model. ZeRO-Infinity leverages the aggregate PCIe bandwidth across GPUs to offload the gradients, resulting in a speedup of nearly 2x at 64 GPUs compared to ZeRO-Offload which is limited by single PCIe bandwidth.
Prefetching and Overlapping Figure 6d shows the relative throughput difference with communication overlapping and prefetching turned on and off for an 8B parameter model with 64 GPUs. The figure shows that prefetching and overlapping are crucial to achieving good performance at small batch sizes per GPU, while its impact diminishes at large batch sizes.
Activation checkpoint offload Figure 6e shows that CPU offloading of activation checkpoints in ZeRO-Infinity reduces the training throughput by up to 1.2x for small hidden sizes, but for hidden sizes 32K and 64K, the impact is minimal, demonstrating that it is possible to offload activation checkpoints to CPU memory without impacting efficiency for large hidden sizes.  Table 3: Bandwidth (bw) requirements for ZeRO-Infinity to remain efficient on a cluster of 512 accelerator devices with 10x and 100x more achievable compute than NVIDIA V100 GPUs.
CONCLUSION & FUTURE IMPLICATIONS
In this paper, we presented ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale that is accessible and easy to use while achieving excellent efficiency. It offers a paradigm shift in how we think about memory for large model training. It is no longer necessary to fit DL training on ultra-fast but expensive and limited memory like HBM2. ZeRO-Infinity demonstrates that it is possible to transcend the GPU memory wall by leveraging cheap and slow, but massive, CPU or NVMe memory in parallel across multiple devices to achieve the aggregate bandwidth necessary for efficient training on current generation of GPU clusters.
As we look into the future, the GPUs and other accelerators will become more powerful, and this aggregate bandwidth required for efficient training will also increase. Table 3 shows that even when the compute of the accelerators increases by 10x compared to the NVIDIA V100 GPUs, on a cluster with 512 of them, ZeRO-Infinity only requires a bandwidth of 30 GB/s between each accelerator and the slow memory to remain efficient. In fact, this is already possible with today's technology by connecting accelerators to slow memory via NVLink [43]. For example, the Summit Supercomputer launched in 2018 [44] connects NVIDIA V100 GPUs with the CPU memory at 40GB/s per GPU.
It is clear that with ZeRO-Infinity, accelerator device memory is no longer a limitation on model scale or training efficiency. However, training models with tens or hundreds of trillions of parameters in a reasonable time still requires massive leaps in compute, and running efficiently on these future devices requires a proportional leap in device-to-device bandwidth ( Table 3).
We hope that, with device memory no longer a limitation, ZeRO-Infinity will inspire a more compute and device-device bandwidth focused innovations of ultra-powerful accelerator devices and supercomputing clusters in the future to support the next 1000x growth in model scale and the advancements that they can offer.         Figure 6(e)
Figure 2 :
2(a) Memory requirements for massive models. (b) Available memory and achievable bandwidth on NVIDIA V100 DGX-2 Cluster (The reported bandwidths represent per GPU bandwidth when all GPUs are reading data in parallel from the designated memory).
Figure 3 :
3Impact of bandwidth on efficiency assuming an accelerator with 70 TFlops of single GPU peak achievable throughput.
Figure 5 :
5Efficiency and scalability of ZeRO-Infinity for training multi-trillion parameter models.
Figure 6 :
6Impact of system features on model scale and performance.
Figure 6(b) Hidden size Number of GPUs MP Layers Model size Attention head Batch size Total batch size 8192 16 1 1 900M 16 1 16 16384 16 1 1 3B 16 1 16 32768 16 1 1 13B 16 1 16 65536 16 1 1 50B 32 1 16
Figure 6(c) Number of GPUs Hidden size MP Layers Model size Attention head Batch size Total batch size [4,16,32,64] 8192 1 10 8B 16 2 [8,32,64,128]
Memory and performance characterization for large model trainpetaflops in throughput on the same hardware, iii) superlinear scalability of a trillion parameter model, iv) accessibility and ease-of-use: fine-tune up to a trillion parameter model on a single DGX-2 node, without using any model parallelism or model code refactoring, and v) impact of different technologies in ZeRO-Infinity on model-scale and efficiency (Sec. 8).• A discussion of ZeRO-Infinity and its potential implications of future hardware system design (Sec. 9) • An open source implementation of ZeRO-Infinity in DeepSpeed 2 , a deep learning optimization library for making distributed training easy, efficient, and effective training that has been extensively adopted by the DL community.Data, Model, Pipeline and 3D Parallelism Parallelization is an important strategy for training large models at scale. For a model that fits in the device memory for training, data parallelism (DP) can be used to scale training to multiple devices. When models do not fit in device memory, model parallelism 3 (MP)ing that describes the memory requirements (Sec. 3) for different 
components of a large model training as well as their bandwidth 
requirements (Sec. 4) for the training to be efficient. 
• ZeRO-Infinity (Sec. 5, 6 & Sec. 7): A novel DL training system 
technology consisting five innovative technologies to address the 
memory and bandwidth requirements for offering unprecedented 
model scale that is accessible and easy to use while achieving 
excellent training efficiency: i) infinity offload engine to fully 
leverage heterogeneous architecture on modern clusters by simul-
taneously exploiting GPU, CPU and NVMe memory, and GPU 
and CPU compute, ii) memory-centric tiling to handle massive 
operators without requiring model parallelism, iii) bandwidth-
centric partitioning for leveraging aggregate memory band-
width across all parallel devices, iv) overlap-centric design for 
overlapping compute and communication, v) ease-inspired im-
plementation to avoid model code refactoring. 
• An extensive evaluation of ZeRO-Infinity demonstrating: i) un-
precedented scale running 32 trillion parameters on 32 NVIDIA 
DGX-2 nodes (512 V100 GPUs), ii) excellent training efficiency 
achieving over 25 2 BACKGROUND AND RELATED WORK 
provides the specific model configurations# nodes # params 
hidden dim 
# layers 
batch/GPU 
mp 
fp16 param Opt State 
1 
10 B 
4K 
50 
8 
1 
GPU 
GPU 
1 
50, 100 B 
8K 
62, 125 
26, 24 
1 
CPU 
NVMe 
1 
0.5, 1 T 
18K, 25K 
124, 128 
8, 7 
1 
NVMe 
NVMe 
32 
0.5, 1 T 
18K, 25K 
124, 128 
7, 5 
4 
GPU 
GPU 
32 
5, 10, 20 T 48K, 64K, 88K 174, 200, 205 
3, 2, 1.25 
4, 4, 8 
NVMe 
NVMe 
Table 1: Experiment configurations. Sizes are expressed in B for billions, T for trillions, and K for 1024. 
(a) ZeRO-Infinity efficiently trains 40x larger 
models than 3D parallelism on 512 GPUs. 
(b) ZeRO-Infinity exceeds linear scaling from 
64 to 512 GPUs for a 1T parameter model. 
(c) ZeRO-Infinity can train up to 1T model on 
a DGX-2 node without model parallelism. 
Figure 5bshows that ZeRO-Infinity achieves super-linear scalability from 4 nodes (64 GPUs) to 32 nodes (512 GPUs) when training a 1T model. This is a weak scaling result where we keep batch size per node as constant and increase total batch size with increased number of nodes. ZeRO-Infinity exceeds perfect linear scaling by effectively leveraging the linear increase in aggregate PCIe and NVMe bandwidth to accelerate the offloading of parameters and optimizer states, and leveraging CPU compute from additional nodes for parameter update. In addition, ZeRO-Infinity already achieves over 2.8 petaflops (44 Tflops/GPU) with just 4 nodes, demonstrating that the aggregated NVMe bandwidth is sufficient to achieve good efficiency even at a modest scale.
ZeRO-Inf-NVMe [NVMe,CPU,GPU] / ✓ [NVMe,CPU,GPU] / ✓Name 
Optimizer + Grad 
(devices/partitioned) 
Parameters 
(devices/partitioned) 
Data parallel 
[GPU] / ✗ 
[GPU] / ✗ 
ZeRO 2 
[GPU] / ✓ 
[GPU] / ✗ 
ZeRO-Offload 
[CPU,GPU] / ✓ 
[GPU] / ✗ 
3D Parallelism 
[GPU] / ✓ 
[GPU] / ✓ 
ZeRO 3 
[GPU] / ✓ 
[GPU] / ✓ 
ZeRO-Inf-CPU 
[CPU, GPU] / ✓ 
[CPU,GPU] / ✓ 
Table 2 :
2Device placement options and partitioning strategies for optimizer, gradient, and parameter states.
Table
size Number of GPUs MP Layers Model size Attention head Batch size Total batch size8192 
16 
1 
1 
900M 
16 
1 
16 
16384 
16 
1 
1 
3B 
16 
1 
16 
32768 
16 
1 
1 
13B 
16 
1 
16 
65536 
16 
1 
1 
50B 
32 
1 
16 
Table
Table 6 :
6Model configurations for Figure 6(c)Figure 6(d) 
Table 7 :
7Model configurations for Figure 6(d) Figure 6(e) Hidden size Number of GPUs Opt Device MP Layers Model size Attention head Batch size Total batch size2048 
32 
CPU 
1 
5 
275M 
16 
4 
128 
8192 
32 
CPU 
1 
5 
4B 
16 
4 
128 
16384 
32 
CPU 
1 
5 
16B 
16 
4 
128 
32768 
32 
CPU 
1 
5 
64B 
16 
4 
128 
65536 
64 
NVMe 
1 
5 
260B 
16 
4 
128 
Table 8 :
8Model configurations for
DeepSpeed (https://www.deepspeed.ai/) is a deep learning optimization library designed to make distributed training easy, efficient, and effective. DeepSpeed has been extensively adopted by the DL community.
https://www.deepspeed.ai/ 3 In this paper, we make a distinction between model parallelism and pipeline parallelism, where the former is limited specifically to mean tensor-slicing based approaches, and does not include pipeline parallelism.
Note that is not the theoretical hardware peak, but instead the achievable peak in the absence of any communication bottleneck.5 Results will vary based on the value of used, and this analysis is a single data point, meant as a guide for understanding relationship between efficiency and bandwidth for DL workloads specifically on the NVIDIA V100 DGX-2 clusters. Furthermore, the result only considers the relationship between efficiency and bandwidth of model states and activations, one at a time, assuming infinite bandwidth for others to isolate the bandwidth requirement for each state separately.
ACKNOWLEDGEMENTWe thank Elton Zheng, Reza Yazdani Aminabadi, Arash Ashari for their help on improving various components of the code, and Cheng Li for her help in proof reading the paper. We thank Andrey Proskurin, Gopi Kumar, Junhua Wang, Mikhail Parakhin, and Rangan Majumder for their continuous support.A APPENDIX
BERT: pretraining of deep bidirectional transformers for language understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, abs/1810.04805CoRRJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
Language models are unsupervised multitask learners. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Exploring the limits of transfer learning with a unified text-to-text transformer. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
Language models are few-shot learners. Benjamin Tom B Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Askell, arXiv:2005.14165arXiv preprintTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
Scaling laws for neural language models. Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
E Matthew, Mark Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer, arXiv:1802.05365Deep contextualized word representations. arXiv preprintMatthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
Megatron-lm: Training multi-billion parameter language models using model parallelism. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick Legresley, Jared Casper, Bryan Catanzaro, Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019.
Pipedream: Fast and efficient pipeline parallel DNN training. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, abs/1806.03377CoRRAaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, and Phillip B. Gibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018.
Gpipe: Efficient training of giant neural networks using pipeline parallelism. Yanping Huang, Yonglong Cheng, Dehao Chen, Hyoukjoong Lee, Jiquan Ngiam, Quoc V Le, Zhifeng Chen, abs/1811.06965ArXiv. Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. ArXiv, abs/1811.06965, 2018.
Memory-efficient pipeline-parallel dnn training. Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia, arXiv:2006.09503arXiv preprintDeepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory-efficient pipeline-parallel dnn training. arXiv preprint arXiv:2006.09503, 2020.
ZeRO: Memory Optimizations toward Training Trillion Parameter Models. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20IEEE PressSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Mem- ory Optimizations toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
ZeRO-Offload: Democratizing Billion-Scale Model Training. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He, 13] Microsoft. DeepSpeed: Extreme-scale model training for everyoneJie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offload: De- mocratizing Billion-Scale Model Training, 2021. [13] Microsoft. DeepSpeed: Extreme-scale model training for everyone.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick Legresley, Jared Casper, Bryan Catanzaro, Megatron-LM: Software repository. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Software repository. https://github.com/ NVIDIA/Megatron-LM, 2021.
DeepSpeed: Extreme-scale model training for everyone. Deepspeed Team, Rangan Majumder, DeepSpeed Team and Rangan Majumder. DeepSpeed: Extreme-scale model train- ing for everyone. https://www.microsoft.com/en-us/research/blog/deepspeed- extreme-scale-model-training-for-everyone/, 2020.
Ai and memory wall. Amir Gholami, Zhewei Yao, Sehoon Kim, Michael W Mahoney, Kurt Keutzer, RiseLab Medium Post. Amir Gholami, Zhewei Yao, Sehoon Kim, Michael W. Mahoney, and Kurt Keutzer. Ai and memory wall. RiseLab Medium Post, 2021.
Mesh-tensorflow: Deep learning for supercomputers. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, Hyoukjoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake A Hechtman, abs/1811.02084Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Pen- porn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Mesh-tensorflow: Deep learning for supercomputers. CoRR, abs/1811.02084, 2018.
Supporting very large models using automatic dataflow graph partitioning. Minjie Wang, Chien-Chin Huang, Jinyang Li, Proceedings of the Fourteenth EuroSys Conference 2019, EuroSys '19. the Fourteenth EuroSys Conference 2019, EuroSys '19New York, NY, USAAssociation for Computing MachineryMinjie Wang, Chien-chin Huang, and Jinyang Li. Supporting very large models using automatic dataflow graph partitioning. In Proceedings of the Fourteenth EuroSys Conference 2019, EuroSys '19, New York, NY, USA, 2019. Association for Computing Machinery.
Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. Tal Ben-Nun, Torsten Hoefler, ACM Computing Surveys (CSUR). 524Tal Ben-Nun and Torsten Hoefler. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys (CSUR), 52(4):1-43, 2019.
Autotm: Automatic tensor movement in heterogeneous memory systems using integer linear programming. Mark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, Venkatesh Akella, Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating SystemsMark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, and Venkatesh Akella. Autotm: Automatic tensor movement in heterogeneous memory systems using integer linear programming. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 875-890, 2020.
Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. Chien-Chin Huang, Gu Jin, Jinyang Li, Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating SystemsChien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty- Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 1341-1355, 2020.
Layer-centric memory reuse and data migration for extremescale deep learning on many-core architectures. Hai Jin, Bo Liu, Wenbin Jiang, Yang Ma, Xuanhua Shi, Bingsheng He, Shaofeng Zhao, ACM Transactions on Architecture and Code Optimization (TACO). 153Hai Jin, Bo Liu, Wenbin Jiang, Yang Ma, Xuanhua Shi, Bingsheng He, and Shaofeng Zhao. Layer-centric memory reuse and data migration for extreme- scale deep learning on many-core architectures. ACM Transactions on Architecture and Code Optimization (TACO), 15(3):1-26, 2018.
Capuchin: Tensor-based gpu memory management for deep learning. Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, Xuehai Qian, Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating SystemsXuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 891-905, 2020.
Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, Dong Li, IEEE International Symposium on High Performance Computer Architecture. Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li. Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. In IEEE International Symposium on High Performance Computer Architecture, 2021.
vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, Stephen W Keckler, 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEEMinsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1-13. IEEE, 2016.
Superneurons: Dynamic gpu memory management for training deep neural networks. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Leon Shuaiwen, Zenglin Song, Tim Xu, Kraska, Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming. the 23rd ACM SIGPLAN symposium on principles and practice of parallel programmingLinnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming, pages 41-53, 2018.
Distributed hierarchical gpu parameter server for massive scale deep learning ads systems. Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, Ping Li, Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. Distributed hierarchical gpu parameter server for massive scale deep learning ads systems, 2020.
Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. Animesh Jain, Amar Phanishayee, International Symposium on Computer Architecture (ISCA 2018). Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhi- menko. Gist: Efficient data encoding for deep neural network training. In International Symposium on Computer Architecture (ISCA 2018), 2018.
Training deep nets with sublinear memory cost. Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin, abs/1604.06174CoRRTianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, abs/1604.06174, 2016.
Checkmate: Breaking the memory wall with optimal tensor rematerialization. Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, Joseph E Gonzalez, abs/1910.02653ArXiv. Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization. ArXiv, abs/1910.02653, 2019.
Superneurons: Dynamic GPU memory management for training deep neural networks. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Leon Shuaiwen, Zenglin Song, Tim Xu, Kraska, abs/1801.04380CoRRLinnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic GPU memory management for training deep neural networks. CoRR, abs/1801.04380, 2018.
Adaptive subgradient methods for online learning and stochastic optimization. John Duchi, Elad Hazan, Yoram Singer, J. Mach. Learn. Res. 12John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for on- line learning and stochastic optimization. J. Mach. Learn. Res., 12(null):2121-2159, July 2011.
Adam: A method for stochastic optimization. P Diederik, Jimmy Kingma, Ba, 3rd International Conference on Learning Representations. San Diego, CA, USAConference Track ProceedingsDiederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learn- ing Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Scaling SGD batch size to 32k for imagenet training. Yang You, Igor Gitman, Boris Ginsburg, abs/1708.03888CoRRYang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. CoRR, abs/1708.03888, 2017.
. Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, Cho-Jui Hsieh, Reducing BERT pre-training time from 3 days to 76 minutes. CoRR, abs/1904.00962Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing BERT pre-training time from 3 days to 76 minutes. CoRR, abs/1904.00962, 2019.
. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu, Mixed precision trainingPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2017.
. Nvidia Tensor Cores, NVIDIA Tensor Cores. https://www.nvidia.com/en-us/data-center/tensor-cores/, 2018. [Online, accessed 5-April-2021].
. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin, Attention is all you need. CoRR, abs/1706.03762Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
turing-nlg: A 17-billion-parameter language model by microsoft. turing-nlg: A 17-billion-parameter language model by microsoft.
SuperPOD delivers world record supercomputing to any enterprise. Nvidia Dgx, NVIDIA. NVIDIA DGX SuperPOD delivers world record supercomputing to any enterprise. https://developer.nvidia.com/blog/dgx-superpod-world-record- supercomputing-enterprise/, 2019.
Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Alec Radford, Karthik Narasimhan, Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, arXiv:2006.15704Experiences on accelerating data parallel training. arXiv preprintShen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020.
Ultra-performance pascal gpu and nvlink interconnect. Denis Foley, John Danskin, IEEE Micro. 372Denis Foley and John Danskin. Ultra-performance pascal gpu and nvlink inter- connect. IEEE Micro, 37(2):7-17, 2017.
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning
https://arxiv.org/abs/2104.07857
https://arxiv.org/abs/2103.06332
Hurdles to Progress in Long-form Question Answering
2021-03-10
Krishna; Kalpesh; Roy; Aurko; Iyyer; Mohit
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2103.06332">Download PDF</a></p><blockquote>
 Abstract: The task of long-form question answering (LFQA) involves retrieving documents
relevant to a given question and using them to generate a paragraph-length
answer. While many models have recently been proposed for LFQA, we show in this
paper that the task formulation raises fundamental challenges regarding
evaluation and dataset creation that currently preclude meaningful modeling
progress. To demonstrate these challenges, we first design a new system that
relies on sparse attention and contrastive retriever learning to achieve
state-of-the-art performance on the ELI5 LFQA dataset. While our system tops
the public leaderboard, a detailed analysis reveals several troubling trends:
(1) our system's generated answers are not actually grounded in the documents
that it retrieves; (2) ELI5 contains significant train / validation overlap, as
at least 81% of ELI5 validation questions occur in paraphrased form in the
training set; (3) ROUGE-L is not an informative metric of generated answer
quality and can be easily gamed; and (4) human evaluations used for other text
generation tasks are unreliable for LFQA. We offer suggestions to mitigate each
of these issues, which we hope will lead to more rigorous LFQA research and
meaningful progress in the future.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Kalpesh Krishna [<a href="/show-email/756466ef/2103.06332">view email</a>]
 <br />
 <strong><a href="/abs/2103.06332v1">[v1]</a></strong>
 Wed, 10 Mar 2021 20:32:30 UTC (288 KB)<br /><strong>[v2]</strong>
Wed, 19 May 2021 12:01:54 UTC (288 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2103.06332

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:2103.06332 (cs)
 [Submitted on 10 Mar 2021 (v1), last revised 19 May 2021 (this version, v2)]

 Title: Hurdles to Progress in Long-form Question Answering

 Authors: Kalpesh Krishna, Aurko Roy, Mohit Iyyer
 Download PDF
 Abstract: The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.
 Comments: NAACL 2021 camera ready (18 pages) 
 Subjects: Computation and Language (cs.CL) ; Machine Learning (cs.LG)
 Cite as: arXiv:2103.06332 [cs.CL] 
 (or arXiv:2103.06332v2 [cs.CL] for this version) 
 https://doi.org/10.48550/arXiv.2103.06332 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Kalpesh Krishna [view email]
 [v1] Wed, 10 Mar 2021 20:32:30 UTC (288 KB)
 [v2] Wed, 19 May 2021 12:01:54 UTC (288 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 2103
 Change to browse by:
 cs
 cs.LG

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 1 blog link

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Kalpesh Krishna
 Aurko Roy
 Mohit Iyyer
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Hurdles to Progress in Long-form Question Answering
Date Published: 2021-05-19
Authors:
Kalpesh  Krishna, University of Massachusetts Amherst, ♦ Google Research, kalpesh@cs.umass.edu
Aurko  Roy, University of Massachusetts Amherst, ♦ Google Research, aurkor@google.com
Abstract
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends:
(1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future. 1 
1 Introduction
Long-form question answering (LFQA) integrates the retrieval component of open-domain QA, which involves searching a large external knowledge source for documents relevant to a given question, with a text generation component to produce paragraph-length answers. Significant progress has been made on open-domain QA datasets such as Natural Questions (Kwiatkowski et al., 2019) , whose questions are answerable with short phrases and entities, by leveraging dense retrieval techniques like ORQA (Lee et al., 2019) , REALM (Guu et al., 2020) , and DPR (Karpukhin et al., 2020; Lewis et al., 2020c; Izacard and Grave, 2020) . Methods inspired by these results have recently been combined with pretrained language models (Lewis et al., 2020b; Petroni et al., 2020) and applied to the Reddit-derived "Explain Like I'm Five" (ELI5) dataset (Fan et al., 2019) , which is the only publicly-available large-scale LFQA dataset.
The recently proposed KILT benchmark (Petroni et al., 2020) , which compares retrieval-augmented models across a variety of knowledge-intensive tasks including ELI5, automatically evaluates LFQA models by the quality of both generated answers (ROUGE-L against reference answers) and retrieved documents (R-precision against humanannotated relevant documents). In this paper, we build a state-of-the-art system 2 for ELI5 by using a sparse Transformer variant (Roy et al., 2020) to condition over Wikipedia paragraphs returned by a REALM-style retriever (Guu et al., 2020) .
However, despite its success on the KILT leaderboard, our system does not actually use the documents that it retrieves! To measure the effect of retrieval on generation quality, we design a control experiment in which retrieved documents are replaced with randomly-sampled documents at inference time. Results from both human A/B tests and automatic metrics like ROUGE-L demonstrate that conditioning on random documents has almost no effect on generated answer quality (Figure 1c ). We recommend that future LFQA research report the results of such control experiments in addition to reporting generation and retrieval quality. How can a system using random retrieval per- form well on ELI5? Our analysis reveals that this result is partially due to significant train / validation overlap in the ELI5 dataset (Figure 1a ), which eliminates the need for external retrieval. A human study shows that at least 81% of validation questions have a paraphrase in the training set, and almost all validation questions are topically similar to a training set question. While Fan et al. (2019) attempted to identify and remove question overlap using TF-IDF similarity, more complex semantic matching methods & human verification is needed to address this issue in future LFQA datasets.
Digging deeper, we identify fundamental issues with using ROUGE-L to evaluate generated answer quality (Figure 1b ). Simple baselines such as just repeatedly copying the question, or choosing a random training set answer, can outperform LFQA systems such as RAG (Lewis et al., 2020c) in terms of ROUGE-L. On the other hand, our system achieves higher ROUGE-L than reference human-written answers, which is misleading since human A/B testers strongly prefer reference answers to our system's. We conclude that ROUGE-L is not a reliable metric to evaluate LFQA due to its large and relatively unconstrained output space (e.g., compared to translation or summarization), and we offer suggestions for better automatic & human evaluations to enable meaningful progress on this task.
2 A state-of-the-art LFQA system
The ELI5 task (Fan et al., 2019) asks models to generate paragraph-length answers to open-ended questions in English that often rely on world knowledge (e.g., how do jellyfish function without brains or nervous systems?). LFQA systems thus benefit from conditioning answer generation on relevant documents from the web (such as the Wikipedia article about jellyfish). While large-scale pretrained language models store surprising amounts of world knowledge within their parameters (Petroni et al., 2019; Roberts et al., 2020) , external document retrieval not only augments this intrinsic knowledge but also grounds model outputs in a knowledge source, which provides interpretability.
In this section, we describe our proposed LFQA system, which conditions answer generation on Wikipedia articles identified by a pretrained retriever. We use a dense retriever trained by scaling up a distantly supervised algorithm from Jernite (2020). Since retrieved articles can be quite long and often exceed the maximum sequence length of pretrained models like BERT (Devlin et al., 2019) , we use a sparse-attention variant of the Transformer to allow modeling over longer sequences. While our system sets a new state-of-the-art on ELI5, we question the significance of this result in Section 3.
2.1 Retriever
We begin by specifying our dense retriever ("contrastive REALM" or C-REALM), which returns documents related to an input question. Consider a corpus of long-form questions and answers, represented by (q i , a i ) N i=1 . Our retriever uses q i as a query to retrieve K documents (r i,j ) K j=1 from a knowledge corpus (Wikipedia), which is enabled by an encoder network that projects both questions and candidate documents to a 128-d shared embedding space. Like REALM (Guu et al., 2020) , our encoder is a BERT-base Transformer (Devlin et al., 2019) with a final projection layer.
Since the ELI5 dataset does not include gold retrievals, we train our retriever by scaling up a method recently introduced by Jernite (2020) that uses gold answers for distant supervision. The key idea is to push the encoded vector for a question close to a vector representation of its groundtruth answer(s), but away from all other answer vectors in the mini-batch (negative examples). Intuitively, this method works because both ELI5 answers and external documents are of paragraph length (documents are paragraph-length chunks from Wikipedia). Concretely, we optimize the loss, loss = -
(q i ,a i )∈B log exp q i • a i a j ∈B exp q i • a j
where B is the mini-batch and q i , a i are the encoded vector representations for (q i , a i ). This objective is based on contrastive learning, a method that has been used effectively for semi-supervised learning (Chen et al., 2020) and dense retriever training (Karpukhin et al., 2020) . Scaling up from Jernite (2020) , who used a mini-batch size of 512 and initialized their retriever with BERT, we use much large mini-batches of size 12,288 (and hence, many more negative examples) and initialize our retriever with a strong pretrained retriever, the REALM model (Guu et al., 2020) trained on the Common Crawl News (CC-News) corpus. These design decisions greatly improve retriever quality, as we observe in an ablation study (see Appendix A.2). During inference, we perform a maximum inner-product search (MIPS) with the ScaNN library (Guo et al., 2020) to efficiently find the top K documents. In all our experiments we use K = 7, following the setup in Guu et al. (2020) .
2.2 Generator
We next describe our generator model, which conditions its generated answers on retrieved documents returned by C-REALM. We use the Routing Transformer (RT) from Roy et al. (2020) , which is the current state-of-the-art in long-form language modeling. The RT is a sparse attention model that employs local attention as well as mini-batch k-means clustering to better model long-range dependencies in sequences (attention maps in Appendix A.1). Long-form language models such as RT are wellsuited to ELI5 as the task requires conditioning answer generation not only on a short question but also many lengthy retrieved documents.
We pretrain our RT model on PG-19, a longform language modeling benchmark (Rae et al., 2020) created from approximately 28,000 Project Gutenberg books published before 1919. PG-19 has 1.9B tokens and an average context size of 69K words. While this data is out-of-domain for ELI5, we choose it to encourage long & coherent generation. Our RT is a 22-layer model with 1032 hidden units (486M parameters), maximum sequence length of 8192 tokens, and a vocabulary of 98K subwords. 3 We fine-tune our model in a decoderonly fashion (Liu et al., 2018; Wolf et al., 2018) by concatenating the top K retrieved documents to the question [r i,K , r i,K-1 ... r i,1 , q i , a i ] and training the model to predict tokens of the answer a i . We do not backpropagate gradients through the retriever. 4 Retrievals slightly improve perplexity (18.1 vs 17.8) as seen in Wang and McAllester (2020) , but do not improve generations ( §3.1).
2.3 Main Experiments
Dataset & Evaluation details: We evaluate our model on the KILT validation & test subsets of ELI5 (Petroni et al., 2020) , since the original ELI5 dataset does not have human annotations to measure retriever performance. We downloaded the ELI5 dataset (Fan et al., 2019) Table 1 : Results on the KILT test set for ELI5 for (1) retrieval performance, using R-precision and Recall@5 (RPrec, R@5), and (2) generation quality, using ROUGE-L (R-L). These scores are combined to produce the final metric KILT R-L (KRL). We outperform prior work on both generation & combined scores.
are hidden, and hosted on a public leaderboard in the EvalAI platform (Yadav et al., 2019) . Answer quality is measured by the maximum overlap of generations with a set of gold answers in terms of unigram F1 score and ROUGE-L (Lin, 2004) . Petroni et al. (2020) collected human annotations of Wikipedia articles which support ELI5 gold answers, which enables measuring retrieval quality by computing R-precision (if the top-1 retrieval matches the annotation) and Recall@5 using the top-5 retrievals. Finally, the KILT benchmark combines R-prec. and ROUGE-L to measure the overall performance of the system by "KILT ROUGE-L". This metric is similar to ROUGE-L, but assigns a score of 0 whenever the top-1 retrieval does not match the gold annotation.
Baselines: We compare our model with the other entries on the ELI5 KILT leaderboard which are either generation-only, like T5-base (Raffel et al., 2020) and BART (Lewis et al., 2020b) , or variants of BART using retrieval such as RAG (Lewis et al., 2020c) and BART + DPR (Petroni et al., 2020) . These systems are based on massive pretrained language models, with similar number of parameters as our model (details in Appendix A.3).
Results: Table 1 contains our results on the test set of the ELI5 (also on the public KILT leaderboard). We present four variants of our system, using a different retriever during inference (REALM or C-REALM), and different nucleus sampling p values (Holtzman et al., 2020) . All variants outper-Q: Why are almost all boats white? A: Boats are generally painted white for obvious reasons, as they are inexpensive and easy to clean. Ships are also often painted to provide a visible visual contrast to the sea. Boats also often have high visibility ports to allow for easy international trade. In addition, many boats have an airbreathing function that makes them easier to spot and stay afloat, which is a good thing if you're a pirate or trying to get into some country's waters. Lastly, many ships are built in the last few decades, so they are not as exotic looking as they were thirty years ago. And lastly, many ships are made out of materials that are easy to work with, and so many of them are painted white as a practical matter. There are many, many more reasons for why ships are white, but those are the ones I know of. I'm sure there are many other reasons, but those are the ones I know of. Edit: I forgot to mention that the other answers about visibility and easy to see ports are also true. Ships are painted to appear to be larger, and more expensive. That makes them appear more impressive, and so they are more likely to be bought by people who want to show off their wealth and status. form prior work in generation quality, with lowerentropy models (p = 0.6) performing best.6 C-REALM performs competitively to RAG and DPR despite being only distantly supervised, and outperforms REALM. Our proposed RT+C-REALM system achieves a new state-of-the-art on combined performance (KILT R-L). Generations from our model are provided in Figure 2 and Appendix A.4.
3 Analysis
In this section, we conduct a thorough analysis of our model's usage of retrievals (Section 3.1), the impact of overlap in ELI5's train / validation / test folds (Section 3.2), issues with ROUGE-L and performance bounds (Section 3.3), and the difficulty in human evaluation for this task (Section 3.4). At the end of each section, we provide short takeaways with suggestions for future work.
3.1 Are generations grounded in retrieval?
While our retrieval-augmented system achieves state-of-the-art performance, we find little evidence that it is actually using the retrieved documents. To measure this, we run an ablation study where at inference time we replace retrieved paragraphs with For p = 0.9 pred. random 31% ( 52) 37% ( 63) 32% (54) pred. gold ans. 17% (49) 72% (203) 11% ( 31) Table 3 : Human evaluation results with exact number of ratings shown in (•). Annotators are shown a question along with two answers (A, B) in random order and ask them to choose one (details in Appendix A.5). For both model variants (p = 0.6, 0.9), we see (1) little difference between generations conditioned on predicted (pred.) or random (rand.) retrievals; (2) strong preference for gold answers over generations.
randomly sampled paragraphs from Wikipedia. We compare this Random baseline with our original system (Predicted) in terms of generation quality as well as the n-gram overlap between the generation and the retrieved paragraphs.
Generations are similar irrespective of type of retrievals: We present our results in Table 2 . Despite not being conditioned on any meaningful retrievals, the Random retrieval model has similar ROUGE-L scores as our Predicted system. Moreover, generations from the Random and Predicted models have similar amounts of 1-gram and 2gram overlap with the paragraphs retrieved by C-REALM, despite the fact that the Random model does not actually see the retrieved paragraphs. 7The n-gram overlaps are possibly overestimates due to stopwords (e.g., prepositions, punctuation) and entities which are copied from the question. To tackle this issue, in Table 4 we measure the fractions of lemmatized nouns, proper nouns and numbers in the generated answer which are present in the predicted retrievals but not in the question. We notice similar trends as before, with only small differences between the two systems. Finally, there is almost no correlation (Spearman ρ = 0.09) between the Predicted model's generation quality and the amount of unigram overlap between its outputs and the retrieved documents (scatter plots in Appendix A.7), strengthening our hypothesis that generations are not grounded in retrievals.8 
Human evaluation validates our findings: As ROUGE-L and n-gram overlap have major limitations for LFQA (Section 3.3), we perform additional human A/B testing on the output of Random and Predicted. Specifically, we ask human volunteers9 to choose between answers generated by the two systems (presented in random order).
As seen in Table 3 , humans struggle to choose which of the two answers is more relevant to the question. For both model variants (p = 0.6, 0.9), there is a less than 7% preference for a particular answer type, with humans preferring answers (by 6%) from the Random model for p = 0.9! Other systems also have this issue, possibly due to source-reference divergence and trainvalidation overlap: We note that this issue is not unique to our system -other systems on the KILT leaderboard like BART + DPR and RAG actually perform worse than their no-retrieval counterpart (BART) in generation quality, as shown in Table 1 . Qualitatively, we found no evidence of retrieval usage in a publicly hosted ELI5 model demo by Jernite (2020). 10 A possible explanation for this issue is high source-reference divergence, a common problem in table-to-text generation (Wiseman et al., 2017; Tian et al., 2019) .
In Table 2 and Table 4 , we measure the n-gram overlap of top-ranked gold validation answers (Gold Ans) with predicted retrievals. This overlap is low and similar to that of our generations, which we suspect encourages our model to ignore retrievals. A second explanation is the large amount of train-validation overlap (Section 3.2), which eliminates the need for retrieval. 2020) are significantly shorter (59 words avg.) than our generations (187 words avg.).
12 https://eval.ai/web/challenges/ challenge-page/689/leaderboard/1909 bined retrieval + generation performance via KILT RL, it does not check whether the generations actually used the retrievals. In other words, one can submit independent retrieval & generation systems, but still perform well on the combined score. This may not be an issue for short-form QA tasks like Natural Questions, since the gold answer is often exactly contained as a span in the gold retrieval. Also, as retrieval might be less important for large language models with parametric knowledge (Roberts et al., 2020) , the KILT-RL strategy of simply aggregating top-1 retrieval score with ROUGE-L unfairly penalizes systems not relying on retrieval.13 
3.2 Training / Validation Overlap
Our experiments in Section 3.1 show that model performance is mostly unchanged by conditioning generation on randomly sampled retrievals instead of predictions from C-REALM. Despite not using retrievals, we observe qualitatively that our model displays a large amount of parametric knowledge ("Faraday Cage" in Figure 1c ), which is surprising since it was pretrained on novels from Project Gutenberg (not Wikipedia). In this section, we discover that a major reason for ignoring retrievals is the large amount of train / validation overlap in ELI5. While Fan et al. (2019) attempted to fix this issue through TF-IDF overlap, this method is insufficient to identify all question paraphrases, as we find significant overlap between the training set and the KILT validation set of ELI5.14 ELI5 is not the only dataset with substantial train / test overlap: Lewis et al. (2020d) identify similar issues with short-form QA datasets like Natural Questions.
Finding similar questions & measuring overlap:
We use our retriever C-REALM to retrieve similar questions from the training set, since it has learned to map questions to a feature-rich embedding space. For each validation question, we retrieve the 7 most similar training set questions. We use both human and automatic evaluation to calculate the amount of overlap. For human evaluation, we show annotators on Amazon Mechanical Turk 15 a validation set question and a retrieved training set question, qns with at least one train set paraphrase 81% qns with at least one train set topically similar 100% % of all pairs marked paraphrases 39.5% % of all pairs marked topically similar 47.8% % of all pairs marked as non-paraphrases 12.7% and ask them to annotate the pair as 0: No paraphrase relationship; 1: on similar topics, but different questions; 2: approximately the same question (an adaptation of the paraphrase evaluation of Kok and Brockett, 2010) . We take 300 validation set questions and ask three crowd-workers to rate them against retrieved training questions on this scale, and consider the label with majority rating. To improve quality, we manually verify their annotations.
Table 5 shows that 81% of validation set questions have at least one paraphrase in the training set, while all annotated questions have at least one topically similar question in the training set, which indicates substantial training / validation overlap. The experiment had "fair agreement" with a Fleiss κ of 0.29 (Fleiss, 1971; Landis and Koch, 1977) .
As manually annotating question overlap can be expensive and time-consuming, we also experiment with automatic overlap detection methods. In particular, we use a RoBERTa-large binary classifier (Liu et al., 2019) fine-tuned on the Quora Question Paraphrase (QQP) dataset (Iyer et al., 2017) from the GLUE benchmark (Wang et al., 2019) . For 43.6% of the ELI5 validation set, this classifier marked at least one retrieved question as a paraphrase (46% for the 300 questions we annotated). Qualitatively, we notice that this classifier often mis-classifies retrieved questions that are valid paraphrases but exhibit significant lexical or syntactic divergence. This observation, along with the smaller fraction of valid paraphrases in the QQP training set (37%), partially explains the gap between automatic & human evaluations.
Using retrieved QA for generation: Since ELI5 contains significant amount of overlap between the training and validation sets, a system can simply copy the answers of retrieved training set questions instead of actually doing generation. Table 7 shows that by using the longest answer within the top-K retrieved questions, we outperform two prior systems (RAG, BART + DPR) that use retrieval-augmented generation. As an upper bound, we also consider a system which uses the best possible answer to retrieved training set questions in terms of ROUGE-L (best top-K train answer). This system gets 28.5 ROUGE-L, outperforming all others.
ELI5 performance on overlapping QA: Finally, we measure the performance difference between validation questions that overlap with the training set vs. those that do not. Since we only have human annotations for 300 questions (the nooverlap subset has only 53 samples), we present this analysis using the QQP classifier's outputs as well. In Table 6 , we notice large differences of 6.6 RPrec, 8.1 R@5 in retrieval performance favoring the overlap subset, but only a small generation score gain of 0.8 F1, 0.4 R-L (which may be misleading as discussed in Section 3.3).
Takeaway (careful held-out curation): Based on our findings, we suggest that more careful dataset curation for LFQA tasks is needed to prevent duplicates. While we acknowledge the efforts of Fan et al. (2019) to fix this issue, we also suggest alternative methods to control overlap and focus on evaluating generalization in held-out sets: (1) automatically retrieving paraphrases and then running human validation to eliminate them; or (2) holding out entire genres or domains to reduce the possibility of overlap -for example, keeping Q/A on Sports only in the held-out sets. Note that simply pruning the existing splits using these criteria will significantly reduce the size of the held-out datasets; so we suggest re-splitting the train/validation/test splits from the entire pool of collected questions.
3.3 ROUGE-L Bounds on ELI5 Performance
We have seen that simply copying the answer of a close question paraphrase from the training set achieves 28.5 ROUGE-L with an optimal selection among retrieved questions and outperforming all computational models. But how "good" is this absolute number? What are some suitable upper & lower bounds to ROUGE-L scores on ELI5? Is ROUGE-L an informative metric for LFQA?
Lower bounds are trivial baselines used to test the vulnerability of datasets or metrics to simple heuristic strategies that do not actually perform the task.
Recent examples include hypothesis-only baselines for natural language inference (Gururangan et al., 2018) and passage-only baselines for reading comprehension (Kaushik and Lipton, 2018) . We evaluate two ROUGE-L lower bounds on ELI5:
(1) copy the question 5 times and concatenate, as longer outputs boost ROUGE-L (Appendix A.6);
(2) retrieve a random training set answer.
Our first baseline contains entities often present in the gold answer, but without actually answering the question. Our second baseline follows the "style" of an answer but is completely off-topic.
As an upper bound, we estimate the ROUGE-L of gold answers themselves. On an average, there are 12 gold answers per question, so we measure the ROUGE-L of the longest gold answer with respect to the other gold answers. We also measure the maximum pairwise ROUGE-L between two gold answers for the same question. 16 We only calculate upper bounds for the validation set, since the gold answers of the KILT test set are hidden.
Lower bounds beat prior work, upper bounds have low ROUGE-L:
We compare our bounds with actual retrieval augmented generation systems in Table 7 . Both our lower bounds (random training answer, copy input) are quite competitive, outperforming RAG (Lewis et al., 2020c) and performing close to BART + DPR (Petroni et al., 2020) without actually answering the question! This shows that ROUGE-L is fairly sensitive to simply copying entities from the question 16 Note that different gold answers were not written independently as Reddit users writing answers can read existing answers and may want to provide a non-overlapping perspective. Due to the high train/valid overlap, the best top-7 retrieved answer could be a better upper bound since it is from another Reddit post (and performs better than best gold answer). as well as stylistic properties of ELI5. On the other hand, upper bounds (longest gold answer) perform worse than our system (21.2 vs 24.4).
Suspecting that this result is misleading, we run another human A/B test by showing volunteers a question and asking them to choose between answers generated by our system and the longest gold answer, shuffled at random. 17 As seen in Table 3 , the majority of humans prefer the gold reference answers vs generations (68% vs 14% for p = 0.6). In interviews with human annotators after completing the task, they reported that both answers were often fluent and stylistically similar, but one eventually veered off-topic.
Takeaway (better automatic metrics needed):
Our experiments demonstrate that computing the ROUGE-L of generations against gold answers is not a meaningful way to evaluate LFQA systems, since it is not selective enough to differentiate between valid/invalid answers. There is a very small margin of improvement between trivial lower bounds and strong upper bounds, with the absolute scores of upper bounds being quite low. We suspect this is due to the long length of answers and fairly unconstrained and large output space.
The ELI5 dataset has several open-ended questions with many plausible answers (like What causes traffic?), often involving analogies. A possible fix is a sentence-level evaluation and then aggregating scores across generated sentences, but appropriate penalties are needed for lack of diversity (Zhu et al., 2018 ) and short lengths. Other possible fixes include learning task-specific metrics to measure semantic overlap (Sellam et al., 2020) or metrics to check factual correctness (Zhang et al., 2020) and faithfulness to input (Wang et al., 2020; Durmus et al., 2020; Zhou et al., 2020) . Ultimately, all automatic metrics have their limitations, and human evaluation is necessary (Celikyilmaz et al., 2020) .
3.4 Difficulty of Human Evaluation
To better understand the inherent difficulty of evaluation in ELI5, we interviewed human annotators (of Table 3 ) and found two challenges:
(1) Unfamiliarity with question topics: While most annotators found the Q/A interesting, they were often unfamiliar with the technical topics discussed in the questions. This made it hard for them to assess answer correctness. The ELI5 dataset has questions in a wide variety of topics (History, Politics, Biology etc.), while most annotators were Computer Science graduate students. While we did allow annotators to use Wikipedia, they mentioned domain-experts will be better judges of factual correctness of answers.
(2) Length of Answers: Annotators mentioned the paragraph-long length of answers made the task quite challenging. Annotators reported taking an average of 2 minutes per answer pair, many of which required careful thought & concentration. This was especially difficult when only part of the answer was correct and the rest had contradictions or repetitions, a common theme in our generations.
Takeaway: Human evaluation is challenging but necessary for evaluating LFQA. Crowd-workers are unlikely to spend time reading & analyzing long text (Akoury et al., 2020) . Hence, it is imperative to design simpler evaluations. One effort in this direction is Dugan et al. (2020) , who reveal one generated sentence at a time and estimate system quality based on the number of sentences which fooled humans. Another promising direction is extrinsic evaluation (Celikyilmaz et al., 2020) where humans actually interact with systems in real-world scenarios such as the Alexa Prize (Ram et al., 2018) or STORIUM (Akoury et al., 2020) .
4 Conclusion
We present a "retrieval augmented" generation system that achieves state-of-the-art performance on the ELI5 long-form question answering dataset. However, an in-depth analysis reveals several issues not only with our model, but also with the ELI5 dataset & evaluation metrics. We hope that the community works towards solving these issues so that we can climb the right hills and make meaningful progress on this important task.
"TPUs are highly efficient chips which have been specifically designed for machine learning applications". These accelerators run on Google Cloud, which has "matched 100% of its electricity consumption with renewable energy purchases, and has committed to fully decarbonize its electricity supply by 2030" (https://cloud.google. com/sustainability). More details on training time are provided in Appendix A.1.
A Appendices for "Hurdles to Progress in Long-form Question Answering"
A.1 Training & Model Details
All our models are developed and trained using TensorFlow 1.15 (Abadi et al., 2016) and Tensor2Tensor (Vaswani et al., 2018) . Our implementations are based on the open-source codebases of REALM 19 and the Routing Transformer. 20Similar to the REALM implementation, we use separate processes to run the retriever and generate training data (using a MIPS search). Since our retriever is frozen, we do not use the document index refresher available in their codebase.
Retriever: Our retriever is trained on 64 Google Cloud TPUs for a total of 4k steps and a batch size of 12288. We do early stopping on the validation data (with a smaller batch size of 512 due to smaller P100 GPU memory).
Our model converges quite fast, reaching its best performance in 1.5k steps (in 43 minutes) and needing 103 minutes for the full set of 4k steps.
Generator: Our generator is trained on 64 Google Cloud TPUs, for a total of 100k steps on the ELI5 training set. We use the pg19_local_cluster8k configuration available in the Routing Transformer implementation.
Besides the default hyperparameters, setting 15% input, attention and ReLU dropout was critical to prevent overfitting on the training set. We use a learning rate of 5e-5. Our retrievals, questions and answers are truncated / padded to 288 subword tokens (using the PG19 subword tokenizer). We use a minibatch size of 128 QA pairs, which corresponds to 332k tokens per mini-batch (of which, the loss is computed over the last 288 answer tokens, or 37k total tokens). We do not compute loss over padded tokens, and use special symbols to separate different parts of the input context. We reverse the retrieved paragraphs in context since the model uses local attention layers, and we wanted higher ranked retrievals to appear closer to the answer tokens. Our models take about 30 hours to finish 100k steps (0.92 steps / second). Hyperparameter Choices: We experimented with several different pretraining strategies (using Wikipedia), smaller model variants and hyperparameter choices manually in preliminary experiments. All these experiments performed quite poorly on ELI5, producing very short and sometimes incoherent responses. Finally, switching to a Routing Transformer model which was pretrained on a longform language modeling dataset (PG-19) significantly improved generation quality. Hyperparameters for this pretrained model (like hidden size / number of layers) were manually chosen with model capacity in mind. For our final experiments with this pretrained model we did not perform any hyperparameter search during training, primarily due to the expensive setup required to train the system. During inference, we tuned the nucleus sampling value from 0.0 to 1.0 in increments of 0.1, choosing the value with the best validation set performance. Our hyperparameter choices for contrastive learning on the retriever have been justified in an ablation study in Appendix A.2. Notably, we use very large minibatches of 12,288 to scale the number of negative examples. To train this model, we used the standard trick of data parallelism across 64 hardware accelerators. This resulted in an effective mini-batch size of 192 per chip, which is small enough to fit a BERT-base sized model on a TPU v3 chip's memory. To accumulate information across different chips before the final softmax, we used the tf.tpu.cross_replica_sum function (using an open-source wrapper found here).
A.2 Ablation Study of C-REALM
One of our contributions is scaling up a distantly supervised objective for training retrievers on ELI5, originally described in Jernite (2020) . This method uses in-batch negative sampling, making minibatch size a critical hyperparameter for better constrastive learning. We perform controlled experiments initializing our retrievers with REALM-CCNews (Guu et al., 2020) and varying batch size and keeping all other hyperparameters consistent.
In Table 8 , we notice a steady increase in performance as minibatch size is increased, with the largest gains coming by doubling the batch size in Jernite (2020) from 512 to 1024. Finally, in preliminary experiments we saw no benefit of more intelligent negative sampling schemes.
Batch size R-Prec Recall@5 REALM (pretrained) 6.6 14.9 256 6.2 11.0 512 (Jernite, 2020) 6.8 12.6 1024 11.5 21.0 12288 (Ours) 13.3 21.2
Table 8 : The effect of minibatch size on the validation performance of C-REALM. As a baseline, we also add the retrieval performance of the REALM pretrained model which is used as an initialization.
Next, we investigate the effect of initialization on the training of C-REALM. Unlike Jernite (2020) who initialize their model with BERT, before training we initialize our retriever with a pretrained self-supervised retriever. As a baseline, we initialize our model with ICT, a weaker self-supervised retriever introduced in Lee et al. (2019) . Both models are trained with minibatch sizes of 12228. In Table 9 , we notice a large improvement in performance when using a better initialization, confirming our design decisions.
A.3 Number of trainable parameters
In Table 10 we present the number of trainable parameters in our model compared to baselines on the leaderboard. Our generator is slightly larger than the models used in prior work, but we utilize a smaller retriever due to the shared query and candidate encoders in REALM. Overall, our system has a similar total number of parameters as baseline models like RAG and BART + DPR. Initialization R-Prec. R@5 REALM (pretrained) 6.6 14.9
ICT (Lee et al., 2019) 9.3 16.5 REALM (Guu et al., 2020) 13.3 21.2
Table 9 : The effect of initialization on C-REALM. As a baseline, we also add the retrieval performance of the REALM-CCNews pretrained model without any finetuning on ELI5.
Model Generator Retriever Index
T5-base 220M - - BART 406M - - RAG 406M 220M 15B BART + DPR 406M 220M 15B RT + C-REALM 486M 110M 15B
Table 10 : The number of parameters used by our model and baselines. Our generator is slightly bigger than other submissions on the leaderboard, but we use a smaller retriever with a similar sized index.
A.4 Generations from our System
More generations have been provided (along with retrievals, highlighted to show n-gram overlap) in the supplementary material (data) as HTML files.
We also present a few samples in Table 16 .
A.5 Human Evaluation Setup
We conducted several A/B tests between variants of our model using human annotators. We asked a total of 20 participants for help who voluntarily agreed to help with the annotation process. Most participants were English-speaking graduate students in computer science. In every test, participants were shown a question along with two answers (generated by different systems) presented in a random order. They were then asked to choose which generation (1) answered the question better / which answer was more relevant to the question;
(2) was more coherent / had less repetition; (3) was more factually correct. Since some annotators had a limited time, we asked them to prioritize question (1) over (2) / (3). Annotators were allowed to select "Tie" if they could not choose between the systems. We also permitted them to use search engines, but suggested restricting search to Wikipedia. We present all our results in Table 15 . We also interviewed some participants after the annotation process and discuss our findings in Section 3.4. Note that while these A/B tests help us understand which system is relatively better, they do not provide an absolute measure of performance (Celikyilmaz et al., 2020) -annotators reported that there were cases where both answers were very good and other cases where both were very poor. This is a limitation of A/B testing.
A.6 Effect of length on ROUGE-L
In this section we measure the effect of outputs lengths on ROUGE-L scores. To conduct this experiment, we truncate generations by our system to a fixed fraction of tokens across all instances. As we see in no correlation between the two quantities, with Spearman ρ = 0.09.
Instances with correct predicted retrieval: In Table 12 , we present results similar to Section 3.1 considering only those instances where at least one retrieved document matched the gold annotation (roughly 23% instances). We also present a scatter plot on the same set of instances in Figure 5 and note a low correlation of ρ = 0.13. (Predicted) and randomly chosen retrievals (Random), for those cases where C-REALM predicted the correct retrieval. Notice very small differences in generation quality (R-L) as well as the fraction of n-grams (n-g) in the generation overlapping with retrievals predicted by C-REALM (vs predicted retr.). To control for overlap due to stopwords, we also add n-gram overlaps with the randomly sampled retrievals.
Experiments with p = 0.9: Table 13 : Comparison of generations (with p = 0.9) conditioned on retrievals from C-REALM (Predicted) and randomly chosen retrievals (Random). Notice very small differences in: (1) ROUGE-L vs gold answers (R-L); (2) n-gram overlap (n-g) with retrievals predicted by C-REALM (vs predicted retr.). Gold answers also have a similar overlap with predicted retrievals. To control for overlap due to stopwords, we also add n-gram overlaps with the randomly sampled retrievals.
higher nucleus sampling values. As we saw in Section 2.3, these generations tend to be more fluent and coherent, but less relevant to the question. In Table 13 and Table 14 Q: What causes the trail behind jets at high altitude?
A: The same thing that causes the clouds you see at the start of a storm to form in the first place. The air pressure changes as the jet moves through the air, causing it to cool down and condense into water droplets. These droplets are in fact a lot bigger than the water droplets in the clouds, so when the plane passes through them, they can cause the air pressure to change, causing the water droplets to condense into clouds. This is what you see when you look at a cloud formation from the ground. In addition to this, when the plane passes through the clouds, it also moves through the air, causing it to change its temperature, causing the water droplets to condense and form clouds. These clouds are often quite large, and can block out the sun. This is what you see when you look at a contrail. Finally, the water vapor that forms at high altitudes cooled by the air above, and as it cools, it condenses into clouds. This is what you see when you look at a cumulonimbus cloud. These clouds are even larger than the water droplets in the clouds, and they block out the sun.
Q: How do shampoo and conditioner work?
A: The key ingredients in shampoo and conditioner are usually surfactants, they tend to attract oil from your skin. Shampoo will usually contain an anti-dandruff agent (dandruff is caused by fungi in your skin which cause dandruff). They also contain some moisturizing agents (moisturizers are things that stay in your skin longer and protect your scalp from cleaning), and a lot of anti-dandruff (dandruff is a side effect of high doses of both surfactants and moisturizers) Conditioner will contain moisturizing agents, like a skin moisturizer. They tend to cause oils to be separated from your hair (e.g. wash your hair with conditioner instead of shampoo). They both have a foaming agent that makes them foam up (usually Hydrogen Peroxide). They also have a foaming agent that stops dirt from attaching to your hair. Some conditioner will also suppress the action of the shampoo (to help prevent dirt from attaching to your hair). So what's the difference between shampoo and conditioner? Well, if you combine both the shampoo and conditioner Q: When glancing at a clock, why does the first second after glancing at it sometimes feel longer than the rest?
A: This is known as chronostasis. Your brain gets used to seeing the second hand move after the first, so it ignores the first one. If you were to look at the clock a lot, you'd see the second hand move a lot more than the first, and you'd be able to tell the difference between the second and the first. The second hand will appear to move much slower than the first hand because your brain is used to seeing the second hand move a lot more than the first. If you were to go on a long trip, you'd get used to the second hand moving a lot more than the first, and you'd have no problem telling the difference between the second and the first hand. It's kind of like how you don't notice your nose when you're driving, but if you look at it for a while, you start to notice it. Your brain is just getting used to seeing the nose move a lot more than the other, and it can't tell the difference between the two. It's kind of like that with your eyes.
Q: Why do Britain and other English empire countries still bow to monarchs? What real purpose does the queen serve?
A: Depends what you define as "purpose". There is no specific purpose for the British monarch -rather, they are figureheads that serve a role. The role is to have a high degree of influence over a country, though the rules of the monarchy can somewhat vary from country to country, and the ceremony of "kissing the royal corpse" does not always involve kissing the monarch. Whether that's the only reason for the custom, or if it was the only reason, I can't say, but that's the reasoning. When the U.K. was in full power, the monarch was a puppet of Parliament, and the powers of the monarch were transferred to the Prime Minister and thus to the Prime Minister's deputy, who then became the Prime Minister. Since then, Parliament has been able to vote on legislation that goes through the monarch, although they may still act as the monarch's representative in negotiating treaties, which can have very very deep consequences. The Queen's role, as a representative of Britain, doesn't necessarily involve her formal approval of any of the laws or legislation that goes through Parliament, though. All that's left is for the legislature to pass laws Q: What exactly is fire, in detail? How can light and heat come from something we can't really touch?
A: Fire is the chemical reaction of fuel (oxygen) with (what we call) impurities (ash, soot, oil etc). These impurities are created due to a number of things. If you were to study the chemical composition of a log (or a candle) and analyse it you would discover the chemical formula (A, C, and G) which is a simplified version of what actually happens. There is always fuel, and there is always the ash/soot. Wood is made up of mostly carbon and hydrogen and these are the fuels that create the fire. Hydrogen and oxygen are reactive, and they form water (H2O) and CO2 (CO2). If we split a log into smaller pieces, we start to see the process of combustion as it occurs. The carbon is burned, and with the explosion of CO, H2O and H2CO2, we get water and carbon dioxide (mostly). If we smash a log together, we start to see what happens when we also hit another log with another log...imagine smashing two sand castles together. We see how the sand castles and brick castles form and collapse at different rates...then it starts to burn. The smoke, ash and flames are the actual combustion of fuel.
Q: What goes on in those tall tower buildings owned by major banks?
A: The actual buildings are not there for profit but for show. There are many real world historical buildings that have little to no effect other than being a symbol of an economic entity or symbol of an era or location, and nothing more. For example look at Sears, Sears didn't care what went on inside, it was all about the _appearance_ of its location, the prestige of the location, the facilities and so on. It didn't care about how long it took it to operate, it was about how much people would pay to go see it. Sears was a landmark as a cultural movement and other big companies followed suit, so if you want to see a building you've never seen before, you have to go see Sears, just like you have to see a Toyota Camry for Toyota Camry. They used to be all about building new factories, some of them if I recall, but now that they're bigger, that means that more factory jobs are coming to them. You've probably seen them in stores as stores where people buy and sell stuff, so there aren't that many places for them to come from. Instead, it's just for show, a symbol of rich people.
Table 16 : Example generations from our LFQA system with p = 0.9.
Our hyperparameters have been chosen manually with minimal tuning. See Appendix A.1 for details.
We tried training the retriever jointly with RT using the attention bias scheme proposed in MARGE(Lewis et al., 2020a). This improved perplexity only in autoencoding settings where the gold answer itself is used as a retrieval query (like the setup inLewis et al., 2020a), which is not valid in LFQA.
5 github.com/facebookresearch/KILT
As inHoltzman et al. (2020), a human study reveals that higher entropy (p = 0.9) answers are slightly more coherent and sensible, but lower entropy answers (p = 0.6) are more relevant to the question (details in Appendix A.5).
Corresponding experiments with the p = 0.9 variant of our model are presented in Appendix A.7.
All these trends persist even on questions for which our retriever predicts the ground-truth document(Appendix A.7)
Details of our experimental setup in Appendix A.5.
Another issue of KILT-RL is ignoring non top-1 retrievals, penalizing models using multiple retrievals together in context.
The ELI5 demo from Jernite (2020) also retrieves the top-1 similar training set question. Qualitatively, we found many validation examples had near-identical train paraphrases.
15 We pay workers 4 cents per question pair ($8-12 / hr). We only hire workers from USA, UK and Australia with a 95% or higher approval rating and at least 1000 approved HITs.
https://www.gstatic.com/ gumdrop/sustainability/ google-2019-environmental-report.pdf
https://github.com/google-research/ language/tree/master/language/realm
https://github.com/google-research/ google-research/tree/master/routing_ transformer Attention Maps: We show the 2D plots of our generator's attention maps in Figure 3. (a) Local attention (b) Routing attention
References
[1]Abadi Martín, Barham Paul, Chen Jianmin, Chen Zhifeng, Davis Andy, Dean Jeffrey, et al.. Tensorflow: A system for large-scale machine learning. 12th {USENIX} symposium on operating systems design and implementation 2016;:265--283.
[2]Akoury Nader, Wang Shufan, Whiting Josh, Hood Stephen, Peng Nanyun, Iyyer Mohit. Storium: A dataset and evaluation platform for machinein-the-loop story generation. Proceedings of Empirical Methods in Natural Language Processing 2020;:.
[3]Tom B Brown Benjamin, Mann Nick, Ryder Melanie, Subbiah Jared, Kaplan Prafulla, Dhariwal Arvind, et al.. Language models are few-shot learners. Advances in Neural Information Processing Systems 2020;:.
[4]Celikyilmaz Asli, Clark Elizabeth, Gao Jianfeng. Evaluation of text generation: A survey.  2020;:.
[5]Chen Ting, Kornblith Simon, Norouzi Mohammad, Hinton Geoffrey. A simple framework for contrastive learning of visual representations. Proceedings of the International Conference of Machine Learning 2020;:.
[6]Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina. BERT: Pre-training of deep bidirectional transformers for language understanding.  2019;:.
[7]Dinan Emily, Roller Stephen, Shuster Kurt, Fan Angela, Auli Michael, Weston Jason. Wizard of wikipedia: Knowledge-powered conversational agents. International Conference on Learning Representations 2019;:.
[8]Dugan Liam, Ippolito Daphne, Kirubarajan Arun, Callison-Burch Chris. RoFT: A tool for evaluating human detection of machine-generated text. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations 2020;:.
[9]Durmus Esin, He He, Diab Mona. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. Proceedings of the 2020;:.
[10]Fan Angela, Jernite Yacine, Perez Ethan, Grangier David, Weston Jason, Auli Michael. ELI5: Long form question answering. Proceedings of the 2019;:.
[11]Joseph L, Fleiss . Measuring nominal scale agreement among many raters. Psychological bulletin 1971;76:.
[12]Guo Ruiqi, Sun Philip, Lindgren Erik, Geng Quan, Simcha David, Chern Felix, et al.. Accelerating large-scale inference with anisotropic vector quantization. Proceedings of the International Conference of Machine Learning 2020;:.
[13]Gururangan Suchin, Swayamdipta Swabha, Levy Omer, Schwartz Roy, Bowman Samuel, Smith NoahA. Annotation artifacts in natural language inference data. Conference of the North American Chapter 2018;:.
[14]Guu Kelvin, Lee Kenton, Tung Zora, Pasupat Panupong, Chang Ming-Wei. REALM: Retrieval-augmented language model pre-training. Proceedings of the International Conference of Machine Learning 2020;:.
[15]Holtzman Ari, Buys Jan, Du Li, Forbes Maxwell, Choi Yejin. The curious case of neural text degeneration. International Conference on Learning Representations 2020;:.
[16]Iyer Shankar, Dandekar Nikhil, Csernai Kornél. First quora dataset release: Question pairs.  2017;:.
[17]Izacard Gautier, Grave Edouard. Leveraging passage retrieval with generative models for open domain question answering.  2020;:.
[18]Jernite Yacine. Explain anything like i'm five: A model for open domain long form question answering.  2020;:.
[19]Karpukhin Vladimir, Oguz Barlas, Min Sewon, Wu Ledell, Edunov Sergey, Chen Danqi, et al.. Dense passage retrieval for open-domain question answering. Proceedings of Empirical Methods in Natural Language Processing 2020;:.
[20]Kaushik Divyansh, Lipton ZacharyC. How much reading does reading comprehension require? a critical investigation of popular benchmarks. Proceedings of Empirical Methods in Natural Language Processing 2018;:.
[21]Kok Stanley, Brockett Chris. Hitting the right paraphrases in good time. Conference of the North American Chapter 2010;:.
[22]Kwiatkowski Tom, Palomaki Jennimaria, Redfield Olivia, Collins Michael, Parikh Ankur, Alberti Chris, et al.. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 2019;7:453--466.
[23]Landis Richard, Koch GaryG. The measurement of observer agreement for categorical data. biometrics 1977;:159--174.
[24]Lee Kenton, Chang Ming-Wei, Toutanova Kristina. Latent retrieval for weakly supervised open domain question answering. Proceedings of the 2019;:.
[25]Lewis Mike, Ghazvininejad Marjan, Ghosh Gargi, Aghajanyan Armen, Wang Sida, Zettlemoyer Luke. Pre-training via paraphrasing. Advances in Neural Information Processing Systems 2020;:.
[26]Lewis Mike, Liu Yinhan, Goyal ; Abdelrahman Mohamed Naman, Levy Omer, Stoyanov Ves, Zettlemoyer Luke. Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Proceedings of the Association for Computational Linguistics 2020;:.
[27]Lewis Patrick, Perez Ethan, Piktus Aleksandara, Petroni Fabio, Karpukhin Vladimir, Goyal Naman, et al.. Tim Rocktäschel, et al. 2020c. Retrieval-augmented generation for knowledge-intensive nlp tasks. Proceedings of Advances in Neural Information Processing Systems None;:.
[28]Lewis Patrick, Stenetorp Pontus, Riedel Sebastian. Question and answer test-train overlap in open-domain question answering datasets.  2020;:.
[29]Lin Chin-Yew. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out 2004;:74--81.
[30]Peter J, Liu Mohammad, Saleh Etienne, Pot Ben, Goodrich Ryan, Sepassi Lukasz, et al.. Generating wikipedia by summarizing long sequences. Proceedings of the International Conference on Learning Representations 2018;:.
[31]Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, et al.. Roberta: A robustly optimized bert pretraining approach.  2019;:.
[32]Petroni Fabio, Piktus Aleksandra, Fan Angela, Lewis Patrick, Yazdani Majid, Cao NicolaDe, et al.. Kilt: a benchmark for knowledge intensive language tasks.  2020;:.
[33]Petroni Fabio, Rocktäschel Tim, Riedel Sebastian, Lewis Patrick, Bakhtin Anton, Wu Yuxiang, et al.. Language models as knowledge bases?. Proceedings of Empirical Methods in Natural Language Processing 2019;:.
[34]Jack W Rae Anna, Potapenko , Siddhant M, Jayakumar Chloe, Hillier TimothyP, Lillicrap . Compressive transformers for long-range sequence modelling. Proceedings of the International Conference on Learning Representations 2020;:.
[35]Raffel Colin, Shazeer Noam, Roberts Adam, Lee Katherine, Narang Sharan, Matena Michael, et al.. Exploring the limits of transfer learning with a unified text-totext transformer. Journal of Machine Learning Research 2020;21:1--67.
[36]Ram Ashwin, Prasad Rohit, Khatri Chandra, Venkatesh Anu, Gabriel Raefer, Liu Qing, et al.. Conversational ai: The science behind the alexa prize.  2018;:.
[37]Roberts Adam, Raffel Colin, Shazeer Noam. How much knowledge can you pack into the parameters of a language model?. Proceedings of Empirical Methods in Natural Language Processing 2020;:.
[38]Roy Aurko, Saffar Mohammad, Vaswani Ashish, Grangier David. Efficient content-based sparse attention with routing transformers.  2020;:.
[39]Sellam Thibault, Das Dipanjan, Parikh Ankur P. Bleurt: Learning robust metrics for text generation. Proceedings of the Association for Computational Linguistics 2020;:.
[40]Sun Simeng, Shapira Ori, Dagan Ido, Nenkova Ani. How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation 2019;:21--29.
[41]Tian Ran, Narayan Shashi, Sellam Thibault, Parikh Ankur P. Sticking to the facts: Confident decoding for faithful data-to-text generation.  2019;:.
[42]Vaswani Ashish, Bengio Samy, Brevdo Eugene, Chollet Francois, Gomez AidanN, Gouws Stephan, et al.. Tensor2tensor for neural machine translation.  2018;:.
[43]Wang Alex, Cho Kyunghyun, Lewis Mike. Asking and answering questions to evaluate the factual consistency of summaries. Proceedings of the 2020;:.
[44]Wang Alex, Singh Amanpreet, Michael Julian, Hill Felix, Levy Omer, Samuel R Bowman . Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the International Conference on Learning Representations 2019;:.
[45]Wang Hai, Mcallester David. On-the-fly information retrieval augmentation for language models. Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events 2020;:114--119.
[46]Wiseman Sam, Shieber StuartM, Rush AlexanderM. Challenges in data-to-document generation. Proceedings of Empirical Methods in Natural Language Processing 2017;:.
[47]Wolf Thomas, Sanh Victor, Chaumond Julien, Delangue Clement. Transfertransfo: A transfer learning approach for neural network based conversational agents. NeurIPS CAI Workshop 2018;:.
[48]Yadav Deshraj, Jain Rishabh, Agrawal Harsh, Chattopadhyay Prithvijit, Singh Taranjeet, Jain Akash, et al.. Evalai: Towards better evaluation systems for ai agents.  2019;:.
[49]Zellers Rowan, Holtzman Ari, Rashkin Hannah, Bisk Yonatan, Farhadi Ali, Roesner Franziska, et al.. Defending against neural fake news. Advances in Neural Information Processing Systems 2019;:9054--9065.
[50]Zhang Yuhao, Merck Derek, Tsai EmilyBao, Manning ChristopherD, Langlotz CurtisP. Optimizing the factual correctness of a summary: A study of summarizing radiology reports.  2020;:.
[51]Zhou Chunting, Gu Jiatao, Diab Mona, Guzman Paco, Zettlemoyer Luke, Ghazvininejad Marjan. Detecting hallucinated content in conditional neural sequence generation.  2020;:.
[52]Zhu Yaoming, Lu Sidi, Zheng Lei, Guo Jiaxian, Zhang Weinan, Wang Jun, et al.. Texygen: A benchmarking platform for text generation models. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval 2018;:.
Hurdles to Progress in Long-form Question Answering
https://arxiv.org/abs/2103.06332
https://arxiv.org/abs/1706.03762
Attention Is All You Need
2017-06-12
Vaswani; Ashish; Shazeer; Noam; Parmar; Niki; Uszkoreit; Jakob; Jones; Llion; Gomez; Aidan N; Kaiser; Lukasz; Polosukhin; Illia
<div><div>
 
 
 
 
 
 
 
 
 
 <p><a href="/pdf/1706.03762">Download PDF</a></p><blockquote>
 Abstract: The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks in an encoder-decoder configuration. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer, based
solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show these models to be
superior in quality while being more parallelizable and requiring significantly
less time to train. Our model achieves 28.4 BLEU on the WMT 2014
English-to-German translation task, improving over the existing best results,
including ensembles by over 2 BLEU. On the WMT 2014 English-to-French
translation task, our model establishes a new single-model state-of-the-art
BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction
of the training costs of the best models from the literature. We show that the
Transformer generalizes well to other tasks by applying it successfully to
English constituency parsing both with large and limited training data.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Ashish Vaswani [<a href="/show-email/f53b7360/1706.03762">view email</a>]
 <br />
 
 <strong><a href="/abs/1706.03762v1">[v1]</a></strong>
 
 Mon, 12 Jun 2017 17:57:34 UTC (1,102 KB)<br />
 
 
 <strong><a href="/abs/1706.03762v2">[v2]</a></strong>
 
 Mon, 19 Jun 2017 16:49:45 UTC (1,125 KB)<br />
 
 
 <strong><a href="/abs/1706.03762v3">[v3]</a></strong>
 
 Tue, 20 Jun 2017 05:20:02 UTC (1,125 KB)<br />
 
 
 <strong><a href="/abs/1706.03762v4">[v4]</a></strong>
 
 Fri, 30 Jun 2017 17:29:30 UTC (1,124 KB)<br />
 
 <strong>[v5]</strong>
 
 Wed, 6 Dec 2017 03:30:32 UTC (1,124 KB)<br />
 </p></div></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:1706.03762

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:1706.03762 (cs)
 [Submitted on 12 Jun 2017 (v1), last revised 6 Dec 2017 (this version, v5)]

 Title: Attention Is All You Need

 Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
 Download a PDF of the paper titled Attention Is All You Need, by Ashish Vaswani and 7 other authors
 Download PDF
 Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
 Comments: 15 pages, 5 figures 
 Subjects: Computation and Language (cs.CL) ; Machine Learning (cs.LG)
 Cite as: arXiv:1706.03762 [cs.CL] 
 (or arXiv:1706.03762v5 [cs.CL] for this version) 
 https://doi.org/10.48550/arXiv.1706.03762 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Ashish Vaswani [view email]
 [v1] Mon, 12 Jun 2017 17:57:34 UTC (1,102 KB)
 [v2] Mon, 19 Jun 2017 16:49:45 UTC (1,125 KB)
 [v3] Tue, 20 Jun 2017 05:20:02 UTC (1,125 KB)
 [v4] Fri, 30 Jun 2017 17:29:30 UTC (1,124 KB)
 [v5] Wed, 6 Dec 2017 03:30:32 UTC (1,124 KB)
 Full-text links:

 Download:

 * Download a PDF of the paper titled Attention Is All You Need, by Ashish Vaswani and 7 other authors
 PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 1706
 Change to browse by:
 cs
 cs.LG

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 91 blog links

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Ashish Vaswani
 Noam Shazeer
 Niki Parmar
 Jakob Uszkoreit
 Llion Jones
 …
 a export BibTeX citation Loading...

 BibTeX formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 DagsHub Toggle
 DagsHub (What is DagsHub?)
 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Link to Influence Flower
 Influence Flower (What are Influence Flowers?)
 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Provided proper attribution is provided, Google hereby grants permission to
reproduce the tables and figures in this paper solely for use in journalistic or
scholarly works.
Attention Is All You Need
Ashish Vaswani∗
Google Brain
avaswani@google.com
Noam Shazeer∗
Google Brain
noam@google.com
Niki Parmar∗
Google Research
nikip@google.com
Jakob Uszkoreit∗
Google Research
usz@google.com
Llion Jones∗
Google Research
llion@google.com
Aidan N. Gomez∗ †
University of Toronto
aidan@cs.toronto.edu
Łukasz Kaiser∗
Google Brain
lukaszkaiser@google.com
Illia Polosukhin∗ ‡
illia.polosukhin@gmail.com
Abstract
The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks that include an encoder and a decoder. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer,
based solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show these models to
be superior in quality while being more parallelizable and requiring significantly
less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including
ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task,
our model establishes a new single-model state-of-the-art BLEU score of 41.8 after
training for 3.5 days on eight GPUs, a small fraction of the training costs of the
best models from the literature. We show that the Transformer generalizes well to
other tasks by applying it successfully to English constituency parsing both with
large and limited training data.
∗Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started
the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and
has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head
attention and the parameter-free position representation and became the other person involved in nearly every
detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and
tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and
efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and
implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating
our research.
†Work performed while at Google Brain.
‡Work performed while at Google Research.
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
arXiv:1706.03762v7 [cs.CL] 2 Aug 20231 Introduction
Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks
in particular, have been firmly established as state of the art approaches in sequence modeling and
transduction problems such as language modeling and machine translation [35, 2, 5]. Numerous
efforts have since continued to push the boundaries of recurrent language models and encoder-decoder
architectures [38, 24, 15].
Recurrent models typically factor computation along the symbol positions of the input and output
sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden
states ht, as a function of the previous hidden state ht−1 and the input for position t. This inherently
sequential nature precludes parallelization within training examples, which becomes critical at longer
sequence lengths, as memory constraints limit batching across examples. Recent work has achieved
significant improvements in computational efficiency through factorization tricks [21] and conditional
computation [32], while also improving model performance in case of the latter. The fundamental
constraint of sequential computation, however, remains.
Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in
the input or output sequences [2, 19]. In all but a few cases [27], however, such attention mechanisms
are used in conjunction with a recurrent network.
In this work we propose the Transformer, a model architecture eschewing recurrence and instead
relying entirely on an attention mechanism to draw global dependencies between input and output.
The Transformer allows for significantly more parallelization and can reach a new state of the art in
translation quality after being trained for as little as twelve hours on eight P100 GPUs.
2 Background
The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU
[16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building
block, computing hidden representations in parallel for all input and output positions. In these models,
the number of operations required to relate signals from two arbitrary input or output positions grows
in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes
it more difficult to learn dependencies between distant positions [12]. In the Transformer this is
reduced to a constant number of operations, albeit at the cost of reduced effective resolution due
to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as
described in section 3.2.
Self-attention, sometimes called intra-attention is an attention mechanism relating different positions
of a single sequence in order to compute a representation of the sequence. Self-attention has been
used successfully in a variety of tasks including reading comprehension, abstractive summarization,
textual entailment and learning task-independent sentence representations [4, 27, 28, 22].
End-to-end memory networks are based on a recurrent attention mechanism instead of sequencealigned recurrence and have been shown to perform well on simple-language question answering and
language modeling tasks [34].
To the best of our knowledge, however, the Transformer is the first transduction model relying
entirely on self-attention to compute representations of its input and output without using sequencealigned RNNs or convolution. In the following sections, we will describe the Transformer, motivate
self-attention and discuss its advantages over models such as [17, 18] and [9].
3 Model Architecture
Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].
Here, the encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence
of continuous representations z = (z1, ..., zn). Given z, the decoder then generates an output
sequence (y1, ..., ym) of symbols one element at a time. At each step the model is auto-regressive
[10], consuming the previously generated symbols as additional input when generating the next.
2Figure 1: The Transformer - model architecture.
The Transformer follows this overall architecture using stacked self-attention and point-wise, fully
connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,
respectively.
3.1 Encoder and Decoder Stacks
Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two
sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. We employ a residual connection [11] around each of
the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is
LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer
itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding
layers, produce outputs of dimension dmodel = 512.
Decoder: The decoder is also composed of a stack of N = 6 identical layers. In addition to the two
sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head
attention over the output of the encoder stack. Similar to the encoder, we employ residual connections
around each of the sub-layers, followed by layer normalization. We also modify the self-attention
sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This
masking, combined with fact that the output embeddings are offset by one position, ensures that the
predictions for position i can depend only on the known outputs at positions less than i.
3.2 Attention
An attention function can be described as mapping a query and a set of key-value pairs to an output,
where the query, keys, values, and output are all vectors. The output is computed as a weighted sum
3Scaled Dot-Product Attention Multi-Head Attention
Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several
attention layers running in parallel.
of the values, where the weight assigned to each value is computed by a compatibility function of the
query with the corresponding key.
3.2.1 Scaled Dot-Product Attention
We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of
queries and keys of dimension dk, and values of dimension dv. We compute the dot products of the
query with all keys, divide each by √
dk, and apply a softmax function to obtain the weights on the
values.
In practice, we compute the attention function on a set of queries simultaneously, packed together
into a matrix Q. The keys and values are also packed together into matrices K and V . We compute
the matrix of outputs as:
Attention(Q, K, V ) = softmax(QKT
√
dk
)V (1)
The two most commonly used attention functions are additive attention [2], and dot-product (multiplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor
of √
1
dk
. Additive attention computes the compatibility function using a feed-forward network with
a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is
much faster and more space-efficient in practice, since it can be implemented using highly optimized
matrix multiplication code.
While for small values of dk the two mechanisms perform similarly, additive attention outperforms
dot product attention without scaling for larger values of dk [3]. We suspect that for large values of
dk, the dot products grow large in magnitude, pushing the softmax function into regions where it has
extremely small gradients 4
. To counteract this effect, we scale the dot products by √
1
dk
.
3.2.2 Multi-Head Attention
Instead of performing a single attention function with dmodel-dimensional keys, values and queries,
we found it beneficial to linearly project the queries, keys and values h times with different, learned
linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of
queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional
4To illustrate why the dot products get large, assume that the components of q and k are independent random
variables with mean 0 and variance 1. Then their dot product, q · k =
Pdk
i=1 qiki, has mean 0 and variance dk.
4output values. These are concatenated and once again projected, resulting in the final values, as
depicted in Figure 2.
Multi-head attention allows the model to jointly attend to information from different representation
subspaces at different positions. With a single attention head, averaging inhibits this.
MultiHead(Q, K, V ) = Concat(head1, ..., headh)WO
where headi = Attention(QWQ
i
, KW K
i
, V WV
i
)
Where the projections are parameter matrices W
Q
i ∈ R
dmodel×dk , W K
i ∈ R
dmodel×dk , WV
i ∈ R
dmodel×dv
and WO ∈ R
hdv×dmodel
.
In this work we employ h = 8 parallel attention layers, or heads. For each of these we use
dk = dv = dmodel/h = 64. Due to the reduced dimension of each head, the total computational cost
is similar to that of single-head attention with full dimensionality.
3.2.3 Applications of Attention in our Model
The Transformer uses multi-head attention in three different ways:
• In "encoder-decoder attention" layers, the queries come from the previous decoder layer,
and the memory keys and values come from the output of the encoder. This allows every
position in the decoder to attend over all positions in the input sequence. This mimics the
typical encoder-decoder attention mechanisms in sequence-to-sequence models such as
[38, 2, 9].
• The encoder contains self-attention layers. In a self-attention layer all of the keys, values
and queries come from the same place, in this case, the output of the previous layer in the
encoder. Each position in the encoder can attend to all positions in the previous layer of the
encoder.
• Similarly, self-attention layers in the decoder allow each position in the decoder to attend to
all positions in the decoder up to and including that position. We need to prevent leftward
information flow in the decoder to preserve the auto-regressive property. We implement this
inside of scaled dot-product attention by masking out (setting to −∞) all values in the input
of the softmax which correspond to illegal connections. See Figure 2.
3.3 Position-wise Feed-Forward Networks
In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully
connected feed-forward network, which is applied to each position separately and identically. This
consists of two linear transformations with a ReLU activation in between.
FFN(x) = max(0, xW1 + b1)W2 + b2 (2)
While the linear transformations are the same across different positions, they use different parameters
from layer to layer. Another way of describing this is as two convolutions with kernel size 1.
The dimensionality of input and output is dmodel = 512, and the inner-layer has dimensionality
df f = 2048.
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input
tokens and output tokens to vectors of dimension dmodel. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In
our model, we share the same weight matrix between the two embedding layers and the pre-softmax
linear transformation, similar to [30]. In the embedding layers, we multiply those weights by √
dmodel.
5Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations
for different layer types. n is the sequence length, d is the representation dimension, k is the kernel
size of convolutions and r the size of the neighborhood in restricted self-attention.
Layer Type Complexity per Layer Sequential Maximum Path Length
Operations
Self-Attention O(n
2
· d) O(1) O(1)
Recurrent O(n · d
2
) O(n) O(n)
Convolutional O(k · n · d
2
) O(1) O(logk(n))
Self-Attention (restricted) O(r · n · d) O(1) O(n/r)
3.5 Positional Encoding
Since our model contains no recurrence and no convolution, in order for the model to make use of the
order of the sequence, we must inject some information about the relative or absolute position of the
tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the
bottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel
as the embeddings, so that the two can be summed. There are many choices of positional encodings,
learned and fixed [9].
In this work, we use sine and cosine functions of different frequencies:
P E(pos,2i) = sin(pos/100002i/dmodel)
P E(pos,2i+1) = cos(pos/100002i/dmodel)
where pos is the position and i is the dimension. That is, each dimension of the positional encoding
corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We
chose this function because we hypothesized it would allow the model to easily learn to attend by
relative positions, since for any fixed offset k, P Epos+k can be represented as a linear function of
P Epos.
We also experimented with using learned positional embeddings [9] instead, and found that the two
versions produced nearly identical results (see Table 3 row (E)). We chose the sinusoidal version
because it may allow the model to extrapolate to sequence lengths longer than the ones encountered
during training.
4 Why Self-Attention
In this section we compare various aspects of self-attention layers to the recurrent and convolutional layers commonly used for mapping one variable-length sequence of symbol representations
(x1, ..., xn) to another sequence of equal length (z1, ..., zn), with xi
, zi ∈ R
d
, such as a hidden
layer in a typical sequence transduction encoder or decoder. Motivating our use of self-attention we
consider three desiderata.
One is the total computational complexity per layer. Another is the amount of computation that can
be parallelized, as measured by the minimum number of sequential operations required.
The third is the path length between long-range dependencies in the network. Learning long-range
dependencies is a key challenge in many sequence transduction tasks. One key factor affecting the
ability to learn such dependencies is the length of the paths forward and backward signals have to
traverse in the network. The shorter these paths between any combination of positions in the input
and output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare
the maximum path length between any two input and output positions in networks composed of the
different layer types.
As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially
executed operations, whereas a recurrent layer requires O(n) sequential operations. In terms of
computational complexity, self-attention layers are faster than recurrent layers when the sequence
6
[1706.03762] Attention Is All You Need
https://arxiv.org/abs/1706.03762
https://arxiv.org/abs/2006.15055
Object-Centric Learning with Slot Attention
2020-06-26
Locatello; Francesco; Weissenborn; Dirk; Unterthiner; Thomas; Mahendran; Aravindh; Heigold; Georg; Uszkoreit; Jakob; Dosovitskiy; Alexey; Kipf
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2006.15055">Download PDF</a></p><blockquote>
 Abstract: Learning object-centric representations of complex scenes is a promising step
towards enabling efficient abstract reasoning from low-level perceptual
features. Yet, most deep learning approaches learn distributed representations
that do not capture the compositional properties of natural scenes. In this
paper, we present the Slot Attention module, an architectural component that
interfaces with perceptual representations such as the output of a
convolutional neural network and produces a set of task-dependent abstract
representations which we call slots. These slots are exchangeable and can bind
to any object in the input by specializing through a competitive procedure over
multiple rounds of attention. We empirically demonstrate that Slot Attention
can extract object-centric representations that enable generalization to unseen
compositions when trained on unsupervised object discovery and supervised
property prediction tasks.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Thomas Kipf [<a href="/show-email/d7e14818/2006.15055">view email</a>]
 <br />
 <strong><a href="/abs/2006.15055v1">[v1]</a></strong>
 Fri, 26 Jun 2020 15:31:57 UTC (2,823 KB)<br /><strong>[v2]</strong>
Wed, 14 Oct 2020 08:51:40 UTC (3,276 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2006.15055

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:2006.15055 (cs)
 [Submitted on 26 Jun 2020 (v1), last revised 14 Oct 2020 (this version, v2)]

 Title: Object-Centric Learning with Slot Attention

 Authors: Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf
 Download PDF
 Abstract: Learning object-centric representations of complex scenes is a promising step towards enabling efficient abstract reasoning from low-level perceptual features. Yet, most deep learning approaches learn distributed representations that do not capture the compositional properties of natural scenes. In this paper, we present the Slot Attention module, an architectural component that interfaces with perceptual representations such as the output of a convolutional neural network and produces a set of task-dependent abstract representations which we call slots. These slots are exchangeable and can bind to any object in the input by specializing through a competitive procedure over multiple rounds of attention. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions when trained on unsupervised object discovery and supervised property prediction tasks.
 Comments: NeurIPS 2020. Code available at this https URL 
 Subjects: Machine Learning (cs.LG) ; Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
 Cite as: arXiv:2006.15055 [cs.LG]  
 (or arXiv:2006.15055v2 [cs.LG] for this version) 
 https://doi.org/10.48550/arXiv.2006.15055 
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: Thomas Kipf [view email]
 [v1] Fri, 26 Jun 2020 15:31:57 UTC (2,823 KB)
 [v2] Wed, 14 Oct 2020 08:51:40 UTC (3,276 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 2006
 Change to browse by:
 cs
 cs.CV
 stat
 stat.ML

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Francesco Locatello
 Dirk Weissenborn
 Thomas Unterthiner
 Aravindh Mahendran
 Georg Heigold
 …
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Object-Centric Learning with Slot Attention
Francesco Locatello 
Dept. of Computer Science
ETH Zurich
Max-Planck Institute for Intelligent Systems
Dirk Weissenborn 
Google Research
Brain Team
Thomas Unterthiner 
Google Research
Brain Team
Aravindh Mahendran 
Google Research
Brain Team
Georg Heigold 
Google Research
Brain Team
Jakob Uszkoreit 
Google Research
Brain Team
Alexey Dosovitskiy 
Google Research
Brain Team
Thomas Kipf 
Google Research
Brain Team
Object-Centric Learning with Slot Attention
Learning object-centric representations of complex scenes is a promising step towards enabling efficient abstract reasoning from low-level perceptual features. Yet, most deep learning approaches learn distributed representations that do not capture the compositional properties of natural scenes. In this paper, we present the Slot Attention module, an architectural component that interfaces with perceptual representations such as the output of a convolutional neural network and produces a set of task-dependent abstract representations which we call slots. These slots are exchangeable and can bind to any object in the input by specializing through a competitive procedure over multiple rounds of attention. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions when trained on unsupervised object discovery and supervised property prediction tasks. † Work done while interning at Google, * equal contribution, ‡ equal advising. Contact:
Introduction
Object-centric representations have the potential to improve sample efficiency and generalization of machine learning algorithms across a range of application domains, such as visual reasoning [1], modeling of structured environments [2], multi-agent modeling [3][4][5], and simulation of interacting physical systems [6][7][8]. Obtaining object-centric representations from raw perceptual input, such as an image or a video, is challenging and often requires either supervision [1,3,9,10] or task-specific architectures [2,11]. As a result, the step of learning an object-centric representation is often skipped entirely. Instead, models are typically trained to operate on a structured representation of the environment that is obtained, for example, from the internal representation of a simulator [6,8] or of a game engine [4,5].
To overcome this challenge, we introduce the Slot Attention module, a differentiable interface between perceptual representations (e.g., the output of a CNN) and a set of variables called slots. Using an iterative attention mechanism, Slot Attention produces a set of output vectors with permutation symmetry. Unlike capsules used in Capsule Networks [12,13], slots produced by Slot Attention do not specialize to one particular type or class of object, which could harm generalization. Instead, they act akin to object files [14], i.e., slots use a common representational format: each slot can store (and bind to) any object in the input. This allows Slot Attention to generalize in a systematic way to unseen compositions, more objects, and more slots.
Slot Attention is a simple and easy to implement architectural component that can be placed, for example, on top of a CNN [15] encoder to extract object representations from an image and is trained end-to-end with a downstream task. In this paper, we consider image reconstruction and set prediction as downstream tasks to showcase the versatility of our module both in a challenging unsupervised object discovery setup and in a supervised task involving set-structured object property prediction.    Our main contributions are as follows: (i) We introduce the Slot Attention module, a simple architectural component at the interface between perceptual representations (such as the output of a CNN) and representations structured as a set. (ii) We apply a Slot Attention-based architecture to unsupervised object discovery, where it matches or outperforms relevant state-of-the-art approaches [16,17], while being more memory efficient and significantly faster to train. (iii) We demonstrate that the Slot Attention module can be used for supervised object property prediction, where the attention mechanism learns to highlight individual objects without receiving direct supervision on object segmentation.
Methods
In this section, we introduce the Slot Attention module (Figure 1a; Section 2.1) and demonstrate how it can be integrated into an architecture for unsupervised object discovery (Figure 1b; Section 2.2) and into a set prediction architecture (Figure 1c; Section 2.3).
Slot Attention Module
The Slot Attention module (Figure 1a) maps from a set of N input feature vectors to a set of K output vectors that we refer to as slots. Each vector in this output set can, for example, describe an object or an entity in the input. The overall module is described in Algorithm 1 in pseudo-code 1 .
Slot Attention uses an iterative attention mechanism to map from its inputs to the slots. Slots are initialized at random and thereafter refined at each iteration t = 1 . . . T to bind to a particular part (or grouping) of the input features. Randomly sampling initial slot representations from a common distribution allows Slot Attention to generalize to a different number of slots at test time.
At each iteration, slots compete for explaining parts of the input via a softmax-based attention mechanism [18-20] and update their representation using a recurrent update function. The final representation in each slot can be used in downstream tasks such as unsupervised object discovery (Figure 1b) or supervised set prediction (Figure 1c).
We now describe a single iteration of Slot Attention on a set of input features, inputs ∈ R N ×Dinputs , with K output slots of dimension D slots (we omit the batch dimension for clarity). We use learnable linear transformations k, q, and v to map inputs and slots to a common dimension D.
Slot Attention uses dot-product attention [19] with attention coefficients that are normalized over the slots, i.e., the queries of the attention mechanism. This choice of normalization introduces competition between the slots for explaining parts of the input. We further follow the common practice of setting the softmax temperature to a fixed value of 
Algorithm 1 Slot Attention module. The input is a set of N vectors of dimension D inputs which is mapped to a set of K slots of dimension D slots . We initialize the slots by sampling their initial values as independent samples from a Gaussian distribution with shared, learnable parameters µ ∈ R Dslots and σ ∈ R Dslots . In our experiments we set the number of iterations to T = 3.
1: Input: inputs ∈ R N ×Dinputs , slots ∼ N (µ, diag(σ)) ∈ R K×Dslots 2: Layer params: k, q, v: linear projections for attention; GRU; MLP; LayerNorm (x3) 3: inputs = LayerNorm (inputs) 4: for t = 0 . . . T
5:
slots_prev = slots 6: slots = LayerNorm (slots) 7: attn = Softmax ( 1 √ D k(inputs) · q(slots) T , axis='slots') # norm. over slots 8: updates = WeightedMean (weights=attn + , values=v(inputs)) # aggregate 9: slots = GRU (state=slots_prev, inputs=updates) # GRU update (per slot) 10: slots += MLP (LayerNorm (slots)) # optional residual MLP (per slot) 11:
return slots
In other words, the normalization ensures that attention coefficients sum to one for each individual input feature vector, which prevents the attention mechanism from ignoring parts of the input. To aggregate the input values to their assigned slots, we use a weighted mean as follows:
updates := W T · v(inputs) ∈ R K×D where W i,j := attn i,j N l=1 attn l,j .(2)
The weighted mean helps improve stability of the attention mechanism (compared to using a weighted sum) as in our case the attention coefficients are normalized over the slots. In practice we further add a small offset to the attention coefficients to avoid numerical instability.
The aggregated updates are finally used to update the slots via a learned recurrent function, for which we use a Gated Recurrent Unit (GRU) [21] with D slots hidden units. We found that transforming the GRU output with an (optional) multi-layer perceptron (MLP) with ReLU activation and a residual connection [22] can help improve performance. Both the GRU and the residual MLP are applied independently on each slot with shared parameters. We apply layer normalization (LayerNorm) [23] both to the inputs of the module and to the slot features at the beginning of each iteration and before applying the residual MLP. While this is not strictly necessary, we found that it helps speed up training convergence. The overall time-complexity of the module is O (T · D · N · K).
We identify two key properties of Slot Attention: (1) permutation invariance with respect to the input (i.e., the output is independent of permutations applied to the input and hence suitable for sets) and
(2) permutation equivariance with respect to the order of the slots (i.e., permuting the order of the slots after their initialization is equivalent to permuting the output of the module). More formally: Proposition 1. Let SlotAttention(inputs, slots) ∈ R K×Dslots be the output of the Slot Attention module (Algorithm 1), where inputs ∈ R N ×Dinputs and slots ∈ R K×Dslots . Let π i ∈ R N ×N and π s ∈ R K×K be arbitrary permutation matrices. Then, the following holds:
SlotAttention(π i · inputs, π s · slots) = π s · SlotAttention(inputs, slots) .
The proof is in the supplementary material. The permutation equivariance property is important to ensure that slots learn a common representational format and that each slot can bind to any object in the input.
Object Discovery
Set-structured hidden representations are an attractive choice for learning about objects in an unsupervised fashion: each set element can capture the properties of an object in a scene, without assuming a particular order in which objects are described. Since Slot Attention transforms input representations into a set of vectors, it can be used as part of the encoder in an autoencoder architecture for unsupervised object discovery.
The autoencoder is tasked to encode an image into a set of hidden representations (i.e., slots) that, taken together, can be decoded back into the image space to reconstruct the original input. The slots thereby act as a representational bottleneck and the architecture of the decoder (or decoding process) is typically chosen such that each slot decodes only a region or part of the image [16, 17, 24-27]. These regions/parts are then combined to arrive at the full reconstructed image.
Encoder Our encoder consists of two components: (i) a CNN backbone augmented with positional embeddings, followed by (ii) a Slot Attention module. The output of Slot Attention is a set of slots, that represent a grouping of the scene (e.g. in terms of objects).
Decoder Each slot is decoded individually with the help of a spatial broadcast decoder [28], as used in IODINE [16]: slot representations are broadcasted onto a 2D grid (per slot) and augmented with position embeddings. Each such grid is decoded using a CNN (with parameters shared across the slots) to produce an output of size W × H × 4, where W and H are width and height of the image, respectively. The output channels encode RGB color channels and an (unnormalized) alpha mask. We subsequently normalize the alpha masks across slots using a Softmax and use them as mixture weights to combine the individual reconstructions into a single RGB image.
Set Prediction
Set representations are commonly used in tasks across many data modalities ranging from point cloud prediction [29,30], classifying multiple objects in an image [31], or generation of molecules with desired properties [32,33]. In the example considered in this paper, we are given an input image and a set of prediction targets, each describing an object in the scene. The key challenge in predicting sets is that there are K! possible equivalent representations for a set of K elements, as the order of the targets is arbitrary. This inductive bias needs to be explicitly modeled in the architecture to avoid discontinuities in the learning process, e.g. when two semantically specialized slots swap their content throughout training [31,34]. The output order of Slot Attention is random and independent of the input order, which addresses this issue. Therefore, Slot Attention can be used to turn a distributed representation of an input scene into a set representation where each object can be separately classified with a standard classifier as shown in Figure 1c.
Encoder We use the same encoder architecture as in the object discovery setting (Section 2.2), namely a CNN backbone augmented with positional embeddings, followed by Slot Attention, to arrive at a set of slot representations.
Classifier For each slot, we apply a MLP with parameters shared between slots. As the order of both predictions and labels is arbitrary, we match them using the Hungarian algorithm [35]. We leave the exploration of other matching algorithms [36,37] for future work.
Related Work
Object discovery Our object discovery architecture is closely related to a line of recent work on compositional generative scene models [16, 17, 24-27, [38][39][40][41][42][43][44] that represent a scene in terms of a collection of latent variables with the same representational format. Closest to our approach is the IODINE [16] model, which uses iterative variational inference [45] to infer a set of latent variables, each describing an object in an image. In each inference iteration, IODINE performs a decoding step followed by a comparison in pixel space and a subsequent encoding step. Related models such as MONet [17] and GENESIS [27] similarly use multiple encode-decode steps. Our model instead replaces this procedure with a single encoding step using iterated attention, which improves computational efficiency. Further, this allows our architecture to infer object representations and attention masks even in the absence of a decoder, opening up extensions beyond auto-encoding, such as contrastive representation learning for object discovery [46] or direct optimization of a downstream task like control or planning. Our attention-based routing procedure could also be employed in conjunction with patch-based decoders, used in architectures such as AIR [26], SQAIR [40], and related approaches [41][42][43][44], as an alternative to the typically employed autoregressive encoder [26,40]. Our approach is orthogonal to methods using adversarial training [47][48][49] or contrastive learning [46] for object discovery: utilizing Slot Attention in such a setting is an interesting avenue for future work.
Neural networks for sets A range of recent methods explore set encoding [34,50,51], generation [31,52], and set-to-set mappings [20,53]. Graph neural networks [54][55][56][57] and in particular the self-attention mechanism of the Transformer model [20] are frequently used to transform sets of elements with constant cardinality (i.e., number of set elements). Slot Attention addresses the problem of mapping from one set to another set of different cardinality while respecting permutation symmetry of both the input and the output set. Most prior works (e.g. [53,58,59]), with the exception of the Deep Set Prediction Network (DSPN) [31,60], learn an ordered representation of the output set with learned per-element initialization, which prevents these approaches from generalizing to a different set cardinality at test time. DSPN respects permutation symmetry by running an inner gradient descent loop for each example, which requires many steps for convergence and careful tuning of several loss hyperparmeters. Instead, Slot Attention directly maps from set to set using only a few attention iterations and a single task-specific loss function. In concurrent work, both the DETR [59] and the TSPN [61] model propose to use a Transformer [20] for conditional set generation. DETR does not respect the permutation symmetry of the output set, as it learns a slot-specific initialization, whereas TSPN uses a symmetric, random initialization.
Iterative routing Our iterative attention mechanism shares similarlities with iterative routing mechanisms typically employed in variants of Capsule Networks [12,13,62]. The closest such variant is inverted dot-product attention routing [62] which similarly uses a dot product attention mechanism to obtain assignment coefficients between representations. Their method (in line with other capsule models) however does not have permutation symmetry as each input-output pair is assigned a separately parameterized transformation. The low-level details in how the attention mechanism is normalized and how updates are aggregated, and the considered applications also differ significantly between the two approaches.
Mixtures of experts Expert models [63][64][65][66][67] are related to our slot-based approach, but do not fully share parameters between individual experts. This results in the specialization of individual experts to, e.g., different tasks or object types. In Slot Attention, slots use a common representational format and each slot can bind to any part of the input.
Soft clustering Our routing procedure is related to soft k-means clustering [68] (where slots corresponds to centroids) with two key differences: We use a dot product similarity with learned linear projections and we use a parameterized, learnable update function.
Recurrent attention Our method is related to recurrent attention models used in image modeling and scene decomposition [26,40,[69][70][71]. Recurrent models for set prediction have also been considered in this context without using attention mechanisms [72,73]. This line of work, however, infers one slot or representation per time step in an auto-regressive manner, whereas Slot Attention updates all slots simultaneously at each step while maintaining permutation symmetry.
Experiments
The goal of this section is to evaluate the Slot Attention module on two object-centric tasks-one being supervised and the other one being unsupervised-as described in  Figure 2: (Left) Adjusted Rand Index (ARI) scores (in %, mean ± stddev for 5 seeds) for unsupervised object discovery in multi-object datasets. In line with previous works [16, 17, 27], we exclude background labels in ARI evaluation. *denotes that one outlier was excluded from evaluation. (Right) Effect of increasing the number of Slot Attention iterations T at test time (for a model trained on CLEVR6 with T = 3 and K = 7 slots), tested on CLEVR6 (K = 7) and CLEVR10 (K = 11).
original CLEVR dataset [75] which contains a training-validation split of 70K and 15K images of rendered objects respectively. Each image can contain between three and ten objects and has property annotations for each object (position, shape, material, color, and size). In some experiments, we filter the CLEVR dataset to contain only scenes with at maximum 6 objects; we call this dataset CLEVR6 and we refer to the original full dataset as CLEVR10 for clarity.
Object Discovery
Training The training setup is unsupervised: the learning signal is provided by the (mean squared) image reconstruction error. We train the model using the Adam optimizer [76] with a learning rate of 4 × 10 −4 and a batch size of 64 (using a single GPU). We further make use of learning rate warmup [77] to prevent early saturation of the attention mechanism and an exponential decay schedule in the learning rate, which we found to reduce variance. At training time, we use T = 3 iterations of Slot Attention. We use the same training setting across all datasets, apart from the number of slots K: we use K = 7 slots for CLEVR6, K = 6 slots for Multi-dSprites (max. 5 objects per scene), and K = 4 for Tetrominoes (3 objects per scene). Even though the number of slots in Slot Attention can be set to a different value for each input example, we use the same value K for all examples in the training set to allow for easier batching.
Metrics In line with previous works [16, 17], we compare the alpha masks produced by the decoder (for each individual object slot) with the ground truth segmentation (excluding the background) using the Adjusted Rand Index (ARI) score [78,79]. ARI is a score to measure clustering similarity, ranging from 0 (random) to 1 (perfect match). To compute the ARI score, we use the implementation provided by Kabra et al. [74].
Results Quantitative results are summarized in Table 1 and Figure 2. In general, we observe that our model compares favorably against two recent state-of-the-art baselines: IODINE [16] and MONet [17]. We also compare against a simple MLP-based baseline (Slot MLP) which performs better than chance, but due to its ordered representation is unable to model the compositional nature of this task. We note a failure mode of our model: In rare cases it can get stuck in a suboptimal solution on the Tetrominoes dataset, where it segments the image into stripes. This leads to a significantly higher reconstruction error on the training set, and hence such an outlier can easily be identified at training time. We excluded a single such outlier (1 out of 5 seeds) from the final score in Table 1. We expect that careful tuning of the training hyperparameters particularly for this dataset could alleviate this issue, but we opted for a single setting shared across all datasets for simplicity. Compared to IODINE, our model is significantly more efficient in terms of both memory consumption and runtime. On CLEVR6, we can use a batch size of up to 64 on a single V100 GPU as opposed to 4 in [16]. Similarly, when using 8 V100 GPUs in parallel, model training on CLEVR6 takes approximately 24hrs for Slot Attention as opposed to approximately 7 days for IODINE 2 .
In Figure 2, we investigate to what degree our model generalizes when using more Slot Attention iterations at test time, while being trained with a fixed number of T = 3 iterations. We further evaluate generalization to more objects (CLEVR10) compared to the training set (CLEVR6). We observe that segmentation scores significantly improve beyond the numbers reported in Table 1 when using more iterations. This improvement is stronger when testing on CLEVR10 scenes with more objects. For this experiment, we increase the number of slots from K = 7 (training) to K = 11 at  test time. Overall, segmentation performance remains strong even when testing on scenes that contain more objects than seen during training.
We visualize discovered object segmentations in Figure 3 for all three datasets. The model learns to keep slots empty (only capturing the background) if there are more slots than objects. We find that Slot Attention typically spreads the uniform background across all slots instead of capturing it in just a single slot, which is likely an artifact of the attention mechanism that does not harm object disentanglement or reconstruction quality. We further visualize how the attention mechanism segments the scene over the individual attention iterations, and we inspect scene reconstructions from each individual iteration (the model has been trained to reconstruct only after the final iteration). It can be seen that the attention mechanism learns to specialize on the extraction of individual objects already at the second iteration, whereas the attention map of the first iteration still maps parts of multiple objects into a single slot. We further visualize discovered object segmentations for a Slot Attention model trained on a greyscale version of CLEVR6 in Figure 4, which Slot Attention handles without issue despite the lack of object color as a distinguishing feature.
As our object discovery architecture uses the same decoder and reconstruction loss as IODINE [16], we expect it to similarly struggle with scenes containing more complicated backgrounds and textures. Utilizing different perceptual [49,80] or contrastive losses [46] could help overcome this limitation. We discuss further limitations and future work in Section 5 and in the supplementary material. Summary Slot Attention is highly competitive with prior approaches on unsupervised scene decomposition, both in terms of quality of object segmentation and in terms of training speed and memory efficiency. At test time, Slot Attention can be used without a decoder to obtain object-centric representations from an unseen scene.
Set Prediction
Training We train our model using the same hyperparameters as in Section 4.1 except we use a batch size of 512 and striding in the encoder. On CLEVR10, we use K = 10 object slots to be in line with [31]. We train our model using a single NVIDIA Tesla V100 GPU.   Metrics Following Zhang et al.
[31], we compute the Average Precision (AP) as commonly used in object detection [81]. A prediction (object properties and position) is considered correct if there is a matching object with exactly the same properties (shape, material, color, and size) within a certain distance threshold (∞ means we do not enforce any threshold). The predicted position coordinates are scaled to [−3, 3]. We zero-pad the targets and predict an additional indicator score in [0, 1] corresponding to the presence probability of an object (1 means there is an object) which we then use as prediction confidence to compute the AP.
Results
In Figure 5 (left) we report results in terms of Average Precision for supervised object property prediction on CLEVR10 (using T = 3 for Slot Attention at both train and test time). We compare to both the DSPN results of [31] and the Slot MLP baseline. Overall, we observe that our approach matches or outperforms the DSPN baseline. The performance of our method degrades gracefully at more challenging distance thresholds (for the object position feature) maintaining a reasonably small variance. Note that the DSPN baseline [31] uses a significantly deeper ResNet 34 [22] image encoder. In Figure 5 (center) we observe that increasing the number of attention iterations at test time generally improves performance. Slot Attention can naturally handle more objects at test time by changing the number of slots. In Figure 5 (right) we observe that the AP degrades gracefully if we train a model on CLEVR6 (with K = 6 slots) and test it with more objects.
Intuitively, to solve this set prediction task each slot should attend to a different object. In Figure 6, we visualize the attention maps of each slot for two CLEVR images. In general, we observe that the attention maps naturally segment the objects. We remark that the method is only trained to predict the property of the objects, without any segmentation mask. Quantitatively, we can evaluate the Adjusted Rand Index (ARI) scores of the attention masks. On CLEVR10 (with masks), the attention masks produced by Slot Attention achieve an ARI of 78.0% ± 2.9 (to compute the ARI we downscale the input image to 32 × 32). Note that the masks evaluated in Table 1 are not the attention maps but are predicted by the object discovery decoder.
Summary Slot Attention learns a representation of objects for set-structured property prediction tasks and achieves results competitive with a prior state-of-the-art approach while being significantly easier to implement and tune. Further, the attention masks naturally segment the scene, which can be valuable for debugging and interpreting the predictions of the model.
Conclusion
We have presented the Slot Attention module, a versatile architectural component that learns objectcentric abstract representations from low-level perceptual input. The iterative attention mechanism used in Slot Attention allows our model to learn a grouping strategy to decompose input features into a set of slot representations. In experiments on unsupervised visual scene decomposition and super-vised object property prediction we have shown that Slot Attention is highly competitive with prior related approaches, while being more efficient in terms of memory consumption and computation.
A natural next step is to apply Slot Attention to video data or to other data modalities, e.g. for clustering of nodes in graphs, on top of a point cloud processing backbone or for textual or speech data. It is also promising to investigate other downstream tasks, such as reward prediction, visual reasoning, control, or planning.
Broader Impact
The Slot Attention module allows to learn object-centric representations from perceptual input. As such, it is a general module that can be used in a wide range of domains and applications. In our paper, we only consider artificially generated datasets under well-controlled settings where slots are expected to specialize to objects. However, the specialization of our model is implicit and fully driven by the downstream task. We remark that as a concrete measure to assess whether the module specialized in unwanted ways, one can visualize the attention masks to understand how the input features are distributed across the slots (see Figure 6). While more work is required to properly address the usefulness of the attention coefficients in explaining the overall predictions of the network (especially if the input features are not human interpretable), we argue that they may serve as a step towards more transparent and interpretable predictions. 
Supplementary Material for Object-Centric Learning with Slot Attention
In Section A, we highlight some limitations of our work as well as potential directions for future work. In Section B, we report results of an ablation study on Slot Attention. In Section C, we report further qualitative and quantitative results. In Section D, we give the proof for Proposition 1. In Section E, we report details on our implementation and experimental setting.
A Limitations
We highlight three limitations of the Slot Attention module that could potentially be addressed in future work: (i) The background of a scene receives no special treatment in Slot Attention as all slots use the same representational format. Providing special treatment for backgrounds (e.g., by assigning a separate background slot) is interesting for future work. (ii) The positional encoding used in our experiments is absolute and hence there is no translation equivariance in the model. Using a patch-based object extraction process as in [44] or an attention mechanism with relative positional encoding [82] are promising extensions. (iii) Slot Attention does not know about objects per-se: segmentation is solely driven by the downstream task, i.e., Slot Attention does not distinguish between clustering objects, colors, or simply spatial regions, but relies on the downstream task to drive the specialization to objects.
B Model Ablations
In this section, we investigate the importance of individual components and modeling choices in the Slot Attention module and compare our default choice to a variety of reasonable alternatives. For simplicity, we report all results on a smaller validation set consisting of 500 validation images for property prediction (instead of 15K) and on 320 training images for object discovery. In the unsupervised case, results on the training set and on held-out validation examples are nearly identical. Figure 7, we show the effect of taking a weighted sum as opposed to a weighted average in Line 8 of Algorithm 1. The average stabilizes training and yields significantly higher ARI and Average Precision scores (especially at the more strict distance thresholds). We can obtain a similar effect by replacing the weighted mean with a weighted sum followed by layer normalization (LayerNorm) [23].
Value aggregation In
Position embedding In Figure 8, we observe that the position embedding is not necessary for predicting categorical object properties. However, the performance in predicting the object position clearly decreases if we remove the position embedding. Similarly, the ARI score in unsupervised object discovery is significantly lower when not adding positional information to the CNN feature maps before the Slot Attention module. Figure 9, we show the effect of learning a separate set of Gaussian mean and variance parameters for the initialization of each slot compared to the default setting of using a shared set of parameters for all slots. We observe that a per-slot parameterization can increase performance slightly for the supervised task, but decreases performance on the unsupervised task, compared to the default shared parameterization. We remark that when learning a separate set of parameters for each slot, adding additional slots at test time is not possible without re-training.
Slot initialization In
Attention normalization axis In Figure 10, we highlight the role of the softmax axis in the attention mechanism, i.e., over which dimension the normalization is performed. Taking the softmax over the slot axis induces competition among the slots for explaining parts of the input. When we take the softmax over the input axis instead 3 (as done in regular self-attention), the attention coefficients for each slot will be independent of all other slots, and hence slots have no means of exchanging information, which significantly harms performance on both tasks. Figure 11, we highlight the role of the GRU in learning the update function for the slots as opposed to simply taking the output of Line 8 as the next value for the slots. We observe that the learned update function yields a noticable improvement. Figure 12, we show the impact of the number of attention iterations while training. We observe a clear benefit in having more than a single attention iteration. Having more than 3 attention iterations significantly slows down training convergence, which results in lower performance when trained for the same number of steps. This can likely be mitigated by instead decoding and applying a loss at every attention iteration as opposed to only the last one. We note that at test time, using more than 3 attention iterations (even when trained with only 3 iterations) generally improves performance.
Recurrent update function In
Attention iterations In
Layer normalization In Figure 13, we show that applying layer normalization (LayerNorm) [23] to the inputs and to the slot representations at each iteration in the Slot Attention module improves predictive performance. For set prediction, it particularly improves its ability to predict position accurately, likely because it leads to faster convergence at training time.
Feedforward network In Figure 14, we show that the residual MLP after the GRU is optional and may slow down convergence in property prediction, but may slightly improve performance on object discovery.
Softmax temperature In Figure 15, we show the effect of the softmax temperature. The scaling of √ D clearly improves the performance on both tasks.
Offset for numerical stability In Figure 16, we show that adding a small offset to the attention maps (for numerical stability) as in Algorithm 1 compared to the alternative of adding an offset to the denominator in the weighted mean does not significantly change the result in either task.
Learning rate schedules In Figure 17 and 18, we show the effect of our decay and warmup schedules. While we observe a clear benefit from the decay schedule, the warmup seem to be mostly useful in the object discovery setting, where it helps avoid failure cases of getting stuck in a suboptimal solution (e.g., clustering the image into stripes as opposed to objects). Figure 19, we show the effect of training with a larger number of slots than necessary. We train both the object discovery and property prediction methods on CLEVR6 with the number of slots we used in our CLEVR10 experiments (note that for object discovery we use an additional slot to be consistent with the baselines). We observe that knowing the precise number of objects in the dataset is generally not required. Training with more slots may even help in the property prediction experiments and is slightly harmful in the object discovery. Overall, this indicates that the model is rather robust to the number of slots (given enough slots to model each object independently). Using a (rough) upper bound to the number of objects in the dataset seem to be a reasonable selection strategy for the number of slots.              
Number of training slots In
C Further Experimental Results
C.1 Object Discovery
In Figure 20, we show qualitative segmentation results for a Slot Attention model trained on the object discovery task. This model is trained on CLEVR6 but uses K = 11 instead of the default setting of K = 7 slots during both training and testing, while all other settings remain unchanged. In this particular experiment, we trained 5 models using this setting with 5 different random seeds for model parameter initialization. Out of these 5 models, we found that a single model learned the solution of placing the background into a separate slot (which is the one we visualize). The typical solution that a Slot Attention-based model finds (for most random seeds) is to distribute the background equally over all slots, which is the solution we highlight in the main paper. In Figure 20 (bottom two rows), we further show how the model generalizes to scenes with more objects (up to 10) despite being trained on CLEVR6, i.e., on scenes containing a maximum of 6 objects. For some random slot initializations, clustering results still improve when run for more iterations. We note that this particular model learned to separate the background out into a separate (but random) slot instead of spreading it out evenly over all slots.
C.2 Set Prediction
Qualitative Results In Table 2, we show the predictions and attention coefficients of a Slot Attention model on several challenging test examples for the supervised property prediction task. The model was trained with default settings (T = 3 attention iterations) and the images are selected by hand to highlight especially difficult cases (e.g., multiple identical objects or many partially overlapping objects in one scene). Overall, we can see that the property prediction typically becomes more accurate with more iterations, although the accuracy of the position prediction may decrease. This is not surprising as we only apply the loss at t = 3, and generalization to more time steps at test time is not guaranteed. We note that one could alternatively apply the loss at every iteration during training, which has the potential to improve accuracy, but would increase computational cost. We observe that the model appears to handle multiple copies of the same object well (top). On very crowded scenes (middle and bottom), we note that the slots have a harder time segmenting the scene, which can lead to errors in the prediction. However, more iterations seem to sharpen the segmentation which in turns improves predictions.
Numerical Results To facilitate comparison with our method, we report the results of Figure 5 of the main paper (left subfigure) in numerical form in Table 3 as well as the performance of DSPN [31] with 10 iterations (as opposed to 30). We note that our approach has generally higher average AP compared to DSPN and lower variance. We remark that the published implementation of DSPN uses a significantly deeper image encoder than our model: Step-wise Loss & Coordinate Scaling We investigate a variant of our model where we apply the set prediction component and loss at every iteration of the attention mechanism, as opposed to only after the final step. A similar experiment was reported in [31] for the DSPN model. As DSPN uses a different scale for the position coordinates of objects by default, we further compare against a version of our model where we similarly use a different scale. Using a different scale for the object location increases its weight in the loss. We observe the effect of the coordinate scale and of computing the loss at each step in Figure 21. A scale of 1 corresponds to our default coordinate normalization of [0, 1], whereas larger scales correspond to a [0, scale] normalization of the coordinates (or shifted by an arbitrary constant). Overall, we observe that computing the loss at each step in Slot Attention   
D Permutation Invariance and Equivariance
D.1 Definitions
Before giving the proof for Proposition 1, we formally define permutation invariance and equivariance. Definition 1 (Permutation Invariance). A function f : R M ×D1 → R M ×D2 is permutation invariant if for any arbitrary permutation matrix π ∈ R M ×M it holds that:
f (πx) = f (x) .
Definition 2 (Permutation Equivariance). A function f : R M ×D1 → R M ×D2 is permutation equivariant if for any arbitrary permutation matrix π ∈ R M ×M it holds that:
f (πx) = πf (x) .
D.2 Proof
The proof is straightforward and is reported for completeness. We rely on the fact that the sum operation is permutation invariant.
Linear projections As the linear projections are applied independently per slot/input element with shared parameters, they are permutation equivariant.
Equation 1
The dot product of the attention mechanism (i.e. computing the matrix M ∈ R N ×K ) involves a sum over the feature axis (of dimension D) and is therefore permutation equivariant w.r.t. both input and slots. The output of the softmax is also equivariant, as:
Softmax (π s · π i · M ) (k,l) = e (πs·πi·M ) k,l s e (πs·πi·M ) k,s = e M π i (k),πs(l)
πs(l) e M π i (k),πs(l) = Softmax (M ) (πi(k),πs(l)) ,
where we indicate with e.g. π i (k) the transformation of the coordinate k with the permutation matrix π i . The second equality follows from the fact that the sum is permutation invariant.
Equation 2
The matrix product in the computation of the updates involves a sum over the input elements which makes the operation invariant w.r.t. permutations of the input order (i.e. π i ) and equivariant w.r.t. the slot order (i.e. π s ).
Slot update: The slot update applies the same network to each slot with shared parameters. Therefore, it is a permutation equivariant operation w.r.t. the slot order.
Combining all steps: As all steps in the algorithms are permutation equivariant wrt π s , the overall module is permutation equivariant. On the other hand, Equation 2 is permutation invariant w.r.t. to π i . Therefore, after the first iteration the algorithm becomes permutation invariant w.r.t. the input order.
E Implementation and Experimental Details
For the Slot Attention module we use a slot feature dimension of D = D slots = 64. The GRU has 64-dimensional hidden state and the feedforward block is a MLP with single hidden layer of size 128 and ReLU activation followed by a linear layer.
E.1 CNN Encoder
The CNN Encoder used in our experiments is depicted in Table 4 for CLEVR and Table 5 for Tetrominoes and Multi-dSprites. For the property prediction task on CLEVR, we reduce the size of the representation by using strides in the CNN backbone. All convolutional layers use padding SAME and have a bias weight. After this backbone, we add position embeddings (Section E.2) and then flatten the spatial dimensions. After applying a layer normalization, we finally add 1 × 1 convolutions which we implement as a shared MLP applied at each spatial location with one hidden layer of 64 units (32 for Tetrominoes and Multi-dSprites) with ReLU non-linearity followed by a linear layer with output dimension of 64 (32 for Tetrominoes and Multi-dSprites).  
E.3 Deconvolutional Slot Decoder
For the object discovery task, our architecture is based on an auto-encoder, where we decode the respresentations produced by Slot Attention with the help of a slot-wise spatial broadcast decoder [28] with shared parameters between slots. Each spatial broadcast decoder produces an output of size width×height×4, where the first 3 output channels denote RGB channels of the reconstructed image and the last output channel denotes a predicted alpha mask, that is later used to recombine individual slot reconstructions into a single image. The overall architecture for used for CLEVR is described in Table 6 and for Tetrominoes and Multi-dSprites in Table 7.
Spatial broadcast decoder The spatial broadcast decoder [28] is applied independently on each slot representation with shared parameters between slots. We first copy the slot representation vector of dimension D slots onto a grid of shape width×height×D slots , after which we add a positional embedding (see Section E.2). Finally, this representation is passed through several de-convolutional layers.
Slot recombination The final output of the spatial broadcast decoder for each slot is of shape width×height×4 (ignoring the slot and batch dimension). We first split the final channels into three RGB channels and a alpha mask channel. We apply a softmax activation function across slots on the alpha masks and lastly recombine all individual slot-based reconstructions into a single reconstructed image by multiplying each alpha mask with each respective reconstructed image (per slot) and lastly by performing a sum reduction on this respective output over the slot dimension to arrive at the final reconstructed image. 
E.4 Set Prediction Architecture
For the property prediction task, we apply a MLP on each slot (with shared parameters between slots) and train the overall network with the Huber loss following [31]. The Huber loss takes the form of a squared error 0.5x 2 for values |x|
Object-Centric Learning with Slot Attention
https://arxiv.org/abs/2006.15055
https://www.quora.com/Is-Arxiv-good-for-academia
Is Arxiv good for academia?
2024-01-01
None
<div><div><p></p></div></div>||||I|||| Skip to content
 Skip to search
 Sign In
 Is Arxiv good for academia?
 All related (33)
 Sort
 Recommended
 Joshua Gross
 Associate Professor of Computer Science at CSUMB · Author has 30.9K answers and 446.1M answer views
 · 5y

 Absolutely! It’s a useful way to share and access papers. It won’t replace traditional publication, but that’s not its purpose. It can’t replace publishing in peer-reviewed venues, because the only major concern about Arxiv: it will publish almost anything that looks like an academic paper. It makes no pretense to ensuring that these are publication-quality or valid work. Again, that’s not its purpose.
 Upvote ·
 9 4
 Sponsored by Online Shopping Tools
 Amazon Hates When Prime Members Do This, But They Can't Stop You.
 This simple trick can save tons of money on Amazon, but most Prime members are ignoring it.
 Learn More
 999 756
 Related questions
 More answers below
 Is publishing a paper on arXiv really a big deal?
 What are some pros and cons of publishing a paper on arXiv?
 Can you submit a paper to a journal if it has already been published on Arxiv?
 Does a publication on arXiv count as an actual publication? I'm preparing a CV for graduate school application and I have a article on arXiv with my supervisor. Should I put that into my CV?
 Which is better, SSRN or Arxiv.org?
 Inna Vishik
 Assistant professor of Physics at UC Davis · Author has 1K answers and 11.2M answer views
 · Updated 7y
 Related
 What are some pros and cons of publishing a paper on arXiv?

 Pros

 * Your paper gets out faster: people have something concrete to read as your manuscript slogs through the (months or year) long peer review process

 * Establishing primacy. If you are working on something similar as other researchers, being the first to post on arXiv is an important public marker of who was first (even if the peer review process works out so that the other one gets published first). Also, this diminishes the motivation for unscrupulous referees to 'sit' on your paper while they rush to get their version out.

 * Open access. People who don't have access to paid academic journals,

 Continue Reading

 Pros

 * Your paper gets out faster: people have something concrete to read as your manuscript slogs through the (months or year) long peer review process

 * Establishing primacy. If you are working on something similar as other researchers, being the first to post on arXiv is an important public marker of who was first (even if the peer review process works out so that the other one gets published first). Also, this diminishes the motivation for unscrupulous referees to 'sit' on your paper while they rush to get their version out.

 * Open access. People who don't have access to paid academic journals, for whatever reason, can read your paper without having to track down your e-mail to request a PDF for themselves

 Some hypothetical cons (for the paranoid)

 * Being scooped (if you post papers before they are accepted for publication). If someone is sitting on similar results, they might finish writing it up quickly and submit it to a journal with a faster peer review process than the one you submitted to. Worse yet, the editor and referees might not notice that a similar paper is already on ArXiv. Thus, those people will win where it really counts (peer reviewed paper), and few people will go back in the chronology to figure out which paper was submitted and/or posted on arXiv first.

 * Unsolicited (negative) peer reviews. If people happen to know what journal a paper was submitted to, they might send unsolicited referee reports to the editor. Seeing the actual paper on arXiv, rather than just hearing about it in a talk, will allow them to write a detailed report that the editor might pay attention to. Sounds absolutely crazy, but this has actually happened to people I know.

 * Embargo violations. Fancy journals like Nature and Science have an embargo policy whereby if they publish a paper, this paper must NOT have appeared in any other media source. ArXiv is fine , BUT if the media happens to pick up on the paper (ha!) before it is published, then Nature/Science can rescind. However, the perception that papers submitted to nature/science cannot be posted on arxiv leads many people to hesitate posting a preprint.


 So in conclusion, there are improbable risks to posting unpublished work on arXiv, but it is almost always better to post a paper while it is hot because peer review and rewriting and re-submission and peer review can take over a year.
 Upvote ·
 999 154
 9 3
 9 1
 David Simmons-Duffin
 Physicist · Author has 171 answers and 2.1M answer views
 · 9y
 Related
 What are some pros and cons of publishing a paper on arXiv?

 Here's something that makes me proud of my field (high energy theoretical physics): for us, there are absolutely no cons to posting a paper on the arXiv. Every new paper goes on the arXiv, and for most of us the arXiv is our exclusive source for new papers. Many of us have never read a high energy theory paper in a journal that was written after 1994 .

 Journals currently offer very little for high energy theorists, and they're basically an afterthought. We submit papers to them for the sake of physicists in other areas who care about journals and might make hiring decisions. I'm not sure how
 Continue Reading

 Here's something that makes me proud of my field (high energy theoretical physics): for us, there are absolutely no cons to posting a paper on the arXiv. Every new paper goes on the arXiv, and for most of us the arXiv is our exclusive source for new papers. Many of us have never read a high energy theory paper in a journal that was written after 1994 .

 Journals currently offer very little for high energy theorists, and they're basically an afterthought. We submit papers to them for the sake of physicists in other areas who care about journals and might make hiring decisions. I'm not sure how this situation originated (I probably haven't been around long enough). A negative consequence is that peer review doesn't play a very big role in high energy theory research. By the time a paper gets peer reviewed, its contents have already been read and absorbed by others. (The peer-reviewed version does usually get posted to the arXiv as a new version, so later researchers can benefit.) The peer review process itself is often not very strenuous or thorough. This is probably both a cause and effect of the fact that hep researchers have gotten used to wading though papers on the "unfiltered" arXiv and using word of mouth and common sense to find good papers.
 Upvote ·
 99 79
 9 1
 Elizabeth H. Simmons
 Particle Theorist and Physics Professor at UC San Diego · Upvoted by
 Mason Porter
 , Professor, Department of Mathematics, UCLA and
 Vijay Chidambaram
 , computer science researcher Author has 1.1K answers and 9.2M answer views
 · 8y
 Related
 Why do researchers publish in arXiv?

 We don't "publish" in arXiv. We release preprints on arXiv. And then we submit those manuscripts for publication in a peer-reviewed journal.

 Why do we put preprints on arXiv? Here are a few compelling reasons:

 * It gives free, instant, searchable and worldwide distribution of our work to a targeted audience that cares about our topic.

 * It immediately establishes when we released an idea into the world. Publication by journals can take months to years, depending on the discipline.

 * It allows us to get feedback from experts so we can improve the manuscript before submitting it for publication.

 * It e

 Continue Reading

 We don't "publish" in arXiv. We release preprints on arXiv. And then we submit those manuscripts for publication in a peer-reviewed journal.

 Why do we put preprints on arXiv? Here are a few compelling reasons:

 * It gives free, instant, searchable and worldwide distribution of our work to a targeted audience that cares about our topic.

 * It immediately establishes when we released an idea into the world. Publication by journals can take months to years, depending on the discipline.

 * It allows us to get feedback from experts so we can improve the manuscript before submitting it for publication.

 * It enters the manuscript into a tracking system that shows which papers reference which other papers. That is helpful both for literature searches and for roughly assessing a paper's intellectual impact.

 Upvote ·
 999 481
 99 15
 99 19
 Promoted by Masterworks
 Lila Fisher
 Art Education | @Masterworks
 · 1y
 Did Harry inherit more than William?

 Yes and no. When the Queen mother died back in 2002 she left more money to Harry, knowing that William would eventually become Prince and inherit the Duchy of Cornwall, a private estate valued at over $1 billion.

 Which is exactly what happened when Queen Elizabeth passed away in 2022. So overall, Willian inherited quite a bit more than Harry.

 But this $1 billion estate is nothing compared to the $10 billion that King Charles received tax free, thanks to a clever investment by the Queen into one of the world’s most exclusive asset classes … fine art.

 The Royal family has been investing a portion o
 Continue Reading

 Yes and no. When the Queen mother died back in 2002 she left more money to Harry, knowing that William would eventually become Prince and inherit the Duchy of Cornwall, a private estate valued at over $1 billion.

 Which is exactly what happened when Queen Elizabeth passed away in 2022. So overall, Willian inherited quite a bit more than Harry.

 But this $1 billion estate is nothing compared to the $10 billion that King Charles received tax free, thanks to a clever investment by the Queen into one of the world’s most exclusive asset classes … fine art.

 The Royal family has been investing a portion of their fortune into art for centuries …

 But the Royals are far from the only billionaires who have invested a significant chunk of their fortunes in this emerging asset class.

 Billionaires like David Geffen, Jeff Bezos, and Bill Gates are all avid art collectors.

 The recently-named “richest man in the world,” Berard Arnault, who is the CEO of luxury-goods conglomerate LVMH, has an extensive collection of art.

 Plus, there’s a long list of celebrities like Jay-Z, Oprah Winfrey, Steven Spielberg, Jack Nicholson, Madonna, and Ellen Degeneres who have invested millions into art.

 The only problem is, regular investors have always been locked out of this asset class because it usually takes millions of dollars to purchase just one iconic piece.

 Until now. Investing in art just got easier, thanks to a revolutionary new investment platform called Masterworks.

 Masterworks provides a simple, secure way for anyone to invest in shares of paintings from famous artists like Picasso, Banksy and Basquiat.

 Which is great, because contemporary art prices have outpaced the S&P 500 by 131% since 1995 … and real estate and gold by more than three to one over the same period.

 Even better, art acts as a great hedge during periods of high inflation and market volatility.

 Which is why Blackrock CEO, Larry Fink, who manages nearly $10 trillion in assets, recently called art “the new gold” because of its remarkable ability to “preserve wealth”

 These are just a few reasons why billionaires like Larry Fink, Jeff Bezos, and Oprah Winfrey are pouring hundreds of millions into this asset class.

 Thanks to Masterworks, you can join them. All you need to do is go to this link to sign up for a free membership.

 See important disclosures: Masterworks

 Upvote ·
 12.6K 12.6K
 1.5K 1.5K
 999 481
 Related questions
 Is publishing a paper on arXiv really a big deal?
 What are some pros and cons of publishing a paper on arXiv?
 Can you submit a paper to a journal if it has already been published on Arxiv?
 Does a publication on arXiv count as an actual publication? I'm preparing a CV for graduate school application and I have a article on arXiv with my supervisor. Should I put that into my CV?
 Which is better, SSRN or Arxiv.org?
 What are some good alternatives to Arxiv?
 How much time does it take to publish a paper in Arxiv?
 What is arxiv.org?
 Does uploading an unpublished paper to arXiv hurt the chances of publishing it in a reputable Journal?
 What percent of papers submitted to arXiv are "junk"?
 How long can a paper be in arXiv without being published in a journal before you suspect it may not have enough academic quality?
 Is it better to post rejected papers on Arxiv or not at all?
 What is the difference between arXiv and Arxiv?
 What are the consequences of submitting an article to ArXiv before trying to get it published?
 What is the meaning of having a paper accepted on Arxiv?
 Related questions
 Is publishing a paper on arXiv really a big deal?
 What are some pros and cons of publishing a paper on arXiv?
 Can you submit a paper to a journal if it has already been published on Arxiv?
 Does a publication on arXiv count as an actual publication? I'm preparing a CV for graduate school application and I have a article on arXiv with my supervisor. Should I put that into my CV?
 Which is better, SSRN or Arxiv.org?
 What are some good alternatives to Arxiv?
 Advertisement
 About · Careers · Privacy · Terms · Contact · Languages · Your Ad Choices · Press ·
 © Quora, Inc. 2024
Please enable Javascript and refresh the page to continue
Skip to content
 Skip to search
 Sign In
 Is Arxiv good for academia?
 All related (33)
 Sort
 Recommended
 Joshua Gross
 Associate Professor of Computer Science at CSUMB · Author has 30.9K answers and 446.1M answer views
 · 5y
 Absolutely! It’s a useful way to share and access papers. It won’t replace traditional publication, but that’s not its purpose. It can’t replace publishing in peer-reviewed venues, because the only major concern about Arxiv: it will publish almost anything that looks like an academic paper. It makes no pretense to ensuring that these are publication-quality or valid work. Again, that’s not its purpose.
 Upvote ·
 9 4
 Sponsored by Online Shopping Tools
 Amazon Hates When Prime Members Do This, But They Can't Stop You.
 This simple trick can save tons of money on Amazon, but most Prime members are ignoring it.
 Learn More
 999 756
 Related questions
 More answers below
 Is publishing a paper on arXiv really a big deal?
 What are some pros and cons of publishing a paper on arXiv?
 Can you submit a paper to a journal if it has already been published on Arxiv?
 Does a publication on arXiv count as an actual publication? I'm preparing a CV for graduate school application and I have a article on arXiv with my supervisor. Should I put that into my CV?
 Which is better, SSRN or Arxiv.org?
 Inna Vishik
 Assistant professor of Physics at UC Davis · Author has 1K answers and 11.2M answer views
 · Updated 7y
 Related
 What are some pros and cons of publishing a paper on arXiv?
 Pros
 * Your paper gets out faster: people have something concrete to read as your manuscript slogs through the (months or year) long peer review process
 * Establishing primacy. If you are working on something similar as other researchers, being the first to post on arXiv is an important public marker of who was first (even if the peer review process works out so that the other one gets published first). Also, this diminishes the motivation for unscrupulous referees to 'sit' on your paper while they rush to get their version out.
 * Open access. People who don't have access to paid academic journals,
 Continue Reading
 Pros
 * Your paper gets out faster: people have something concrete to read as your manuscript slogs through the (months or year) long peer review process
 * Establishing primacy. If you are working on something similar as other researchers, being the first to post on arXiv is an important public marker of who was first (even if the peer review process works out so that the other one gets published first). Also, this diminishes the motivation for unscrupulous referees to 'sit' on your paper while they rush to get their version out.
 * Open access. People who don't have access to paid academic journals, for whatever reason, can read your paper without having to track down your e-mail to request a PDF for themselves
 Some hypothetical cons (for the paranoid)
 * Being scooped (if you post papers before they are accepted for publication). If someone is sitting on similar results, they might finish writing it up quickly and submit it to a journal with a faster peer review process than the one you submitted to. Worse yet, the editor and referees might not notice that a similar paper is already on ArXiv. Thus, those people will win where it really counts (peer reviewed paper), and few people will go back in the chronology to figure out which paper was submitted and/or posted on arXiv first.
 * Unsolicited (negative) peer reviews. If people happen to know what journal a paper was submitted to, they might send unsolicited referee reports to the editor. Seeing the actual paper on arXiv, rather than just hearing about it in a talk, will allow them to write a detailed report that the editor might pay attention to. Sounds absolutely crazy, but this has actually happened to people I know.
 * Embargo violations. Fancy journals like Nature and Science have an embargo policy whereby if they publish a paper, this paper must NOT have appeared in any other media source. ArXiv is fine , BUT if the media happens to pick up on the paper (ha!) before it is published, then Nature/Science can rescind. However, the perception that papers submitted to nature/science cannot be posted on arxiv leads many people to hesitate posting a preprint.
 So in conclusion, there are improbable risks to posting unpublished work on arXiv, but it is almost always better to post a paper while it is hot because peer review and rewriting and re-submission and peer review can take over a year.
 Upvote ·
 999 154
 9 3
 9 1
 David Simmons-Duffin
 Physicist · Author has 171 answers and 2.1M answer views
 · 9y
 Related
 What are some pros and cons of publishing a paper on arXiv?
 Here's something that makes me proud of my field (high energy theoretical physics): for us, there are absolutely no cons to posting a paper on the arXiv. Every new paper goes on the arXiv, and for most of us the arXiv is our exclusive source for new papers. Many of us have never read a high energy theory paper in a journal that was written after 1994 .
 Journals currently offer very little for high energy theorists, and they're basically an afterthought. We submit papers to them for the sake of physicists in other areas who care about journals and might make hiring decisions. I'm not sure how
 Continue Reading
 Here's something that makes me proud of my field (high energy theoretical physics): for us, there are absolutely no cons to posting a paper on the arXiv. Every new paper goes on the arXiv, and for most of us the arXiv is our exclusive source for new papers. Many of us have never read a high energy theory paper in a journal that was written after 1994 .
 Journals currently offer very little for high energy theorists, and they're basically an afterthought. We submit papers to them for the sake of physicists in other areas who care about journals and might make hiring decisions. I'm not sure how this situation originated (I probably haven't been around long enough). A negative consequence is that peer review doesn't play a very big role in high energy theory research. By the time a paper gets peer reviewed, its contents have already been read and absorbed by others. (The peer-reviewed version does usually get posted to the arXiv as a new version, so later researchers can benefit.) The peer review process itself is often not very strenuous or thorough. This is probably both a cause and effect of the fact that hep researchers have gotten used to wading though papers on the "unfiltered" arXiv and using word of mouth and common sense to find good papers.
 Upvote ·
 99 79
 9 1
 Elizabeth H. Simmons
 Particle Theorist and Physics Professor at UC San Diego · Upvoted by
 Mason Porter
 , Professor, Department of Mathematics, UCLA and
 Vijay Chidambaram
 , computer science researcher Author has 1.1K answers and 9.2M answer views
 · 8y
 Related
 Why do researchers publish in arXiv?
 We don't "publish" in arXiv. We release preprints on arXiv. And then we submit those manuscripts for publication in a peer-reviewed journal.
 Why do we put preprints on arXiv? Here are a few compelling reasons:
 * It gives free, instant, searchable and worldwide distribution of our work to a targeted audience that cares about our topic.
 * It immediately establishes when we released an idea into the world. Publication by journals can take months to years, depending on the discipline.
 * It allows us to get feedback from experts so we can improve the manuscript before submitting it for publication.
 * It e
 Continue Reading
 We don't "publish" in arXiv. We release preprints on arXiv. And then we submit those manuscripts for publication in a peer-reviewed journal.
 Why do we put preprints on arXiv? Here are a few compelling reasons:
 * It gives free, instant, searchable and worldwide distribution of our work to a targeted audience that cares about our topic.
 * It immediately establishes when we released an idea into the world. Publication by journals can take months to years, depending on the discipline.
 * It allows us to get feedback from experts so we can improve the manuscript before submitting it for publication.
 * It enters the manuscript into a tracking system that shows which papers reference which other papers. That is helpful both for literature searches and for roughly assessing a paper's intellectual impact.
 Upvote ·
 999 481
 99 15
 99 19
 Promoted by Masterworks
 Lila Fisher
 Art Education | @Masterworks
 · 1y
 Did Harry inherit more than William?
 Yes and no. When the Queen mother died back in 2002 she left more money to Harry, knowing that William would eventually become Prince and inherit the Duchy of Cornwall, a private estate valued at over $1 billion.
 Which is exactly what happened when Queen Elizabeth passed away in 2022. So overall, Willian inherited quite a bit more than Harry.
 But this $1 billion estate is nothing compared to the $10 billion that King Charles received tax free, thanks to a clever investment by the Queen into one of the world’s most exclusive asset classes … fine art.
 The Royal family has been investing a portion o
 Continue Reading
 Yes and no. When the Queen mother died back in 2002 she left more money to Harry, knowing that William would eventually become Prince and inherit the Duchy of Cornwall, a private estate valued at over $1 billion.
 Which is exactly what happened when Queen Elizabeth passed away in 2022. So overall, Willian inherited quite a bit more than Harry.
 But this $1 billion estate is nothing compared to the $10 billion that King Charles received tax free, thanks to a clever investment by the Queen into one of the world’s most exclusive asset classes … fine art.
 The Royal family has been investing a portion of their fortune into art for centuries …
 But the Royals are far from the only billionaires who have invested a significant chunk of their fortunes in this emerging asset class.
 Billionaires like David Geffen, Jeff Bezos, and Bill Gates are all avid art collectors.
 The recently-named “richest man in the world,” Berard Arnault, who is the CEO of luxury-goods conglomerate LVMH, has an extensive collection of art.
 Plus, there’s a long list of celebrities like Jay-Z, Oprah Winfrey, Steven Spielberg, Jack Nicholson, Madonna, and Ellen Degeneres who have invested millions into art.
 The only problem is, regular investors have always been locked out of this asset class because it usually takes millions of dollars to purchase just one iconic piece.
 Until now. Investing in art just got easier, thanks to a revolutionary new investment platform called Masterworks.
 Masterworks provides a simple, secure way for anyone to invest in shares of paintings from famous artists like Picasso, Banksy and Basquiat.
 Which is great, because contemporary art prices have outpaced the S&P 500 by 131% since 1995 … and real estate and gold by more than three to one over the same period.
 Even better, art acts as a great hedge during periods of high inflation and market volatility.
 Which is why Blackrock CEO, Larry Fink, who manages nearly $10 trillion in assets, recently called art “the new gold” because of its remarkable ability to “preserve wealth”
 These are just a few reasons why billionaires like Larry Fink, Jeff Bezos, and Oprah Winfrey are pouring hundreds of millions into this asset class.
 Thanks to Masterworks, you can join them. All you need to do is go to this link to sign up for a free membership.
 See important disclosures: Masterworks
 Upvote ·
 12.6K 12.6K
 1.5K 1.5K
 999 481
 Related questions
 Is publishing a paper on arXiv really a big deal?
 What are some pros and cons of publishing a paper on arXiv?
 Can you submit a paper to a journal if it has already been published on Arxiv?
 Does a publication on arXiv count as an actual publication? I'm preparing a CV for graduate school application and I have a article on arXiv with my supervisor. Should I put that into my CV?
 Which is better, SSRN or Arxiv.org?
 What are some good alternatives to Arxiv?
 How much time does it take to publish a paper in Arxiv?
 What is arxiv.org?
 Does uploading an unpublished paper to arXiv hurt the chances of publishing it in a reputable Journal?
 What percent of papers submitted to arXiv are "junk"?
 How long can a paper be in arXiv without being published in a journal before you suspect it may not have enough academic quality?
 Is it better to post rejected papers on Arxiv or not at all?
 What is the difference between arXiv and Arxiv?
 What are the consequences of submitting an article to ArXiv before trying to get it published?
 What is the meaning of having a paper accepted on Arxiv?
 Related questions
 Is publishing a paper on arXiv really a big deal?
 What are some pros and cons of publishing a paper on arXiv?
 Can you submit a paper to a journal if it has already been published on Arxiv?
 Does a publication on arXiv count as an actual publication? I'm preparing a CV for graduate school application and I have a article on arXiv with my supervisor. Should I put that into my CV?
 Which is better, SSRN or Arxiv.org?
 What are some good alternatives to Arxiv?
 Advertisement
 About · Careers · Privacy · Terms · Contact · Languages · Your Ad Choices · Press ·
 © Quora, Inc. 2024
Please enable Javascript and refresh the page to continue
Is Arxiv good for academia?
https://www.quora.com/Is-Arxiv-good-for-academia
<div><div>
      
      
        
          
        
      
      <main>
        <div>
              <article>
                
                  

  <a href="https://github.com/arXiv/arxiv-docs/blob/develop/source/about/index.md">
    
    
  </a>



<p>arXiv is a curated research-sharing platform open to anyone. As a pioneer in digital open access, arXiv.org now hosts more than two million scholarly articles in eight subject areas, curated by our strong community of volunteer moderators.</p>
<p>arXiv offers researchers a broad range of services: article submission, compilation, production, retrieval, search and discovery, web distribution for human readers, and API access for machines, together with content curation and preservation. Our emphasis on openness, collaboration, and scholarship provide the strong foundation on which arXiv thrives.</p>
<p>arXiv currently serves the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. arXiv was founded by Paul Ginsparg in 1991 and is now maintained and operated by <a href="https://www.tech.cornell.edu/">Cornell Tech</a>.</p>
<p>Operations are maintained by the <a href="https://info.arxiv.org/about/people/leadership_team.html">arXiv Leadership Team</a> and arXiv staff at Cornell University. Governance of arXiv is led by the Leadership Team with guidance from the arXiv <a href="https://info.arxiv.org/about/people/editorial_advisory_council.html">Editorial Advisory Council</a>, <a href="https://info.arxiv.org/about/people/institutions_advisory_council.html">Institutions Advisory Council</a>, and <a href="https://info.arxiv.org/about/people/science_advisory_council.html">Science Advisory Council</a>. arXiv is a community-supported resource funded by <a href="https://www.cornell.edu/">Cornell University</a>, the <a href="https://www.simonsfoundation.org">Simons Foundation</a>, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and <a href="https://info.arxiv.org/about/give.html">donors</a>.</p>
<p>Registered users may <a href="https://info.arxiv.org/help/submit/index.html">submit articles</a> to be announced by arXiv. There are no fees or costs for article submission. Submissions to arXiv are subject to a <a href="https://info.arxiv.org/help/moderation/index.html">moderation process</a> that classifies material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv - the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee. By hosting works and other materials on this site, arXiv, Cornell University, and their agents do not in any way convey implied approval of the assumptions, methods, results, or conclusions of the work.</p>
<h2>News</h2>
<ul>
<li><a href="https://arxiv.org/stats/main">Usage Stats</a></li>
<li><a href="https://blogs.cornell.edu/arxiv">arXiv Blog</a></li>
<li><a href="https://twitter.com/arxiv">arXiv Twitter</a></li>
</ul>
<h2>Funding &amp; Governance</h2>
<ul>
<li><a href="https://info.arxiv.org/about/funding.html">Funding Support</a></li>
<li><a href="https://info.arxiv.org/about/governance.html">Governance</a></li>
<li><a href="https://info.arxiv.org/about/reports-financials.html">Business Model</a></li>
<li><a href="https://info.arxiv.org/about/reports/index.html">Annual Reports and Financials</a></li>
</ul>
<h2>Membership &amp; Giving</h2>
<ul>
<li><a href="https://info.arxiv.org/about/membership.html">Institutional Membership Program</a></li>
<li><a href="https://info.arxiv.org/about/ourmembers.html">Our Members</a></li>
<li><a href="https://info.arxiv.org/about/supporters.html">Our Supporters</a></li>
<li><a href="https://info.arxiv.org/about/donate.html">Donate to arXiv</a></li>
</ul>
<h2>Who We Are</h2>
<ul>
<li><a href="https://info.arxiv.org/about/people/staff.html">arXiv Staff</a></li>
<li><a href="https://info.arxiv.org/about/people/leadership_team.html">Leadership Team</a></li>
<li><a href="https://info.arxiv.org/about/arxiv_advisory_board.md">arXiv Advisory Board</a></li>
<li><a href="https://info.arxiv.org/about/people/science_advisory_council.html">Science Advisory Council</a></li>
<li><a href="https://info.arxiv.org/about/people/editorial_advisory_council.html">Editorial Advisory Council</a></li>
<li>Section Editorial Committees<ul>
<li><a href="https://info.arxiv.org/help/physics/index.html#AdvisoryCommittee">Physics</a></li>
<li><a href="https://info.arxiv.org/help/math/index.html#AdvisoryCommittee">Mathematics</a></li>
<li><a href="https://info.arxiv.org/help/cs/index.html">Computer science (CoRR)</a></li>
<li><a href="https://info.arxiv.org/help/q-bio/index.html#AdvisoryCommittee">Quantitative biology</a></li>
<li><a href="https://info.arxiv.org/help/q-fin/index.html#AdvisoryCommittee">Quantitative finance</a></li>
<li><a href="https://info.arxiv.org/help/statistics/index.html#AdvisoryCommittee">Statistics</a></li>
<li><a href="https://info.arxiv.org/help/eess/index.html#AdvisoryCommittee">Electrical engineering and systems science</a></li>
<li><a href="https://info.arxiv.org/help/econ/index.html#AdvisoryCommittee">Economics</a></li>
</ul>
</li>
<li><a href="https://info.arxiv.org/about/people/institutions_advisory_council.html">Institutions Advisory Council</a></li>
<li><a href="https://arxiv.org/moderators">Volunteer Moderators</a></li>
<li><a href="https://info.arxiv.org/about/people/developers.html">Volunteer Developers</a></li>
<li><a href="https://info.arxiv.org/about/user-testing.html">Join arXiv's User Testing Group</a></li>
<li><a href="https://info.arxiv.org/brand/index.html">Brand Guide and Logo Usage</a></li>
</ul>





                
              </article>
            </div>
        
      </main>
      






    </div></div>||||I||||  Skip to content

  We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors.

      arXiv info
      About arXiv
            Initializing search
    GitHub
          arXiv info
              GitHub
            * Home
            * About
              About
                + About arXiv About arXiv
                  Table of contents
                    o About arXiv
                        - News
                        - Funding & Governance
                        - Membership & Giving
                        - Who We Are
                + Who we are
                + Governance
                  Governance
                    o arXiv Governance Model
                    o Bylaws
                    o arXiv Advisory Board
                    o Science Advisory Council
                    o Editorial Advisory Council
                    o Institutions Advisory Council
                + Membership
                + Funding
                + Reports
                + Accessibility
                + arXiv Blog
            * Help
              Help
                + arXiv Help Contents
                + Submit an Article
                  Submit an Article
                    o None
                    o Submission Overview
                    o Submit TeX/LaTeX
                    o Submit a PDF
                    o FAQ
                      FAQ
                        - Frequently Asked Questions
                        - LaTeX2e class for Astronomy & Astrophysics
                        - AMS LaTeX packages and AMS Journal styles
                        - Downloaded .gz Files that are not Gzipped
                        - Why do my citations appear in long form
                        - Examples of TeX Double Subscripts
                        - Papersize/Layout Problems: Margins are Different and/or Text is Truncated
                        - Using Feynmf on arXiv
                        - Proprietary fonts and/or their free equivalents
                        - How to Prepare Pages for Landscape Printout
                        - Common Mistakes that cause Automated Processing to Fail
                        - Why do my user defined symbols display incorrectly
                        - Why are some pages in the generated PDF file rotated?
                        - Why are there Problems with my PostScript File?
                        - Why is my PostScript File so Large?
                        - Using PSTricks on arXiv
                        - References to and in arXiv Documents
                        - Status of REVTeX 4 and REVTeX 4.1 support
                        - Nota Bene (updated 30 Jun '97)
                        - Frequently Asked Questions on Public Statistics
                        - Notes about arXiv's TeXLive Version
                        - Why doesn't my processed TeX submission look the way I expected it?
                        - Textures
                        - Why Does My Paper Give the Wrong Date?
                        - Why doesn't my paper have the arXiv id stamped on the side of the page?
                    o Status Information
                    o Ancillary Files (data, code, images)
                    o Availability of submissions
                    o Category cross listing
                    o Endorsement
                    o Adding Journal Reference and DOI
                    o Licenses
                    o Text Overlap
                    o Metadata for Required and Optional Fields
                    o Submit a new version of a work
                    o Oversized Submissions
                    o Submit a Paper List for Conference Proceedings
                    o Creating tar and zip Files for Upload
                    o What is TeX
                    o Proxy / Third Party Submission
                    o Translations
                    o Version Availability
                    o Why Submit TeX?
                    o Withdraw / Retract a Submission
                + Find and Read Articles
                  Find and Read Articles
                    o Finding Articles
                    o Search for articles
                    o Viewing submissions
                    o Email subscriptions for new papers
                    o arXiv Identifier
                    o Configuring your browser
                    o RSS Feeds
                    o Licenses
                    o Permissions and Reuse
                    o What is MathJax?
                + Accounts
                  Accounts
                    o Index
                    o Directions for Author Registration
                    o Author Identifiers
                    o ORCID identifiers
                    o Authority Records
                    o Why isn't a Person "Registered as an Author?"
                + Bulk Access to Metadata and Full Text
                  Bulk Access to Metadata and Full Text
                    o Index
                    o arXiv Identifier for Interacting Services
                    o Full Text via S3
                    o API for Metadata
                      API for Metadata
                        - arXiv API Access
                        - arXiv API Basics
                        - Terms of Use for arXiv APIs
                        - arXiv API User's Manual
                    o RSS Feeds
                    o Institutional Repository Interoperability
                    o Automated DOI and journal reference updates from publishers
            * Brand
              Brand
                + Our Brand
                + Logo Use
                + Brand pillars
                + Colors
                + Fonts
                + Images
                + Logos
                + Quotes
                + Swag
                + Tagline
                + Typography
                + Voice
            * Labs
              Labs
                + arXivLabs: An invitation to collaborate
                + arXivLabs Criteria
                + Project proposal
                + arXivLabs: Showcase
            * Policies
              Policies
                + arXiv Policies
                + Code of conduct
                + Privacy policy
                + Identity, affiliation, and registration
                + Submission terms and agreement
                + Paper ownership
                + General submission policies
                + Submission schedule and cutoff time
                + Content Moderation
                + License and copyright
                + Withdrawals
                + Cross listing
                + Translations
                + Non-English submissions
                + Text Overlap
                + Requiring TeX when possible
                + Third party submission
            * Categories
            * arXiv Usage Stats
          Table of contents
            * About arXiv
                + News
                + Funding & Governance
                + Membership & Giving
                + Who We Are

      About arXiv

      arXiv is a curated research-sharing platform open to anyone. As a pioneer in digital open access, arXiv.org now hosts more than two million scholarly articles in eight subject areas, curated by our strong community of volunteer moderators.

      arXiv offers researchers a broad range of services: article submission, compilation, production, retrieval, search and discovery, web distribution for human readers, and API access for machines, together with content curation and preservation. Our emphasis on openness, collaboration, and scholarship provide the strong foundation on which arXiv thrives.

      arXiv currently serves the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. arXiv was founded by Paul Ginsparg in 1991 and is now maintained and operated by Cornell Tech.

      Operations are maintained by the arXiv Leadership Team and arXiv staff at Cornell University. Governance of arXiv is led by the Leadership Team with guidance from the arXiv Editorial Advisory Council, Institutions Advisory Council, and Science Advisory Council. arXiv is a community-supported resource funded by Cornell University, the Simons Foundation, member institutions, and donors.

      Registered users may submit articles to be announced by arXiv. There are no fees or costs for article submission. Submissions to arXiv are subject to a moderation process that classifies material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv - the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee. By hosting works and other materials on this site, arXiv, Cornell University, and their agents do not in any way convey implied approval of the assumptions, methods, results, or conclusions of the work.

      News

        * Usage Stats
        * arXiv Blog
        * arXiv Twitter

      Funding & Governance

        * Funding Support
        * Governance
        * Business Model
        * Annual Reports and Financials

      Membership & Giving

        * Institutional Membership Program
        * Our Members
        * Our Supporters
        * Donate to arXiv

      Who We Are

        * arXiv Staff
        * Leadership Team
        * arXiv Advisory Board
        * Science Advisory Council
        * Editorial Advisory Council
        * Section Editorial Committees
            + Physics
            + Mathematics
            + Computer science (CoRR)
            + Quantitative biology
            + Quantitative finance
            + Statistics
            + Electrical engineering and systems science
            + Economics
        * Institutions Advisory Council
        * Volunteer Moderators
        * Volunteer Developers
        * Join arXiv's User Testing Group
        * Brand Guide and Logo Usage
        * About
        * Help
        * Copyright
        * Privacy Policy
        * Click here to contact arXiv Contact
        * Click here to subscribe Subscribe
        * Click here to report an issue with arXiv's documentation in github Report a documentation issue
        * Web Accessibility Assistance

        * arXiv Operational Status
          Get status notifications via email or slack

arXiv is a curated research-sharing platform open to anyone. As a pioneer in digital open access, arXiv.org now hosts more than two million scholarly articles in eight subject areas, curated by our strong community of volunteer moderators.
arXiv offers researchers a broad range of services: article submission, compilation, production, retrieval, search and discovery, web distribution for human readers, and API access for machines, together with content curation and preservation. Our emphasis on openness, collaboration, and scholarship provide the strong foundation on which arXiv thrives.
arXiv currently serves the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. arXiv was founded by Paul Ginsparg in 1991 and is now maintained and operated by Cornell Tech.
Operations are maintained by the arXiv Leadership Team and arXiv staff at Cornell University. Governance of arXiv is led by the Leadership Team with guidance from the arXiv Editorial Advisory Council, Institutions Advisory Council, and Science Advisory Council. arXiv is a community-supported resource funded by Cornell University, the Simons Foundation, member institutions, and donors.
Registered users may submit articles to be announced by arXiv. There are no fees or costs for article submission. Submissions to arXiv are subject to a moderation process that classifies material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv - the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee. By hosting works and other materials on this site, arXiv, Cornell University, and their agents do not in any way convey implied approval of the assumptions, methods, results, or conclusions of the work.
News
 Usage Stats 
 arXiv Blog 
 arXiv Twitter 
Funding &amp; Governance
 Funding Support 
 Governance 
 Business Model 
 Annual Reports and Financials 
Membership &amp; Giving
 Institutional Membership Program 
 Our Members 
 Our Supporters 
 Donate to arXiv 
Who We Are
 arXiv Staff 
 Leadership Team 
 arXiv Advisory Board 
 Science Advisory Council 
 Editorial Advisory Council 
Section Editorial Committees
 Physics 
 Mathematics 
 Computer science (CoRR) 
 Quantitative biology 
 Quantitative finance 
 Statistics 
 Electrical engineering and systems science 
 Economics 
 Institutions Advisory Council 
 Volunteer Moderators 
 Volunteer Developers 
 Join arXiv's User Testing Group 
 Brand Guide and Logo Usage 
                
              
            
        
      
      
     ||||I||||  Skip to content
  We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors.
      arXiv info
      About arXiv
            Initializing search
    GitHub
          arXiv info
              GitHub
            * Home
            * About
              About
                + About arXiv About arXiv
                  Table of contents
                    o About arXiv
                        - News
                        - Funding & Governance
                        - Membership & Giving
                        - Who We Are
                + Who we are
                + Governance
                  Governance
                    o arXiv Governance Model
                    o Bylaws
                    o arXiv Advisory Board
                    o Science Advisory Council
                    o Editorial Advisory Council
                    o Institutions Advisory Council
                + Membership
                + Funding
                + Reports
                + Accessibility
                + arXiv Blog
            * Help
              Help
                + arXiv Help Contents
                + Submit an Article
                  Submit an Article
                    o None
                    o Submission Overview
                    o Submit TeX/LaTeX
                    o Submit a PDF
                    o FAQ
                      FAQ
                        - Frequently Asked Questions
                        - LaTeX2e class for Astronomy & Astrophysics
                        - AMS LaTeX packages and AMS Journal styles
                        - Downloaded .gz Files that are not Gzipped
                        - Why do my citations appear in long form
                        - Examples of TeX Double Subscripts
                        - Papersize/Layout Problems: Margins are Different and/or Text is Truncated
                        - Using Feynmf on arXiv
                        - Proprietary fonts and/or their free equivalents
                        - How to Prepare Pages for Landscape Printout
                        - Common Mistakes that cause Automated Processing to Fail
                        - Why do my user defined symbols display incorrectly
                        - Why are some pages in the generated PDF file rotated?
                        - Why are there Problems with my PostScript File?
                        - Why is my PostScript File so Large?
                        - Using PSTricks on arXiv
                        - References to and in arXiv Documents
                        - Status of REVTeX 4 and REVTeX 4.1 support
                        - Nota Bene (updated 30 Jun '97)
                        - Frequently Asked Questions on Public Statistics
                        - Notes about arXiv's TeXLive Version
                        - Why doesn't my processed TeX submission look the way I expected it?
                        - Textures
                        - Why Does My Paper Give the Wrong Date?
                        - Why doesn't my paper have the arXiv id stamped on the side of the page?
                    o Status Information
                    o Ancillary Files (data, code, images)
                    o Availability of submissions
                    o Category cross listing
                    o Endorsement
                    o Adding Journal Reference and DOI
                    o Licenses
                    o Text Overlap
                    o Metadata for Required and Optional Fields
                    o Submit a new version of a work
                    o Oversized Submissions
                    o Submit a Paper List for Conference Proceedings
                    o Creating tar and zip Files for Upload
                    o What is TeX
                    o Proxy / Third Party Submission
                    o Translations
                    o Version Availability
                    o Why Submit TeX?
                    o Withdraw / Retract a Submission
                + Find and Read Articles
                  Find and Read Articles
                    o Finding Articles
                    o Search for articles
                    o Viewing submissions
                    o Email subscriptions for new papers
                    o arXiv Identifier
                    o Configuring your browser
                    o RSS Feeds
                    o Licenses
                    o Permissions and Reuse
                    o What is MathJax?
                + Accounts
                  Accounts
                    o Index
                    o Directions for Author Registration
                    o Author Identifiers
                    o ORCID identifiers
                    o Authority Records
                    o Why isn't a Person "Registered as an Author?"
                + Bulk Access to Metadata and Full Text
                  Bulk Access to Metadata and Full Text
                    o Index
                    o arXiv Identifier for Interacting Services
                    o Full Text via S3
                    o API for Metadata
                      API for Metadata
                        - arXiv API Access
                        - arXiv API Basics
                        - Terms of Use for arXiv APIs
                        - arXiv API User's Manual
                    o RSS Feeds
                    o Institutional Repository Interoperability
                    o Automated DOI and journal reference updates from publishers
            * Brand
              Brand
                + Our Brand
                + Logo Use
                + Brand pillars
                + Colors
                + Fonts
                + Images
                + Logos
                + Quotes
                + Swag
                + Tagline
                + Typography
                + Voice
            * Labs
              Labs
                + arXivLabs: An invitation to collaborate
                + arXivLabs Criteria
                + Project proposal
                + arXivLabs: Showcase
            * Policies
              Policies
                + arXiv Policies
                + Code of conduct
                + Privacy policy
                + Identity, affiliation, and registration
                + Submission terms and agreement
                + Paper ownership
                + General submission policies
                + Submission schedule and cutoff time
                + Content Moderation
                + License and copyright
                + Withdrawals
                + Cross listing
                + Translations
                + Non-English submissions
                + Text Overlap
                + Requiring TeX when possible
                + Third party submission
            * Categories
            * arXiv Usage Stats
          Table of contents
            * About arXiv
                + News
                + Funding & Governance
                + Membership & Giving
                + Who We Are
      About arXiv
      arXiv is a curated research-sharing platform open to anyone. As a pioneer in digital open access, arXiv.org now hosts more than two million scholarly articles in eight subject areas, curated by our strong community of volunteer moderators.
      arXiv offers researchers a broad range of services: article submission, compilation, production, retrieval, search and discovery, web distribution for human readers, and API access for machines, together with content curation and preservation. Our emphasis on openness, collaboration, and scholarship provide the strong foundation on which arXiv thrives.
      arXiv currently serves the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. arXiv was founded by Paul Ginsparg in 1991 and is now maintained and operated by Cornell Tech.
      Operations are maintained by the arXiv Leadership Team and arXiv staff at Cornell University. Governance of arXiv is led by the Leadership Team with guidance from the arXiv Editorial Advisory Council, Institutions Advisory Council, and Science Advisory Council. arXiv is a community-supported resource funded by Cornell University, the Simons Foundation, member institutions, and donors.
      Registered users may submit articles to be announced by arXiv. There are no fees or costs for article submission. Submissions to arXiv are subject to a moderation process that classifies material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv - the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee. By hosting works and other materials on this site, arXiv, Cornell University, and their agents do not in any way convey implied approval of the assumptions, methods, results, or conclusions of the work.
      News
        * Usage Stats
        * arXiv Blog
        * arXiv Twitter
      Funding & Governance
        * Funding Support
        * Governance
        * Business Model
        * Annual Reports and Financials
      Membership & Giving
        * Institutional Membership Program
        * Our Members
        * Our Supporters
        * Donate to arXiv
      Who We Are
        * arXiv Staff
        * Leadership Team
        * arXiv Advisory Board
        * Science Advisory Council
        * Editorial Advisory Council
        * Section Editorial Committees
            + Physics
            + Mathematics
            + Computer science (CoRR)
            + Quantitative biology
            + Quantitative finance
            + Statistics
            + Electrical engineering and systems science
            + Economics
        * Institutions Advisory Council
        * Volunteer Moderators
        * Volunteer Developers
        * Join arXiv's User Testing Group
        * Brand Guide and Logo Usage
        * About
        * Help
        * Copyright
        * Privacy Policy
        * Click here to contact arXiv Contact
        * Click here to subscribe Subscribe
        * Click here to report an issue with arXiv's documentation in github Report a documentation issue
        * Web Accessibility Assistance
        * arXiv Operational Status
          Get status notifications via email or slack
About arXiv
https://arxiv.org/about
https://www.reddit.com/r/MachineLearning/
Reddit - Dive into anything
2023-12-03
None
<div><div>
<p>
 (Reposting since the previous post was removed, I think because non-Arxiv posts are only allowed on weekends and now it's a weekend)
 </p><p>
 Materials discovery is critical but tough. New materials enable big innovations like batteries or LEDs. But there are ~infinitely many combinations to try. Testing for them experimentally is slow and expensive.
 </p><p>
 So scientists and engineers want to simulate and screen materials on computers first. This can check way more candidates before real-world experiments. However, models historically struggled at accurately predicting if materials are stable.
 </p><p>
 Researchers at DeepMind made a system called GNoME that uses graph neural networks and active learning to push past these limits.
 </p><p>
 GNoME models materials' crystal structures as graphs and predicts formation energies. It actively generates and filters candidates, evaluating the most promising with simulations. This expands its knowledge and improves predictions over multiple cycles.
 </p><p>
 The authors introduced new ways to generate derivative structures that respect symmetries, further diversifying discoveries.
 </p><p>
 The results:
 </p><ol>
<li>
<p>
 GNoME found 2.2 million new stable materials - equivalent to 800 years of normal discovery.
 </p>
</li><li>
<p>
 Of those, 380k were the most stable and candidates for validation.
 </p>
</li><li>
<p>
 736 were validated in external labs. These include a totally new diamond-like optical material and another that may be a superconductor.
 </p>
</li>
</ol><p>
 Overall this demonstrates how scaling up deep learning can massively speed up materials innovation. As data and models improve together, it'll accelerate solutions to big problems needing new engineered materials.
 </p><p>
<strong>TLDR: DeepMind made an AI system that uses graph neural networks to discover possible new materials. It found 2.2 million candidates, and over 300k are most stable. Over 700 have already been synthesized.</strong>
</p><p>
<a href="https://aimodels.substack.com/p/google-deepmind-announces-its-found">Full summary available here</a>. Paper is <a href="https://www.nature.com/articles/s41586-023-06735-9">here</a>.
 </p>
</div></div>||||I|||| Open menu Open navigation Go to Reddit Home
 r/MachineLearning A chip A close button
 Get app Get the Reddit app Log In Log in to Reddit
 Expand user menu Open settings menu
 * Log In / Sign Up
 * Advertise on Reddit
 * Shop Collectible Avatars

Get the Reddit app

 Scan this QR code to download the app now
 Or check it out in the app stores
 r/MachineLearning
 members
 online
 Create a post
 Sort By:
 Hot
 Open sort options
 * Hot
 * New
 * Top
 * Rising
 Change post view
 * Card
 * Classic
 Feed About
 Hot
 Open sort options
 * Hot
 * New
 * Top
 * Rising
 Change post view
 * Card
 * Classic
 u/AutoModerator
 •
 [D] Simple Questions Thread Discussion
 u/Successful-Western27
 • [R] Google DeepMind: 2.2 million new materials discovered using GNN (380k most stable, 736 already validated in labs) Research

 (Reposting since the previous post was removed, I think because non-Arxiv posts are only allowed on weekends and now it's a weekend)

 Materials discovery is critical but tough. New materials enable big innovations like batteries or LEDs. But there are ~infinitely many combinations to try. Testing for them experimentally is slow and expensive.

 So scientists and engineers want to simulate and screen materials on computers first. This can check way more candidates before real-world experiments. However, models historically struggled at accurately predicting if materials are stable.

 Researchers at DeepMind made a system called GNoME that uses graph neural networks and active learning to push past these limits.

 GNoME models materials' crystal structures as graphs and predicts formation energies. It actively generates and filters candidates, evaluating the most promising with simulations. This expands its knowledge and improves predictions over multiple cycles.

 The authors introduced new ways to generate derivative structures that respect symmetries, further diversifying discoveries.

 The results:

 1. GNoME found 2.2 million new stable materials - equivalent to 800 years of normal discovery.

 2. Of those, 380k were the most stable and candidates for validation.

 3. 736 were validated in external labs. These include a totally new diamond-like optical material and another that may be a superconductor.

 Overall this demonstrates how scaling up deep learning can massively speed up materials innovation. As data and models improve together, it'll accelerate solutions to big problems needing new engineered materials.

 TLDR: DeepMind made an AI system that uses graph neural networks to discover possible new materials. It found 2.2 million candidates, and over 300k are most stable. Over 700 have already been synthesized.

 Full summary available here. Paper is here.

 View full post
 u/raybanmeta
 • Promoted
 Introducing the Ray-Ban Meta smart glasses collection, available now.
 meta.com
 Learn More
 u/tanweer_m
 • [R] Do you cite Arxiv pre-prints in your research? Research

 Hello fellow ML researchers!

 A bit of context:

 I am writing a survey paper (surveys + proposed research directions). It has been growing to include new developments. Nothing novel. I am away from academia and working at a startup as a staff scientist.

 Reality is: I won't at all be able to deal with the review process due to my schedule. At the same time, I really want the paper to be out and be read/used/cited. I will be uploading it in arxiv.

 My question is: in ML research, how often do you consider arxiv as a good reference? Due to the super fast nature of ML research, my guess would be it is a little different than other fields.

 (My PhD is in applied maths where we rarely cite arxiv pre-prints).

 View full post
 Top 1% Rank by size

 Twitter

 @slashML

 Rules

 * 1

 Be nice: no offensive behavior, insults or attacks

 We encourage a diverse community in which members feel safe and have a voice. Refer to our Community Standards for a full description.

 * 2

 Make your post clear and comprehensive

 Posts that lack insight or effort will be removed. (ex: questions which are easily googled)

 * 3

 Posts without appropriate tag in title will be removed

 Allowed Tags: "[Discussion]", "[D]", "[News]", "[N]", "[Research]", "[R]", "[Project]", "[P]"

 * 4

 Beginner or career related questions go elsewhere

 Post beginner questions in the bi-weekly "Simple Questions Thread", r/LearnMachineLearning , r/MLQuestions http://stackoverflow.com/ and career questions in r/cscareerquestions/

 * 5

 Non-arxiv link posts only allowed on weekends (must be demos)*

 Link posts must include context (ie: a comment in the reddit thread linking to the paper/code), and either be demos of novel research, include code, or include a description of how it was done. Note that blog links will still not be allowed, nor will beginner projects.

 During weekdays, if you'd like to share a link, place it in a self-post and provide some context.

 *Arxiv and OpenReview links are exceptions to this rule.

 * 6

 Beginner's tutorials and projects go elsewhere

 Beginner's guides, tutorials, and projects should be posted to r/LearnMachineLearning

 * 7

 Quality Contribution

 This report reason is just a convenient way for users to notify mods when they find a quality contribution. See https://www.reddit.com/r/MachineLearning/comments/hvzyz7/d_quality_contributions_roundup_722/ for more details.

 * 8

 Limit self-promotion

 Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

 Promoted
 * Home
 * Popular
 * TOPICS
 * Gaming
 * Valheim
 * Genshin Impact
 * Minecraft
 * Pokimane
 * Halo Infinite
 * Call of Duty: Warzone
 * Path of Exile
 * Hollow Knight: Silksong
 * Escape from Tarkov
 * Watch Dogs: Legion
 * Sports
 * NFL
 * NBA
 * Megan Anderson
 * Atlanta Hawks
 * Los Angeles Lakers
 * Boston Celtics
 * Arsenal F.C.
 * Philadelphia 76ers
 * Premier League
 * UFC
 * Business
 * GameStop
 * Moderna
 * Pfizer
 * Johnson & Johnson
 * AstraZeneca
 * Walgreens
 * Best Buy
 * Novavax
 * SpaceX
 * Tesla
 * Crypto
 * Cardano
 * Dogecoin
 * Algorand
 * Bitcoin
 * Litecoin
 * Basic Attention Token
 * Bitcoin Cash
 * Television
 * The Real Housewives of Atlanta
 * The Bachelor
 * Sister Wives
 * 90 Day Fiance
 * Wife Swap
 * The Amazing Race Australia
 * Married at First Sight
 * The Real Housewives of Dallas
 * My 600-lb Life
 * Last Week Tonight with John Oliver
 * Celebrity
 * Kim Kardashian
 * Doja Cat
 * Iggy Azalea
 * Anya Taylor-Joy
 * Jamie Lee Curtis
 * Natalie Portman
 * Henry Cavill
 * Millie Bobby Brown
 * Tom Hiddleston
 * Keanu Reeves
 * RESOURCES
 * About Reddit
 * Advertise
 * Help
 * Blog
 * Careers
 * Press
 Reddit, Inc. © 2023. All rights reserved.
(Reposting since the previous post was removed, I think because non-Arxiv posts are only allowed on weekends and now it's a weekend)
  
 Materials discovery is critical but tough. New materials enable big innovations like batteries or LEDs. But there are ~infinitely many combinations to try. Testing for them experimentally is slow and expensive.
  
 So scientists and engineers want to simulate and screen materials on computers first. This can check way more candidates before real-world experiments. However, models historically struggled at accurately predicting if materials are stable.
  
 Researchers at DeepMind made a system called GNoME that uses graph neural networks and active learning to push past these limits.
  
 GNoME models materials' crystal structures as graphs and predicts formation energies. It actively generates and filters candidates, evaluating the most promising with simulations. This expands its knowledge and improves predictions over multiple cycles.
  
 The authors introduced new ways to generate derivative structures that respect symmetries, further diversifying discoveries.
  
 The results:
  
 GNoME found 2.2 million new stable materials - equivalent to 800 years of normal discovery.
 
 
 Of those, 380k were the most stable and candidates for validation.
 
 
 736 were validated in external labs. These include a totally new diamond-like optical material and another that may be a superconductor.
 
 
 Overall this demonstrates how scaling up deep learning can massively speed up materials innovation. As data and models improve together, it'll accelerate solutions to big problems needing new engineered materials.
  
TLDR: DeepMind made an AI system that uses graph neural networks to discover possible new materials. It found 2.2 million candidates, and over 300k are most stable. Over 700 have already been synthesized.
 
Full summary available here. Paper is here.
Reddit - Dive into anything
https://www.reddit.com/r/MachineLearning/
https://towardsdatascience.com/the-best-machine-learning-blogs-resources-to-follow-f8177ab4f08f?gi=423b6fcb0a4a
The Best Machine Learning Blogs & Resources To Follow
2021-06-25T19:39:47Z
Kurtis Pykes
<html><body><div><article><div><div><div><h2>Resources That Are Constantly Being Updated</h2><div><div><div><a href="https://kurtispykes.medium.com/?source=post_page-----f8177ab4f08f--------------------------------"><div><p></p></div></a><a href="https://towardsdatascience.com/?source=post_page-----f8177ab4f08f--------------------------------"><div><p></p></div></a></div><div><div><p><span></span></p><div><div><p><a href="https://kurtispykes.medium.com/?source=post_page-----f8177ab4f08f--------------------------------">Kurtis Pykes</a></p></div><p><span><a href="https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fsubscribe%2Fuser%2F5ba760786877&amp;operation=register&amp;redirect=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-machine-learning-blogs-resources-to-follow-f8177ab4f08f&amp;user=Kurtis+Pykes&amp;userId=5ba760786877&amp;source=post_page-5ba760786877----f8177ab4f08f---------------------post_header-----------">Follow</a></span></p></div><p></p></div><div><p><span></span></p><div><div><p><span>Published in</span></p><div><a href="https://towardsdatascience.com/?source=post_page-----f8177ab4f08f--------------------------------"><p>Towards Data Science</p></a></div></div><p><span></span></p><div><p><span>4 min read</span></p><p><span>Jun 25, 2021</span></p></div><p></p></div><p></p></div></div></div><div><div><div><p><span>--</span></p></div></div><div></div></div></div></div><figure><figcaption>Photo by <a href="https://unsplash.com/@eddrobertson?utm_source=medium&amp;utm_medium=referral">Ed Robertson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Switching to Freelancing has given me much more free time. After hanging out with my family, I like to invest a lot of that time into educating myself and building new skills that would allow me to yield a greater return for the time I spend working on projects for clients — all in hopes of gaining…</p></div></div></article></div></body></html>
Resources That Are Constantly Being Updated                          Kurtis Pykes     Follow               Published in    Towards Data Science          4 min read   Jun 25, 2021               --          Photo by Ed Robertson on Unsplash   Switching to Freelancing has given me much more free time. After hanging out with my family, I like to invest a lot of that time into educating myself and building new skills that would allow me to yield a greater return for the time I spend working on projects for clients — all in hopes of gaining…
The Best Machine Learning Blogs & Resources To Follow
https://towardsdatascience.com/the-best-machine-learning-blogs-resources-to-follow-f8177ab4f08f
https://en.wikipedia.org/wiki/ArXiv
arXiv - Wikipedia
2002-02-11
None
<div><div>
	<p>From Wikipedia, the free encyclopedia</p>
	</div><div>
<table><caption>arXiv</caption><tbody><tr><td><a href="/wiki/File:ArXiv_logo_2022.svg"></a></td></tr><tr><td><a href="/wiki/File:ArXiv-org_screenshot_20140706.png"></a></td></tr><tr><th><p>Type of site</p></th><td>Science</td></tr><tr><th>Available in</th><td>English</td></tr><tr><th>Owner</th><td><a href="/wiki/Cornell_University">Cornell University</a></td></tr><tr><th>Created by</th><td><a href="/wiki/Paul_Ginsparg">Paul Ginsparg</a></td></tr><tr><th>URL</th><td><a href="https://arxiv.org/">arxiv<wbr></wbr>.org</a></td></tr><tr><th>Commercial</th><td>No</td></tr><tr><th>Launched</th><td>August 14, 1991; 31 years ago</td></tr><tr><th>Current status</th><td>Online</td></tr><tr><th><a href="/wiki/ISSN_(identifier)">ISSN</a></th><td><a href="https://www.worldcat.org/search?fq=x0:jrnl&amp;q=n2:2331-8422">2331-8422</a></td></tr><tr><th><a href="/wiki/OCLC_(identifier)">OCLC</a> number</th><td><a href="https://www.worldcat.org/oclc/228652809">228652809 </a></td></tr></tbody></table>
<p><b>arXiv</b> (pronounced "<a href="/wiki/Archive">archive</a>"—the X represents the <a href="/wiki/Chi_(letter)">Greek letter chi</a> ⟨χ⟩)<sup><a href="#cite_note-Steele-1">[1]</a></sup> is an <a href="/wiki/Open-access_repository">open-access repository</a> of electronic <a href="/wiki/Preprint">preprints</a> and <a href="/wiki/Postprints">postprints</a> (known as <a href="/wiki/E-prints">e-prints</a>) approved for posting after moderation, but not <a href="/wiki/Scholarly_peer_review">peer review</a>. It consists of <a href="/wiki/Scientific_papers">scientific papers</a> in the fields of <a href="/wiki/Mathematics">mathematics</a>, <a href="/wiki/Physics">physics</a>, <a href="/wiki/Astronomy">astronomy</a>, <a href="/wiki/Electrical_engineering">electrical engineering</a>, <a href="/wiki/Computer_science">computer science</a>, <a href="/wiki/Quantitative_biology">quantitative biology</a>, <a href="/wiki/Statistics">statistics</a>, <a href="/wiki/Mathematical_finance">mathematical finance</a> and <a href="/wiki/Economics">economics</a>, which can be accessed online. In many fields of mathematics and physics, almost all scientific papers are <a href="/wiki/Self-archiving">self-archived</a> on the arXiv repository before publication in a peer-reviewed journal. Some publishers also grant permission for authors to archive the peer-reviewed <a href="/wiki/Postprint">postprint</a>. Begun on August 14, 1991, arXiv.org passed the half-million-article milestone on October 3, 2008,<sup><a href="#cite_note-2">[2]</a></sup><sup><a href="#cite_note-3">[3]</a></sup> and had hit a million by the end of 2014.<sup><a href="#cite_note-4">[4]</a></sup><sup><a href="#cite_note-dddmag2015-5">[5]</a></sup> 
As of April 2021, the submission rate is about 16,000 articles per month.<sup><a href="#cite_note-6">[6]</a></sup>
</p>
<h2>History[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=1">edit</a>]</h2>
<div><p><a href="/wiki/File:ArXiv_1994.png"></a></p><div><p>A screenshot of the arXiv taken in 1994,<sup><a href="#cite_note-7">[7]</a></sup> using the browser <a href="/wiki/Mosaic_(web_browser)">NCSA Mosaic</a>. At the time, <a href="/wiki/Form_(HTML)">HTML forms</a> were a new technology.</p></div></div>
<div><p><a href="/wiki/File:ArXiv%27s_yearly_submission_rate_plot.jpg"></a></p><div><p>ArXiv's yearly submission rate growth over 30 years since its beginning with topics labelled by the standard abbreviations used on arxiv.org<sup><a href="#cite_note-8">[8]</a></sup></p></div></div>
<p>arXiv was made possible by the compact <a href="/wiki/TeX">TeX</a> file format, which allowed scientific papers to be easily transmitted over the <a href="/wiki/Internet">Internet</a> and rendered <a href="/wiki/Client-side">client-side</a>.<sup><a href="#cite_note-9">[9]</a></sup> Around 1990, <a href="/wiki/Joanne_Cohn">Joanne Cohn</a> began emailing <a href="/wiki/Physics">physics</a> preprints to colleagues as TeX files, but the number of papers being sent soon filled mailboxes to capacity.<sup><a href="#cite_note-10">[10]</a></sup> <a href="/wiki/Paul_Ginsparg">Paul Ginsparg</a> recognized the need for central storage, and in August 1991 he created a central <a href="/wiki/Disciplinary_repository">repository</a> mailbox stored at the <a href="/wiki/Los_Alamos_National_Laboratory">Los Alamos National Laboratory</a> (LANL) which could be accessed from any computer.<sup><a href="#cite_note-11">[11]</a></sup> Additional modes of access were soon added: <a href="/wiki/FTP">FTP</a> in 1991, <a href="/wiki/Gopher_(protocol)">Gopher</a> in 1992, and the <a href="/wiki/World_Wide_Web">World Wide Web</a> in 1993.<sup><a href="#cite_note-dddmag2015-5">[5]</a></sup><sup><a href="#cite_note-12">[12]</a></sup> The term <a href="/wiki/E-print">e-print</a> was quickly adopted to describe the articles.
</p><p>It began as a physics archive, called the <a href="/wiki/LANL">LANL</a> preprint archive, but soon expanded to include astronomy, mathematics, computer science, quantitative biology and, most recently, statistics. Its original <a href="/wiki/Domain_name">domain name</a> was xxx.lanl.gov. Due to LANL's lack of interest in the rapidly expanding technology, in 2001 Ginsparg changed institutions to <a href="/wiki/Cornell_University">Cornell University</a> and changed the name of the repository to arXiv.org.<sup><a href="#cite_note-13">[13]</a></sup> It is now hosted principally by Cornell, with five <a href="/wiki/Mirror_website">mirrors</a> around the world.<sup><a href="#cite_note-14">[14]</a></sup>
</p><p>ArXiv was an early adopter and promoter of <a href="/wiki/Preprints">preprints</a>.<sup><a href="#cite_note-SPARC_2021-15">[15]</a></sup> Its success in sharing preprints was one of the precipitating factors that led to the later movement in <a href="/wiki/Scientific_publishing">scientific publishing</a> known as <a href="/wiki/Open_access_(publishing)">open access</a>.<sup><a href="#cite_note-SPARC_2021-15">[15]</a></sup> <a href="/wiki/Mathematician">Mathematicians</a> and scientists regularly upload their papers to arXiv.org for worldwide access<sup><a href="#cite_note-16">[16]</a></sup> and sometimes for reviews before they are published in <a href="/wiki/Peer_review">peer-reviewed</a> <a href="/wiki/Academic_journal">journals</a>. Ginsparg was awarded a <a href="/wiki/MacArthur_Fellowship">MacArthur Fellowship</a> in 2002 for his establishment of arXiv.<sup><a href="#cite_note-17">[17]</a></sup> 
The annual budget for arXiv was approximately $826,000 for 2013 to 2017, funded jointly by Cornell University Library, the <a href="/wiki/Simons_Foundation">Simons Foundation</a> (in both gift and <a href="/wiki/Challenge_grant">challenge grant</a> forms) and annual fee income from member institutions.<sup><a href="#cite_note-18">[18]</a></sup> This model arose in 2010, when Cornell sought to broaden the financial funding of the project by asking institutions to make annual voluntary contributions based on the amount of download usage by each institution. Each member institution pledges a five-year funding commitment to support arXiv. Based on institutional usage ranking, the annual fees are set in four tiers from $1,000 to $4,400. Cornell's goal is to raise at least $504,000 per year through membership fees generated by approximately 220 institutions.<sup><a href="#cite_note-19">[19]</a></sup>
</p><p>In September 2011, Cornell University Library took overall administrative and financial responsibility for arXiv's operation and development. Ginsparg was quoted in the <i><a href="/wiki/Chronicle_of_Higher_Education">Chronicle of Higher Education</a></i> as saying it "was supposed to be a three-hour tour, not a life sentence".<sup><a href="#cite_note-CHE_2011-20">[20]</a></sup> However, Ginsparg remains on the arXiv's Scientific Advisory Board and its Physics Advisory Committee.<sup><a href="#cite_note-21">[21]</a></sup><sup><a href="#cite_note-22">[22]</a></sup>
</p>
<h2>Moderation process and endorsement[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=2">edit</a>]</h2>
<p>Although arXiv is not <a href="/wiki/Peer_review">peer reviewed</a>, a collection of moderators for each area review the <a href="/wiki/Electronic_submission">submissions</a>; they may recategorize any that are deemed off-topic,<sup><a href="#cite_note-mckinney-23">[23]</a></sup> or reject submissions that are not scientific papers, or sometimes for undisclosed reasons.<sup><a href="#cite_note-Merali-24">[24]</a></sup> The lists of moderators for many sections of arXiv are publicly available,<sup><a href="#cite_note-25">[25]</a></sup> but moderators for most of the physics sections remain unlisted.
</p><p>Additionally, an "endorsement" system was introduced in 2004 as part of an effort to ensure content is relevant and of interest to current research in the specified disciplines.<sup><a href="#cite_note-26">[26]</a></sup> Under the system, for categories that use it, an author must be endorsed by an established arXiv author before being allowed to submit papers to those categories. Endorsers are not asked to review the paper for errors, but to check whether the paper is appropriate for the intended subject area.<sup><a href="#cite_note-mckinney-23">[23]</a></sup> New authors from recognized academic institutions generally receive automatic endorsement, which in practice means that they do not need to deal with the endorsement system at all. However, the endorsement system has attracted criticism for allegedly restricting scientific inquiry.<sup><a href="#cite_note-27">[27]</a></sup><sup><a href="#cite_note-28">[28]</a></sup>
</p><p>A majority of the <a href="/wiki/E-print">e-prints</a> are also submitted to <a href="/wiki/Scientific_journal">journals</a> for publication, but some work, including some very influential papers, remain purely as e-prints and are never published in a peer-reviewed journal. A well-known example of the latter is an outline of a proof of <a href="/wiki/Thurston%27s_geometrization_conjecture">Thurston's geometrization conjecture</a>, including the <a href="/wiki/Poincar%C3%A9_conjecture">Poincaré conjecture</a> as a particular case, uploaded by <a href="/wiki/Grigori_Perelman">Grigori Perelman</a> in November 2002.<sup><a href="#cite_note-29">[29]</a></sup> Perelman appears content to forgo the traditional peer-reviewed journal process, stating: "If anybody is interested in my way of solving the problem, it's all there [on the arXiv] – let them go and read about it".<sup><a href="#cite_note-30">[30]</a></sup> Despite this non-traditional method of publication, other mathematicians recognized this work by offering the <a href="/wiki/Fields_Medal">Fields Medal</a> and <a href="/wiki/Millennium_Prize_Problems">Clay Mathematics Millennium Prizes</a> to Perelman, both of which he refused.<sup><a href="#cite_note-31">[31]</a></sup>
</p><p>While arXiv does contain some dubious e-prints, such as those claiming to refute famous theorems or proving famous conjectures such as <a href="/wiki/Fermat%27s_Last_Theorem">Fermat's Last Theorem</a> using only high-school mathematics, a 2002 article which appeared in <i><a href="/wiki/Notices_of_the_American_Mathematical_Society">Notices of the American Mathematical Society</a></i> described those as "surprisingly rare".<sup><a href="#cite_note-32">[32]</a></sup> arXiv generally re-classifies these works, e.g. in "General mathematics", rather than deleting them;<sup><a href="#cite_note-33">[33]</a></sup> however, some authors have voiced concern over the lack of transparency in the arXiv screening process.<sup><a href="#cite_note-Merali-24">[24]</a></sup>
</p>
<h2>Submission formats[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=3">edit</a>]</h2>
<p>Papers can be submitted in any of several formats, including <a href="/wiki/LaTeX">LaTeX</a>, and <a href="/wiki/PDF">PDF</a> printed from a <a href="/wiki/Word_processor">word processor</a> other than TeX or LaTeX. The <a href="/wiki/Electronic_submission">submission</a> is rejected by the arXiv software if generating the final <a href="/wiki/PDF">PDF</a> file fails, if any image file is too large, or if the total size of the submission is too large. arXiv now allows one to store and modify an incomplete submission, and only finalize the submission when ready. The time stamp on the article is set when the submission is finalized.
</p>
<h2>Access[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=4">edit</a>]</h2>
<div><p><a href="/wiki/File:Arxiv.org_abstract_view.png"></a></p><div><p>A screenshot of viewing a paper's abstract on arxiv.org in 2021</p></div></div>
<p>The standard access route is through the arXiv.org website or one of several mirrors. Other interfaces and access routes have also been created by other un-associated organisations.
</p><p><a href="/wiki/Metadata">Metadata</a> for arXiv is made available through <a href="/wiki/OAI-PMH">OAI-PMH</a>, the standard for <a href="/wiki/Open_access_repositories">open access repositories</a>.<sup><a href="#cite_note-34">[34]</a></sup> Content is therefore indexed in all major consumers of such data, such as <a href="/wiki/BASE_(search_engine)">BASE</a>, <a href="/wiki/CORE_(research_service)">CORE</a> and <a href="/wiki/Unpaywall">Unpaywall</a>. As of 2020, the Unpaywall dump links over 500,000 arxiv URLs as the <a href="/wiki/Open_access">open access</a> version of a work found in <a href="/wiki/CrossRef">CrossRef</a> data from the publishers, making arXiv a top 10 global host of <a href="/wiki/Green_open_access">green open access</a>.
</p><p>Finally, researchers can select sub-fields and receive daily e-mailings or <a href="/wiki/RSS_feed">RSS feeds</a> of all submissions in them.
</p>
<h2>Copyright status of files[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=5">edit</a>]</h2>
<p>Files on arXiv can have a number of different copyright statuses:<sup><a href="#cite_note-35">[35]</a></sup>
</p>
<ol><li>Some are <a href="/wiki/Public_domain">public domain</a>, in which case they will have a statement saying so.</li>
<li>Some are available under either the <a href="/wiki/Creative_Commons">Creative Commons</a> 4.0 <a href="/wiki/Creative_Commons_licenses">Attribution-ShareAlike license</a> or the Creative Commons 4.0 <a href="/wiki/Creative_Commons_licenses">Attribution-Noncommercial-ShareAlike license</a>.</li>
<li>Some are copyright to the publisher, but the author has the right to distribute them and has given arXiv a non-exclusive irrevocable license to distribute them.</li>
<li>Most are copyright to the author, and arXiv has only a non-exclusive irrevocable license to distribute them.</li></ol>
<h2>See also[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=6">edit</a>]</h2>
<ul><li><a href="/wiki/List_of_academic_databases_and_search_engines">List of academic databases and search engines</a></li>
<li><a href="/wiki/List_of_academic_journals_by_preprint_policy">List of academic journals by preprint policy</a></li>
<li><a href="/wiki/List_of_preprint_repositories">List of preprint repositories</a></li></ul>
<h2>Citations[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=7">edit</a>]</h2>
<div><ol>
<li><b><a href="#cite_ref-Steele_1-0">^</a></b> <cite>Steele, Bill (Fall 2012). <a href="http://ezramagazine.cornell.edu/FALL12/CoverStorySidebar2.html">"Library-managed 'arXiv' spreads scientific advances rapidly and worldwide"</a>. <i>Ezra</i>. Ithaca, New York: Cornell University. p. 9. <a href="/wiki/OCLC_(identifier)">OCLC</a> <a href="//www.worldcat.org/oclc/263846378">263846378</a>. <a href="https://web.archive.org/web/20150111003819/http://ezramagazine.cornell.edu/FALL12/CoverStorySidebar2.html">Archived</a> from the original on January 11, 2015. <q>Pronounce it 'archive'. The X represents the Greek letter <a href="/wiki/Chi_(letter)">chi</a> [ χ ].</q></cite>
</li>
<li><b><a href="#cite_ref-2">^</a></b> <cite>Ginsparg, Paul (2011). "It was twenty years ago today ...". <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/1108.2700">1108.2700</a> [<a href="//arxiv.org/archive/cs.DL">cs.DL</a>].</cite>
</li>
<li><b><a href="#cite_ref-3">^</a></b> <cite><a href="http://news.library.cornell.edu/content/online-scientific-repository-hits-milestone">"Online Scientific Repository Hits Milestone: With 500,000 Articles, arXiv Established as Vital Library Resource"</a>. News.library.cornell.edu. October 3, 2008. Retrieved July 21, 2013.</cite>
</li>
<li><b><a href="#cite_ref-4">^</a></b> <cite>Vence, Tracy (December 29, 2014), <a href="http://www.the-scientist.com/?articles.view/articleNo/41677/title/Q-A--One-Million-Preprints-and-Counting">"One Million Preprints and Counting: A conversation with arXiv founder Paul Ginsparg"</a>, <i><a href="/wiki/The_Scientist_(magazine)">The Scientist</a></i></cite>
</li>
<li>^ <a href="#cite_ref-dddmag2015_5-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-dddmag2015_5-1"><sup><i><b>b</b></i></sup></a> <cite>Staff (January 13, 2015). <a href="http://www.dddmag.com/articles/2015/01/news-open-access-journals">"In the News: Open Access Journals"</a>. <i><a href="/wiki/Drug_Discovery_%26_Development">Drug Discovery &amp; Development</a></i>.</cite>
</li>
<li><b><a href="#cite_ref-6">^</a></b> <cite><a href="https://arxiv.org/show_monthly_submissions">"arXiv monthly submission rate statistics"</a>. Arxiv.org. Retrieved April 3, 2021.</cite>
</li>
<li><b><a href="#cite_ref-7">^</a></b> <cite><a href="http://www.cs.cornell.edu/~ginsparg/physics/blurb/fig3.gif">"Image"</a> (GIF). <i>Cs.cornell.edu</i>. Retrieved March 9, 2019.</cite>
</li>
<li><b><a href="#cite_ref-8">^</a></b> <cite>Ginsparg, Paul (August 4, 2021). <a href="//www.ncbi.nlm.nih.gov/pmc/articles/PMC8335983">"Lessons from arXiv's 30 years of information sharing"</a>. <i>Nature Reviews Physics</i>. <b>3</b> (9): 602–603. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2Fs42254-021-00360-z">10.1038/s42254-021-00360-z</a>. <a href="/wiki/ISSN_(identifier)">ISSN</a> <a href="//www.worldcat.org/issn/2522-5820">2522-5820</a>. <a href="/wiki/PMC_(identifier)">PMC</a> <a href="//www.ncbi.nlm.nih.gov/pmc/articles/PMC8335983">8335983</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/34377944">34377944</a>.</cite>
</li>
<li><b><a href="#cite_ref-9">^</a></b> <cite>O'Connell, Heath (2002). <a href="https://core.ac.uk/download/pdf/11876491.pdf">"Physicists Thriving with Paperless Publishing"</a> (PDF). <i>High Energy Physics Libraries Webzine</i>. <b>6</b> (6): 3. <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/physics/0007040">physics/0007040</a>. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2000physics...7040O">2000physics...7040O</a>. <a href="https://ghostarchive.org/archive/20221009/https://core.ac.uk/download/pdf/11876491.pdf">Archived</a> (PDF) from the original on October 9, 2022.</cite>
</li>
<li><b><a href="#cite_ref-10">^</a></b> <cite>Feder, Toni (November 8, 2021). <a href="https://physicstoday.scitation.org/do/10.1063/PT.6.4.20211108a/abs/">"Joanne Cohn and the email list that led to arXiv"</a>. <i><a href="/wiki/Physics_Today">Physics Today</a></i>. <b>2021</b> (4): 1108a. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2021PhT..2021d1108.">2021PhT..2021d1108.</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1063%2FPT.6.4.20211108a">10.1063/PT.6.4.20211108a</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:244015728">244015728</a>.</cite>
</li>
<li><b><a href="#cite_ref-11">^</a></b> <cite>Feder, Toni (November 8, 2021). <a href="https://physicstoday.scitation.org/do/10.1063/PT.6.4.20211108a/full/">"Joanne Cohn and the email list that led to arXiv"</a>. <i><a href="/wiki/Physics_Today">Physics Today</a></i>. <b>2021</b> (4): 1108a. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2021PhT..2021d1108.">2021PhT..2021d1108.</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1063%2FPT.6.4.20211108a">10.1063/PT.6.4.20211108a</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:244015728">244015728</a>.</cite>
</li>
<li><b><a href="#cite_ref-12">^</a></b> <cite>Ginsparg, Paul (October 1, 2008). <a href="https://physicsworld.com/a/the-global-village-pioneers/">"The global-village pioneers"</a>. <i><a href="/wiki/Physics_World">Physics World</a></i>. Retrieved October 10, 2020.</cite>
</li>
<li><b><a href="#cite_ref-13">^</a></b> <cite>Butler, Declan (July 5, 2001). <a href="https://doi.org/10.1038%2F35083708">"Los Alamos Loses Physics Archive as Preprint Pioneer Heads East"</a>. <i>Nature</i>. <b>412</b> (6842): 3–4. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2001Natur.412....3B">2001Natur.412....3B</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2F35083708">10.1038/35083708</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/11452262">11452262</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:1527860">1527860</a>.</cite>
</li>
<li><b><a href="#cite_ref-14">^</a></b> <cite><a href="https://arxiv.org/servers.html">"arXiv mirror sites"</a>. arXiv. <a href="https://web.archive.org/web/20200316205130/http://arxiv.org/help/mirrors">Archived</a> from the original on March 16, 2020. Retrieved April 6, 2020.</cite>
</li>
<li>^ <a href="#cite_ref-SPARC_2021_15-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-SPARC_2021_15-1"><sup><i><b>b</b></i></sup></a> <cite><a href="https://sparcopen.org/news/2021/celebrating-30-years-of-arxiv-and-its-lasting-legacy-on-scientific-advancement/">"Celebrating 30 Years of arXiv and Its Lasting Legacy on Scientific Advancement"</a>. <i>SPARC</i>. October 25, 2021.</cite>
</li>
<li><b><a href="#cite_ref-16">^</a></b> <cite>Glanz, James (May 1, 2001). <a href="https://www.nytimes.com/2001/05/01/science/world-science-becomes-global-village-archive-opens-new-realm-research.html">"The World of Science Becomes a Global Village; Archive Opens a New Realm of Research"</a>. <i><a href="/wiki/The_New_York_Times">The New York Times</a></i>.</cite>
</li>
<li><b><a href="#cite_ref-17">^</a></b> <cite>Bill Steele (September 23, 2002). <a href="http://www.news.cornell.edu/releases/sept02/ginsparg-MacArthur.ws.html">"Cornell professor Paul Ginsparg, science communication rebel, named a MacArthur Foundation fellow; three other alumni also receive 'genius award' fellowships"</a>.</cite>
</li>
<li><b><a href="#cite_ref-18">^</a></b> <cite><a href="https://confluence.cornell.edu/download/attachments/127116484/arXiv+Business+Model.pdf">"CORNELL UNIVERSITY LIBRARY ARXIV FINANCIAL PROJECTIONS FOR 2013-2017"</a> (PDF). <i>Confluence.cornell.edu</i>. March 28, 2012. Retrieved February 26, 2017.</cite>
</li>
<li><b><a href="#cite_ref-19">^</a></b> <cite><a href="https://arxiv.org/about/ourmembers">"arXiv Member Institutions (2021) - arXiv about - Our Members"</a>. <i>arXiv.org</i>. Retrieved December 27, 2021.</cite>
</li>
<li><b><a href="#cite_ref-CHE_2011_20-0">^</a></b> <cite>Fischman, Joah (August 10, 2011). <a href="http://chronicle.com/blogs/wiredcampus/the-first-free-research-sharing-site-arxiv-turns-20/32778">"The First Free Research-Sharing Site, arXiv, Turns 20 With an Uncertain Future"</a>. <i>Chronicle of Higher Education</i>. Retrieved August 12, 2011.</cite>
</li>
<li><b><a href="#cite_ref-21">^</a></b> <cite><a href="https://arxiv.org/about/people/scientific_ad_board">"arXiv Scientific Advisory Board | arXiv e-print repository"</a>. <i>arxiv.org</i>. Retrieved October 10, 2020.</cite>
</li>
<li><b><a href="#cite_ref-22">^</a></b> <cite><a href="https://arxiv.org/help/physics">"About the Physics Archive | arXiv e-print repository"</a>. <i>arxiv.org</i>. Retrieved October 10, 2020.</cite>
</li>
<li>^ <a href="#cite_ref-mckinney_23-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-mckinney_23-1"><sup><i><b>b</b></i></sup></a> <cite>McKinney, Michelle (2011), "ArXiv.org", <i>Reference Reviews</i>, <b>25</b> (7): 35–36, <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1108%2F09504121111168622">10.1108/09504121111168622</a></cite>
</li>
<li>^ <a href="#cite_ref-Merali_24-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-Merali_24-1"><sup><i><b>b</b></i></sup></a> <cite>Merali, Zeeya (January 29, 2016). <a href="http://www.nature.com/news/arxiv-rejections-lead-to-spat-over-screening-process-1.19267">"ArXiv rejections lead to spat over screening process"</a>. <i><a href="/wiki/Nature_(journal)">Nature</a></i>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2Fnature.2016.19267">10.1038/nature.2016.19267</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:189061969">189061969</a>. Retrieved December 14, 2017.</cite>
</li>
<li><b><a href="#cite_ref-25">^</a></b> <a href="https://arxiv.org/corr/subjectclasses">Computing Research Repository Subject Areas and Moderators</a>; <a href="https://arxiv.org/new/stat.html">Statistics archive</a>; <a href="https://arxiv.org/new/q-bio.html">Quantitative Biology archive</a>; <a href="https://arxiv.org/new/physics.html">Physics archive</a>
</li>
<li><b><a href="#cite_ref-26">^</a></b> <cite>Ginsparg, Paul (2006), "As we may read", <i>Journal of Neuroscience</i>, <b>26</b> (38): 9606–9608, <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1523%2FJNEUROSCI.3161-06.2006">10.1523/JNEUROSCI.3161-06.2006</a>, <a href="/wiki/PMC_(identifier)">PMC</a> <a href="//www.ncbi.nlm.nih.gov/pmc/articles/PMC6674456">6674456</a>, <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/16988030">16988030</a></cite>
</li>
<li><b><a href="#cite_ref-27">^</a></b> <cite>Greechie, Richard; Pulmannova, Sylvia; Svozil, Karl (July 2005), "Preface to the Proceedings of Quantum Structures 2002", <i>International Journal of Theoretical Physics</i>, <b>44</b> (7): 691–692, <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2005IJTP...44..691G">2005IJTP...44..691G</a>, <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1007%2Fs10773-005-7053-z">10.1007/s10773-005-7053-z</a>, <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:121442106">121442106</a>, <q>The new endorsement system may contribute to an effective barrier, a digital divide</q></cite>
</li>
<li><b><a href="#cite_ref-28">^</a></b> <cite>Josephson, Brian (February 23, 2005). <a href="https://doi.org/10.1038%2F433800a">"Vital resource should be open to all physicists"</a>. <i>Nature</i>. <b>433</b> (7028): 800. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2005Natur.433..800J">2005Natur.433..800J</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2F433800a">10.1038/433800a</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/15729314">15729314</a>.</cite>
</li>
<li><b><a href="#cite_ref-29">^</a></b> <cite>Perelman, Grisha (November 11, 2002). "The entropy formula for the Ricci flow and its geometric applications". <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/math.DG/0211159">math.DG/0211159</a>.</cite>
</li>
<li><b><a href="#cite_ref-30">^</a></b> <cite>Lobastova, Nadejda; Hirst, Michael (August 21, 2006). <a href="http://www.smh.com.au/news/world/maths-genius-living-in-poverty/2006/08/20/1156012411120.html">"Maths genius living in poverty"</a>. <i><a href="/wiki/Sydney_Morning_Herald">Sydney Morning Herald</a></i>.</cite>
</li>
<li><b><a href="#cite_ref-31">^</a></b> <cite>Kaufman, Marc (July 2, 2010), <a href="https://www.washingtonpost.com/wp-dyn/content/article/2010/07/01/AR2010070106247.html">"Russian mathematician wins $1 million prize, but he appears to be happy with $0"</a>, <i>Washington Post</i></cite>
</li>
<li><b><a href="#cite_ref-32">^</a></b> <cite>Jackson, Allyn (2002). <a href="https://www.ams.org/notices/200201/fea-preprints.pdf">"From Preprints to E-prints: The Rise of Electronic Preprint Servers in Mathematics"</a> (PDF). <i>Notices of the American Mathematical Society</i>. <b>49</b> (1): 23–32.</cite>
</li>
<li><b><a href="#cite_ref-33">^</a></b> <cite>Ginsparg, Paul (August 2011). <a href="https://dx.doi.org/10.1038/476145a">"ArXiv at 20"</a>. <i>Nature</i>. <b>476</b> (7359): 145–147. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2011Natur.476..145G">2011Natur.476..145G</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2F476145a">10.1038/476145a</a>. <a href="/wiki/ISSN_(identifier)">ISSN</a> <a href="//www.worldcat.org/issn/0028-0836">0028-0836</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/21833066">21833066</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:4421407">4421407</a>.</cite>
</li>
<li><b><a href="#cite_ref-34">^</a></b> <cite><a href="https://arxiv.org/help/oa/index">"Open Archives Initiative (OAI)"</a>. <i>arxiv.org</i>. Retrieved April 25, 2020.</cite>
</li>
<li><b><a href="#cite_ref-35">^</a></b> <cite><a href="https://arxiv.org/help/license">"arXiv License Information"</a>. Arxiv.org. Retrieved July 21, 2013.</cite>
</li>
</ol></div>
<h2>General and cited sources[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=8">edit</a>]</h2>
<div>
<ul><li><cite>Butler, Declan (2003). <a href="https://doi.org/10.1038%2F425548b">"Biologists Join Physics Preprint Club"</a>. <i><a href="/wiki/Nature_(journal)">Nature</a></i>. <b>425</b> (6958): 548. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2003Natur.425..548B">2003Natur.425..548B</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2F425548b">10.1038/425548b</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/14534551">14534551</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:4374168">4374168</a>.</cite></li>
<li><cite>Choi, Charles Q. (2003). <a href="https://web.archive.org/web/20050313151648/http://www.biomedcentral.com/news/20030930/03">"Biology's New Online Archive"</a>. <i><a href="/wiki/The_Scientist_(magazine)">The Scientist</a></i>. Archived from <a href="http://www.biomedcentral.com/news/20030930/03/">the original</a> on March 13, 2005. Retrieved June 21, 2005.</cite></li>
<li><cite>Giles, Jim (2003). <a href="https://doi.org/10.1038%2F426007a">"Preprint Server Seeks Way to Halt Plagiarists"</a>. <i><a href="/wiki/Nature_(journal)">Nature</a></i>. <b>426</b> (6962): 7. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2003Natur.426Q...7G">2003Natur.426Q...7G</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2F426007a">10.1038/426007a</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/14603280">14603280</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:29003994">29003994</a>.</cite></li>
<li><cite>Ginsparg, Paul (1997). <a href="https://www.cs.cornell.edu/~ginsparg/physics/blurb/pg96unesco.html">"Winners and Losers in the Global Research Village"</a>. <i>The Serials Librarian</i>. <b>30</b> (3–4): 83–95. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1300%2FJ123v30n03_13">10.1300/J123v30n03_13</a>.</cite></li>
<li><cite>Halpern, Joseph Y. (1998). <a href="https://doi.org/10.1045%2Fnovember98-halpern">"A Computing Research Repository"</a>. <i><a href="/wiki/D-Lib_Magazine">D-Lib Magazine</a></i>. <b>4</b> (11). <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1045%2Fnovember98-halpern">10.1045/november98-halpern</a>.</cite></li>
<li><cite>Halpern, Joseph Y. (2000). "CoRR: A Computing Research Repository". <i><a href="/w/index.php?title=Journal_of_Computer_Documentation&amp;action=edit&amp;redlink=1">Journal of Computer Documentation</a></i>. <b>24</b> (2): 41–48. <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/cs.DL/0005003">cs.DL/0005003</a>. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2000cs........5003H">2000cs........5003H</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F337271.337274">10.1145/337271.337274</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:5453868">5453868</a>.</cite></li>
<li><cite>Luce, Richard E. (2001). <a href="http://www.istl.org/01-winter/article3.html">"E-Prints Intersect the Digital Library: Inside the Los Alamos arXiv"</a>. <i>Issues in Science and Technology Librarianship</i> (29). <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.5062%2FF44B2Z95">10.5062/F44B2Z95</a>.</cite></li>
<li><cite>McKiernan, Gerry (2000). <a href="https://web.archive.org/web/20050505044157/http://www.public.iastate.edu/~gerrymck/arXiv.org.pdf">"ArXiv.org: The los Alamos National Laboratory e‐print server"</a> (PDF). <i><a href="/w/index.php?title=International_Journal_on_Grey_Literature&amp;action=edit&amp;redlink=1">International Journal on Grey Literature</a></i>. <b>1</b> (3): 127–138. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1108%2F14666180010345564">10.1108/14666180010345564</a>. Archived from <a href="http://www.public.iastate.edu/~gerrymck/arXiv.org.pdf">the original</a> (PDF) on May 5, 2005.</cite></li>
<li><cite>Pinfield, Stephen (2001). <a href="https://doi.org/10.1045%2Fdecember2001-pinfield">"How Do Physicists Use an E-Print Archive? Implications for Institutional E-Print Services"</a>. <i><a href="/wiki/D-Lib_Magazine">D-Lib Magazine</a></i>. <b>7</b> (12). <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1045%2Fdecember2001-pinfield">10.1045/december2001-pinfield</a>.</cite></li>
<li><cite>Quigley, Brian (2000). <a href="http://www.allbusiness.com/technology/internet-search-engines/1149636-1.html">"Physics Databases and the Los Alamos e-Print Archive"</a>. <i><a href="/wiki/EContent">EContent</a></i>. <b>23</b> (5): 22–26.</cite></li>
<li><cite>Taubes, Gary (1993). "Publication by Electronic Mail Takes Physics by Storm". <i><a href="/wiki/Science_(journal)">Science</a></i>. <b>259</b> (5099): 1246–1248. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/1993Sci...259.1246T">1993Sci...259.1246T</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1126%2Fscience.259.5099.1246">10.1126/science.259.5099.1246</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/17732237">17732237</a>.</cite></li>
<li><cite>Warner, Simeon (2001). "Open Archives Initiative protocol development and implementation at arXiv". <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/cs/0101027">cs/0101027</a>.</cite></li>
<li><cite>"What Is q-bio?". <i>Open Access Now</i>. 2004.</cite></li></ul>
</div>
<h2>External links[<a href="/w/index.php?title=ArXiv&amp;action=edit&amp;section=9">edit</a>]</h2>
<div>
<p></p>
<p>Wikimedia Commons has media related to <a href="https://commons.wikimedia.org/wiki/Category:ArXiv.org">ArXiv.org</a>.</p></div>
<ul><li><a href="https://arxiv.org/">Official website</a></li></ul>
</div>||||I|||| Jump to content
 Toggle sidebar
 Search
 * Create account
 Personal tools
 * Create account
 * Log in

 Pages for logged out editors learn more

 * Talk
 * Contributions
 Navigation
 * Main page
 * Contents
 * Current events
 * Random article
 * About Wikipedia
 * Contact us
 * Donate
 Contribute
 * Help
 * Learn to edit
 * Community portal
 * Recent changes
 * Upload file
 Tools
 * What links here
 * Related changes
 * Upload file
 * Special pages
 * Permanent link
 * Page information
 * Cite this page
 * Wikidata item
 Print/export
 * Download as PDF
 * Printable version
 In other projects
 * Wikimedia Commons
 * Wikiversity
 Languages
 On this Wikipedia the language links are at the top of the page across from the article title. Go to top.

 Contents

 move to sidebar hide
 * (Top)
 * 1 History
 * 2 Moderation process and endorsement
 * 3 Submission formats
 * 4 Access
 * 5 Copyright status of files
 * 6 See also
 * 7 Citations
 * 8 General and cited sources
 * 9 External links
 Toggle the table of contents

 arXiv

 50 languages
 * Afrikaans
 * العربية
 * Asturianu
 * বাংলা
 * Bân-lâm-gú
 * Беларуская
 * Български
 * Català
 * Čeština
 * Dansk
 * Deutsch
 * Eesti
 * Ελληνικά
 * Español
 * Esperanto
 * فارسی
 * Français
 * Gàidhlig
 * Galego
 * 한국어
 * Հայերեն
 * हिन्दी
 * Hornjoserbsce
 * Ilokano
 * Bahasa Indonesia
 * Italiano
 * Македонски
 * Bahasa Melayu
 * မြန်မာဘာသာ
 * Nederlands
 * 日本語
 * Norsk bokmål
 * ਪੰਜਾਬੀ
 * Polski
 * Português
 * Română
 * Русский
 * Scots
 * Simple English
 * Slovenčina
 * Slovenščina
 * Српски / srpski
 * Srpskohrvatski / српскохрватски
 * Suomi
 * Svenska
 * Türkçe
 * Українська
 * اردو
 * Tiếng Việt
 * 中文
 Edit links
 * Article
 * Talk
 English
 * Read
 * Edit
 * View history
 More
 * Read
 * Edit
 * View history
 From Wikipedia, the free encyclopedia

 arXiv
 
 
 Type of site Science 
 Available in English 
 Owner Cornell University 
 Created by Paul Ginsparg 
 URL arxiv.org 
 Commercial No 
 Launched August 14, 1991 ; 31 years ago
 Current status Online 
 ISSN 2331-8422 
 OCLC number 228652809 
 

 arXiv (pronounced "archive"—the X represents the Greek letter chi ⟨χ⟩)[1] is an open-access repository of electronic preprints and postprints (known as e-prints) approved for posting after moderation, but not peer review. It consists of scientific papers in the fields of mathematics, physics, astronomy, electrical engineering, computer science, quantitative biology, statistics, mathematical finance and economics, which can be accessed online. In many fields of mathematics and physics, almost all scientific papers are self-archived on the arXiv repository before publication in a peer-reviewed journal. Some publishers also grant permission for authors to archive the peer-reviewed postprint. Begun on August 14, 1991, arXiv.org passed the half-million-article milestone on October 3, 2008,[2][3] and had hit a million by the end of 2014.[4][5] As of April 2021, the submission rate is about 16,000 articles per month.[6]

 History [ edit ]

 A screenshot of the arXiv taken in 1994,[7] using the browser NCSA Mosaic. At the time, HTML forms were a new technology.
 ArXiv's yearly submission rate growth over 30 years since its beginning with topics labelled by the standard abbreviations used on arxiv.org[8]

 arXiv was made possible by the compact TeX file format, which allowed scientific papers to be easily transmitted over the Internet and rendered client-side.[9] Around 1990, Joanne Cohn began emailing physics preprints to colleagues as TeX files, but the number of papers being sent soon filled mailboxes to capacity.[10] Paul Ginsparg recognized the need for central storage, and in August 1991 he created a central repository mailbox stored at the Los Alamos National Laboratory (LANL) which could be accessed from any computer.[11] Additional modes of access were soon added: FTP in 1991, Gopher in 1992, and the World Wide Web in 1993.[5][12] The term e-print was quickly adopted to describe the articles.

 It began as a physics archive, called the LANL preprint archive, but soon expanded to include astronomy, mathematics, computer science, quantitative biology and, most recently, statistics. Its original domain name was xxx.lanl.gov. Due to LANL's lack of interest in the rapidly expanding technology, in 2001 Ginsparg changed institutions to Cornell University and changed the name of the repository to arXiv.org.[13] It is now hosted principally by Cornell, with five mirrors around the world.[14]

 ArXiv was an early adopter and promoter of preprints.[15] Its success in sharing preprints was one of the precipitating factors that led to the later movement in scientific publishing known as open access.[15] Mathematicians and scientists regularly upload their papers to arXiv.org for worldwide access[16] and sometimes for reviews before they are published in peer-reviewed journals. Ginsparg was awarded a MacArthur Fellowship in 2002 for his establishment of arXiv.[17] The annual budget for arXiv was approximately $826,000 for 2013 to 2017, funded jointly by Cornell University Library, the Simons Foundation (in both gift and challenge grant forms) and annual fee income from member institutions.[18] This model arose in 2010, when Cornell sought to broaden the financial funding of the project by asking institutions to make annual voluntary contributions based on the amount of download usage by each institution. Each member institution pledges a five-year funding commitment to support arXiv. Based on institutional usage ranking, the annual fees are set in four tiers from $1,000 to $4,400. Cornell's goal is to raise at least $504,000 per year through membership fees generated by approximately 220 institutions.[19]

 In September 2011, Cornell University Library took overall administrative and financial responsibility for arXiv's operation and development. Ginsparg was quoted in the Chronicle of Higher Education as saying it "was supposed to be a three-hour tour, not a life sentence".[20] However, Ginsparg remains on the arXiv's Scientific Advisory Board and its Physics Advisory Committee.[21][22]

 Moderation process and endorsement [ edit ]

 Although arXiv is not peer reviewed, a collection of moderators for each area review the submissions; they may recategorize any that are deemed off-topic,[23] or reject submissions that are not scientific papers, or sometimes for undisclosed reasons.[24] The lists of moderators for many sections of arXiv are publicly available,[25] but moderators for most of the physics sections remain unlisted.

 Additionally, an "endorsement" system was introduced in 2004 as part of an effort to ensure content is relevant and of interest to current research in the specified disciplines.[26] Under the system, for categories that use it, an author must be endorsed by an established arXiv author before being allowed to submit papers to those categories. Endorsers are not asked to review the paper for errors, but to check whether the paper is appropriate for the intended subject area.[23] New authors from recognized academic institutions generally receive automatic endorsement, which in practice means that they do not need to deal with the endorsement system at all. However, the endorsement system has attracted criticism for allegedly restricting scientific inquiry.[27][28]

 A majority of the e-prints are also submitted to journals for publication, but some work, including some very influential papers, remain purely as e-prints and are never published in a peer-reviewed journal. A well-known example of the latter is an outline of a proof of Thurston's geometrization conjecture, including the Poincaré conjecture as a particular case, uploaded by Grigori Perelman in November 2002.[29] Perelman appears content to forgo the traditional peer-reviewed journal process, stating: "If anybody is interested in my way of solving the problem, it's all there [on the arXiv] – let them go and read about it".[30] Despite this non-traditional method of publication, other mathematicians recognized this work by offering the Fields Medal and Clay Mathematics Millennium Prizes to Perelman, both of which he refused.[31]

 While arXiv does contain some dubious e-prints, such as those claiming to refute famous theorems or proving famous conjectures such as Fermat's Last Theorem using only high-school mathematics, a 2002 article which appeared in Notices of the American Mathematical Society described those as "surprisingly rare".[32] arXiv generally re-classifies these works, e.g. in "General mathematics", rather than deleting them;[33] however, some authors have voiced concern over the lack of transparency in the arXiv screening process.[24]

 Submission formats [ edit ]

 Papers can be submitted in any of several formats, including LaTeX, and PDF printed from a word processor other than TeX or LaTeX. The submission is rejected by the arXiv software if generating the final PDF file fails, if any image file is too large, or if the total size of the submission is too large. arXiv now allows one to store and modify an incomplete submission, and only finalize the submission when ready. The time stamp on the article is set when the submission is finalized.

 Access [ edit ]

 A screenshot of viewing a paper's abstract on arxiv.org in 2021

 The standard access route is through the arXiv.org website or one of several mirrors. Other interfaces and access routes have also been created by other un-associated organisations.

 Metadata for arXiv is made available through OAI-PMH, the standard for open access repositories.[34] Content is therefore indexed in all major consumers of such data, such as BASE, CORE and Unpaywall. As of 2020, the Unpaywall dump links over 500,000 arxiv URLs as the open access version of a work found in CrossRef data from the publishers, making arXiv a top 10 global host of green open access.

 Finally, researchers can select sub-fields and receive daily e-mailings or RSS feeds of all submissions in them.

 Copyright status of files [ edit ]

 Files on arXiv can have a number of different copyright statuses:[35]

 1. Some are public domain, in which case they will have a statement saying so.
 2. Some are available under either the Creative Commons 4.0 Attribution-ShareAlike license or the Creative Commons 4.0 Attribution-Noncommercial-ShareAlike license.
 3. Some are copyright to the publisher, but the author has the right to distribute them and has given arXiv a non-exclusive irrevocable license to distribute them.
 4. Most are copyright to the author, and arXiv has only a non-exclusive irrevocable license to distribute them.

 See also [ edit ]

 * List of academic databases and search engines
 * List of academic journals by preprint policy
 * List of preprint repositories

 Citations [ edit ]

 1. ^ Steele, Bill (Fall 2012). "Library-managed 'arXiv' spreads scientific advances rapidly and worldwide". Ezra. Ithaca, New York: Cornell University. p. 9. OCLC 263846378. Archived from the original on January 11, 2015. "Pronounce it 'archive'. The X represents the Greek letter "chi" [ χ ]."
 2. ^ Ginsparg, Paul (2011). "It was twenty years ago today ...". arXiv:1108.2700 [cs.DL].
 3. ^ "Online Scientific Repository Hits Milestone: With 500,000 Articles, arXiv Established as Vital Library Resource". News.library.cornell.edu. October 3, 2008 . Retrieved July 21, 2013 .
 4. ^ Vence, Tracy (December 29, 2014), "One Million Preprints and Counting: A conversation with arXiv founder Paul Ginsparg", The Scientist
 5. ^ a b Staff (January 13, 2015). "In the News: Open Access Journals". Drug Discovery & Development.
 6. ^ "arXiv monthly submission rate statistics". Arxiv.org . Retrieved April 3, 2021 .
 7. ^ "Image" (GIF) . Cs.cornell.edu . Retrieved March 9, 2019 .
 8. ^ Ginsparg, Paul (August 4, 2021). "Lessons from arXiv's 30 years of information sharing". Nature Reviews Physics. 3 (9): 602–603. doi:10.1038/s42254-021-00360-z. ISSN 2522-5820. PMC 8335983. PMID 34377944.
 9. ^ O'Connell, Heath (2002). "Physicists Thriving with Paperless Publishing" (PDF) . High Energy Physics Libraries Webzine. 6 (6): 3. arXiv:physics/0007040. Bibcode:2000physics...7040O. Archived (PDF) from the original on October 9, 2022.
 10. ^ Feder, Toni (November 8, 2021). "Joanne Cohn and the email list that led to arXiv". Physics Today. 2021 (4): 1108a. Bibcode:2021PhT..2021d1108.. doi:10.1063/PT.6.4.20211108a. S2CID 244015728.
 11. ^ Feder, Toni (November 8, 2021). "Joanne Cohn and the email list that led to arXiv". Physics Today. 2021 (4): 1108a. Bibcode:2021PhT..2021d1108.. doi:10.1063/PT.6.4.20211108a. S2CID 244015728.
 12. ^ Ginsparg, Paul (October 1, 2008). "The global-village pioneers". Physics World . Retrieved October 10, 2020 .
 13. ^ Butler, Declan (July 5, 2001). "Los Alamos Loses Physics Archive as Preprint Pioneer Heads East". Nature. 412 (6842): 3–4. Bibcode:2001Natur.412....3B. doi:10.1038/35083708. PMID 11452262. S2CID 1527860.
 14. ^ "arXiv mirror sites". arXiv. Archived from the original on March 16, 2020 . Retrieved April 6, 2020 .
 15. ^ a b "Celebrating 30 Years of arXiv and Its Lasting Legacy on Scientific Advancement". SPARC. October 25, 2021.
 16. ^ Glanz, James (May 1, 2001). "The World of Science Becomes a Global Village; Archive Opens a New Realm of Research". The New York Times.
 17. ^ Bill Steele (September 23, 2002). "Cornell professor Paul Ginsparg, science communication rebel, named a MacArthur Foundation fellow; three other alumni also receive 'genius award' fellowships".
 18. ^ "CORNELL UNIVERSITY LIBRARY ARXIV FINANCIAL PROJECTIONS FOR 2013-2017" (PDF) . Confluence.cornell.edu. March 28, 2012 . Retrieved February 26, 2017 .
 19. ^ "arXiv Member Institutions (2021) - arXiv about - Our Members". arXiv.org . Retrieved December 27, 2021 .
 20. ^ Fischman, Joah (August 10, 2011). "The First Free Research-Sharing Site, arXiv, Turns 20 With an Uncertain Future". Chronicle of Higher Education . Retrieved August 12, 2011 .
 21. ^ "arXiv Scientific Advisory Board | arXiv e-print repository". arxiv.org . Retrieved October 10, 2020 .
 22. ^ "About the Physics Archive | arXiv e-print repository". arxiv.org . Retrieved October 10, 2020 .
 23. ^ a b McKinney, Michelle (2011), "ArXiv.org", Reference Reviews, 25 (7): 35–36, doi:10.1108/09504121111168622
 24. ^ a b Merali, Zeeya (January 29, 2016). "ArXiv rejections lead to spat over screening process". Nature. doi:10.1038/nature.2016.19267. S2CID 189061969 . Retrieved December 14, 2017 .
 25. ^ Computing Research Repository Subject Areas and Moderators ; Statistics archive ; Quantitative Biology archive ; Physics archive
 26. ^ Ginsparg, Paul (2006), "As we may read", Journal of Neuroscience, 26 (38): 9606–9608, doi:10.1523/JNEUROSCI.3161-06.2006, PMC 6674456, PMID 16988030
 27. ^ Greechie, Richard; Pulmannova, Sylvia; Svozil, Karl (July 2005), "Preface to the Proceedings of Quantum Structures 2002", International Journal of Theoretical Physics, 44 (7): 691–692, Bibcode:2005IJTP...44..691G, doi:10.1007/s10773-005-7053-z, S2CID 121442106, "The new endorsement system may contribute to an effective barrier, a digital divide"
 28. ^ Josephson, Brian (February 23, 2005). "Vital resource should be open to all physicists". Nature. 433 (7028): 800. Bibcode:2005Natur.433..800J. doi:10.1038/433800a. PMID 15729314.
 29. ^ Perelman, Grisha (November 11, 2002). "The entropy formula for the Ricci flow and its geometric applications". arXiv:math.DG/0211159.
 30. ^ Lobastova, Nadejda; Hirst, Michael (August 21, 2006). "Maths genius living in poverty". Sydney Morning Herald.
 31. ^ Kaufman, Marc (July 2, 2010), "Russian mathematician wins $1 million prize, but he appears to be happy with $0", Washington Post
 32. ^ Jackson, Allyn (2002). "From Preprints to E-prints: The Rise of Electronic Preprint Servers in Mathematics" (PDF) . Notices of the American Mathematical Society. 49 (1): 23–32.
 33. ^ Ginsparg, Paul (August 2011). "ArXiv at 20". Nature. 476 (7359): 145–147. Bibcode:2011Natur.476..145G. doi:10.1038/476145a. ISSN 0028-0836. PMID 21833066. S2CID 4421407.
 34. ^ "Open Archives Initiative (OAI)". arxiv.org . Retrieved April 25, 2020 .
 35. ^ "arXiv License Information". Arxiv.org . Retrieved July 21, 2013 .

 General and cited sources [ edit ]

 * Butler, Declan (2003). "Biologists Join Physics Preprint Club". Nature. 425 (6958): 548. Bibcode:2003Natur.425..548B. doi:10.1038/425548b. PMID 14534551. S2CID 4374168.
 * Choi, Charles Q. (2003). "Biology's New Online Archive". The Scientist. Archived from the original on March 13, 2005 . Retrieved June 21, 2005 .
 * Giles, Jim (2003). "Preprint Server Seeks Way to Halt Plagiarists". Nature. 426 (6962): 7. Bibcode:2003Natur.426Q...7G. doi:10.1038/426007a. PMID 14603280. S2CID 29003994.
 * Ginsparg, Paul (1997). "Winners and Losers in the Global Research Village". The Serials Librarian. 30 (3–4): 83–95. doi:10.1300/J123v30n03_13.
 * Halpern, Joseph Y. (1998). "A Computing Research Repository". D-Lib Magazine. 4 (11). doi:10.1045/november98-halpern.
 * Halpern, Joseph Y. (2000). "CoRR: A Computing Research Repository". Journal of Computer Documentation. 24 (2): 41–48. arXiv:cs.DL/0005003. Bibcode:2000cs........5003H. doi:10.1145/337271.337274. S2CID 5453868.
 * Luce, Richard E. (2001). "E-Prints Intersect the Digital Library: Inside the Los Alamos arXiv". Issues in Science and Technology Librarianship (29). doi:10.5062/F44B2Z95.
 * McKiernan, Gerry (2000). "ArXiv.org: The los Alamos National Laboratory e‐print server" (PDF) . International Journal on Grey Literature. 1 (3): 127–138. doi:10.1108/14666180010345564. Archived from the original (PDF) on May 5, 2005.
 * Pinfield, Stephen (2001). "How Do Physicists Use an E-Print Archive? Implications for Institutional E-Print Services". D-Lib Magazine. 7 (12). doi:10.1045/december2001-pinfield.
 * Quigley, Brian (2000). "Physics Databases and the Los Alamos e-Print Archive". EContent. 23 (5): 22–26.
 * Taubes, Gary (1993). "Publication by Electronic Mail Takes Physics by Storm". Science. 259 (5099): 1246–1248. Bibcode:1993Sci...259.1246T. doi:10.1126/science.259.5099.1246. PMID 17732237.
 * Warner, Simeon (2001). "Open Archives Initiative protocol development and implementation at arXiv". arXiv:cs/0101027.
 * "What Is q-bio?". Open Access Now. 2004.

 External links [ edit ]

 Wikimedia Commons has media related to ArXiv.org.
 * Official website
 * v 
 * t 
 * e 
 Cornell University
 * Colleges: Agriculture and Life Sciences 
 * Architecture, Art, and Planning 
 * Arts and Sciences 
 * Engineering 
 * Hotel Administration 
 * Human Ecology 
 * Industrial and Labor Relations 
 Academics * Forestry (Defunct) 
 * Graduate and Professional Schools: Business 
 * Graduate 
 * Law 
 * Medical (Qatar, Sciences, Tri-Institutional) 
 * Public Affairs 
 * Veterinary 
 * Non-student-enrolling: Computing and Information Science
 * Sports: Ivy League 
 * Basketball (Men, Women) 
 * Baseball (Men) 
 * Football (Men) 
 * Ice Hockey (Men, Women) 
 * Lacrosse (Men) 
 * Soccer (Men) 
 * Squash (Men) 
 * Wrestling (Men) 
 Athletics * Fields and Arenas: Barton Hall 
 * Lynah Rink 
 * Schoellkopf Field 
 * Hoy Field 
 * Newman Arena 
 * Charles F. Berman Field 
 * Robert Trent Jones Golf Course 
 * Culture: "Give My Regards to Davy" 
 * Harvard Hockey Rivalry 
 * Touchdown (mascot) 
 * Campuses: Central 
 * West 
 * North 
 * Tech 
 * Buildings: Andrew Dickson White House 
 * Arboretum 
 * Art Museum 
 * Bailey Hall 
 * Balch Hall 
 * Barnes Hall 
 * Botanic Gardens 
 * Boyce Thompson Institute 
 * Bradfield Hall 
 * Brain Collection 
 * Caldwell Hall 
 * Computing and Communications Center 
 * Comstock Hall 
 * Dairy 
 Campus * Deke 
 * Fernow Hall 
 * Llenroc 
 * Libraries 
 * Observatory 
 * Hartung–Boothroyd Observatory 
 * McGraw Tower 
 * Morrill Hall 
 * Ornithology Lab 
 * Press 
 * Rice Hall 
 * Risley Residential College 
 * Sage Chapel 
 * Sage Hall 
 * Statue of Ezra Cornell 
 * Synchrotron 
 * Telluride House 
 * Theory Center 
 * Willard Straight Hall 
 * Ezra Cornell 
 * Andrew Dickson White 
 History * Morrill Land Grant Act of 1862 
 * Literary societies 
 + Irving Literary Society 
 * Cornell gorge suicides 
 * Chimes 
 * Cornell Club of New York 
 Cornelliana * Dragon Day 
 * Far Above Cayuga's Waters 
 * Slope Day 
 * Songs 
 * Presidents 
 People * Alumni 
 * Faculty 
 * Trustees 
 * Daily Sun 
 * Cornell Chronicle 
 * Cornell Review 
 * Journal of Empirical Legal Studies 
 * International Law Journal 
 * Cornell Lunatic 
 * Kitsch Magazine 
 * International Affairs Review 
 * Cornell Policy Review 
 Publications * Journal of Law and Public Policy 
 * Law Review 
 * Legal Information Institute 
 + Oyez Project 
 * Administrative Science Quarterly 
 * ILR Review 
 * Diacritics 
 * Epoch 
 * New German Critique 
 * The Philosophical Review 
 * arXiv 
 * All Sports Competition 
 * Fraternities and sororities 
 + Aleph Samach 
 + Quill and Dagger 
 + Sphinx Head 
 Student life * Glee Club 
 * Chorus 
 * Hotel Ezra Cornell 
 * Pep Band 
 * Marching Band 
 * WVBR-FM 
 * Cornell Catholic Community 
 
 Retrieved from "https://en.wikipedia.org/w/index.php?title=ArXiv&oldid=1135704930"
 Categories:
 * 1991 establishments in New Mexico
 * American digital libraries
 * Cornell University
 * Eprint archives
 * Internet properties established in 1991
 * Open science
 * Open-access archives
 * Physics websites
 Hidden categories:
 * Articles with short description
 * Short description is different from Wikidata
 * Use mdy dates from May 2016
 * Commons category link is on Wikidata
 * This page was last edited on 26 January 2023, at 09:28 (UTC) .
 * Text is available under the Creative Commons Attribution-ShareAlike License 3.0; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.
 * Privacy policy
 * About Wikipedia
 * Disclaimers
 * Contact Wikipedia
 * Mobile view
 * Developers
 * Statistics
 * Cookie statement
 * 
 * 
From Wikipedia, the free encyclopedia
	 
 arXiv                Type of site  Science   Available in English   Owner  Cornell University    Created by  Paul Ginsparg    URL  arxiv .org    Commercial No   Launched August 14, 1991; 31 years ago   Current status Online    ISSN   2331-8422     OCLC number  228652809     
 arXiv (pronounced "archive"—the X represents the Greek letter chi ⟨χ⟩) [1]  is an open-access repository of electronic preprints and postprints (known as e-prints) approved for posting after moderation, but not peer review. It consists of scientific papers in the fields of mathematics, physics, astronomy, electrical engineering, computer science, quantitative biology, statistics, mathematical finance and economics, which can be accessed online. In many fields of mathematics and physics, almost all scientific papers are self-archived on the arXiv repository before publication in a peer-reviewed journal. Some publishers also grant permission for authors to archive the peer-reviewed postprint. Begun on August 14, 1991, arXiv.org passed the half-million-article milestone on October 3, 2008, [2]   [3]  and had hit a million by the end of 2014. [4]   [5]  
As of April 2021, the submission rate is about 16,000 articles per month. [6] 
History[edit]
      A screenshot of the arXiv taken in 1994, [7]  using the browser NCSA Mosaic. At the time, HTML forms were a new technology.  
      ArXiv's yearly submission rate growth over 30 years since its beginning with topics labelled by the standard abbreviations used on arxiv.org [8]    
arXiv was made possible by the compact TeX file format, which allowed scientific papers to be easily transmitted over the Internet and rendered client-side. [9]  Around 1990, Joanne Cohn began emailing physics preprints to colleagues as TeX files, but the number of papers being sent soon filled mailboxes to capacity. [10]  Paul Ginsparg recognized the need for central storage, and in August 1991 he created a central repository mailbox stored at the Los Alamos National Laboratory (LANL) which could be accessed from any computer. [11]  Additional modes of access were soon added: FTP in 1991, Gopher in 1992, and the World Wide Web in 1993. [5]   [12]  The term e-print was quickly adopted to describe the articles.
 It began as a physics archive, called the LANL preprint archive, but soon expanded to include astronomy, mathematics, computer science, quantitative biology and, most recently, statistics. Its original domain name was xxx.lanl.gov. Due to LANL's lack of interest in the rapidly expanding technology, in 2001 Ginsparg changed institutions to Cornell University and changed the name of the repository to arXiv.org. [13]  It is now hosted principally by Cornell, with five mirrors around the world. [14] 
 ArXiv was an early adopter and promoter of preprints. [15]  Its success in sharing preprints was one of the precipitating factors that led to the later movement in scientific publishing known as open access. [15]  Mathematicians and scientists regularly upload their papers to arXiv.org for worldwide access [16]  and sometimes for reviews before they are published in peer-reviewed journals. Ginsparg was awarded a MacArthur Fellowship in 2002 for his establishment of arXiv. [17]  
The annual budget for arXiv was approximately $826,000 for 2013 to 2017, funded jointly by Cornell University Library, the Simons Foundation (in both gift and challenge grant forms) and annual fee income from member institutions. [18]  This model arose in 2010, when Cornell sought to broaden the financial funding of the project by asking institutions to make annual voluntary contributions based on the amount of download usage by each institution. Each member institution pledges a five-year funding commitment to support arXiv. Based on institutional usage ranking, the annual fees are set in four tiers from $1,000 to $4,400. Cornell's goal is to raise at least $504,000 per year through membership fees generated by approximately 220 institutions. [19] 
 In September 2011, Cornell University Library took overall administrative and financial responsibility for arXiv's operation and development. Ginsparg was quoted in the  Chronicle of Higher Education  as saying it "was supposed to be a three-hour tour, not a life sentence". [20]  However, Ginsparg remains on the arXiv's Scientific Advisory Board and its Physics Advisory Committee. [21]   [22] 
Moderation process and endorsement[edit]
Although arXiv is not peer reviewed, a collection of moderators for each area review the submissions; they may recategorize any that are deemed off-topic, [23]  or reject submissions that are not scientific papers, or sometimes for undisclosed reasons. [24]  The lists of moderators for many sections of arXiv are publicly available, [25]  but moderators for most of the physics sections remain unlisted.
 Additionally, an "endorsement" system was introduced in 2004 as part of an effort to ensure content is relevant and of interest to current research in the specified disciplines. [26]  Under the system, for categories that use it, an author must be endorsed by an established arXiv author before being allowed to submit papers to those categories. Endorsers are not asked to review the paper for errors, but to check whether the paper is appropriate for the intended subject area. [23]  New authors from recognized academic institutions generally receive automatic endorsement, which in practice means that they do not need to deal with the endorsement system at all. However, the endorsement system has attracted criticism for allegedly restricting scientific inquiry. [27]   [28] 
 A majority of the e-prints are also submitted to journals for publication, but some work, including some very influential papers, remain purely as e-prints and are never published in a peer-reviewed journal. A well-known example of the latter is an outline of a proof of Thurston's geometrization conjecture, including the Poincaré conjecture as a particular case, uploaded by Grigori Perelman in November 2002. [29]  Perelman appears content to forgo the traditional peer-reviewed journal process, stating: "If anybody is interested in my way of solving the problem, it's all there [on the arXiv] – let them go and read about it". [30]  Despite this non-traditional method of publication, other mathematicians recognized this work by offering the Fields Medal and Clay Mathematics Millennium Prizes to Perelman, both of which he refused. [31] 
 While arXiv does contain some dubious e-prints, such as those claiming to refute famous theorems or proving famous conjectures such as Fermat's Last Theorem using only high-school mathematics, a 2002 article which appeared in  Notices of the American Mathematical Society  described those as "surprisingly rare". [32]  arXiv generally re-classifies these works, e.g. in "General mathematics", rather than deleting them; [33]  however, some authors have voiced concern over the lack of transparency in the arXiv screening process. [24] 
Submission formats[edit]
Papers can be submitted in any of several formats, including LaTeX, and PDF printed from a word processor other than TeX or LaTeX. The submission is rejected by the arXiv software if generating the final PDF file fails, if any image file is too large, or if the total size of the submission is too large. arXiv now allows one to store and modify an incomplete submission, and only finalize the submission when ready. The time stamp on the article is set when the submission is finalized.
Access[edit]
      A screenshot of viewing a paper's abstract on arxiv.org in 2021  
The standard access route is through the arXiv.org website or one of several mirrors. Other interfaces and access routes have also been created by other un-associated organisations.
  Metadata for arXiv is made available through OAI-PMH, the standard for open access repositories. [34]  Content is therefore indexed in all major consumers of such data, such as BASE, CORE and Unpaywall. As of 2020, the Unpaywall dump links over 500,000 arxiv URLs as the open access version of a work found in CrossRef data from the publishers, making arXiv a top 10 global host of green open access.
 Finally, researchers can select sub-fields and receive daily e-mailings or RSS feeds of all submissions in them.
Copyright status of files[edit]
Files on arXiv can have a number of different copyright statuses: [35] 
 Some are public domain, in which case they will have a statement saying so.
Some are available under either the Creative Commons 4.0 Attribution-ShareAlike license or the Creative Commons 4.0 Attribution-Noncommercial-ShareAlike license.
Some are copyright to the publisher, but the author has the right to distribute them and has given arXiv a non-exclusive irrevocable license to distribute them.
Most are copyright to the author, and arXiv has only a non-exclusive irrevocable license to distribute them. 
See also[edit]
  List of academic databases and search engines 
 List of academic journals by preprint policy 
 List of preprint repositories  
Citations[edit]
 
  ^  Steele, Bill (Fall 2012). "Library-managed 'arXiv' spreads scientific advances rapidly and worldwide". Ezra. Ithaca, New York: Cornell University. p. 9. OCLC 263846378. Archived from the original on January 11, 2015. Pronounce it 'archive'. The X represents the Greek letter chi [ χ ]. 
  ^  Ginsparg, Paul (2011). "It was twenty years ago today ...". arXiv:1108.2700 [cs.DL].
  ^   "Online Scientific Repository Hits Milestone: With 500,000 Articles, arXiv Established as Vital Library Resource". News.library.cornell.edu. October 3, 2008. Retrieved July 21, 2013.
  ^  Vence, Tracy (December 29, 2014), "One Million Preprints and Counting: A conversation with arXiv founder Paul Ginsparg",  The Scientist  
^    a       b    Staff (January 13, 2015). "In the News: Open Access Journals".  Drug Discovery &amp; Development .
  ^   "arXiv monthly submission rate statistics". Arxiv.org. Retrieved April 3, 2021.
  ^   "Image" (GIF). Cs.cornell.edu. Retrieved March 9, 2019.
  ^  Ginsparg, Paul (August 4, 2021). "Lessons from arXiv's 30 years of information sharing". Nature Reviews Physics. 3 (9): 602–603. doi:10.1038/s42254-021-00360-z. ISSN 2522-5820. PMC 8335983. PMID 34377944.
  ^  O'Connell, Heath (2002). "Physicists Thriving with Paperless Publishing" (PDF). High Energy Physics Libraries Webzine. 6 (6): 3. arXiv:physics/0007040. Bibcode:2000physics...7040O. Archived (PDF) from the original on October 9, 2022.
  ^  Feder, Toni (November 8, 2021). "Joanne Cohn and the email list that led to arXiv".  Physics Today . 2021 (4): 1108a. Bibcode:2021PhT..2021d1108.. doi:10.1063/PT.6.4.20211108a. S2CID 244015728.
  ^  Feder, Toni (November 8, 2021). "Joanne Cohn and the email list that led to arXiv".  Physics Today . 2021 (4): 1108a. Bibcode:2021PhT..2021d1108.. doi:10.1063/PT.6.4.20211108a. S2CID 244015728.
  ^  Ginsparg, Paul (October 1, 2008). "The global-village pioneers".  Physics World . Retrieved October 10, 2020.
  ^  Butler, Declan (July 5, 2001). "Los Alamos Loses Physics Archive as Preprint Pioneer Heads East". Nature. 412 (6842): 3–4. Bibcode:2001Natur.412....3B. doi:10.1038/35083708. PMID 11452262. S2CID 1527860.
  ^   "arXiv mirror sites". arXiv. Archived from the original on March 16, 2020. Retrieved April 6, 2020.
^    a       b     "Celebrating 30 Years of arXiv and Its Lasting Legacy on Scientific Advancement". SPARC. October 25, 2021.
  ^  Glanz, James (May 1, 2001). "The World of Science Becomes a Global Village; Archive Opens a New Realm of Research".  The New York Times .
  ^  Bill Steele (September 23, 2002). "Cornell professor Paul Ginsparg, science communication rebel, named a MacArthur Foundation fellow; three other alumni also receive 'genius award' fellowships".
  ^   "CORNELL UNIVERSITY LIBRARY ARXIV FINANCIAL PROJECTIONS FOR 2013-2017" (PDF). Confluence.cornell.edu. March 28, 2012. Retrieved February 26, 2017.
  ^   "arXiv Member Institutions (2021) - arXiv about - Our Members". arXiv.org. Retrieved December 27, 2021.
  ^  Fischman, Joah (August 10, 2011). "The First Free Research-Sharing Site, arXiv, Turns 20 With an Uncertain Future". Chronicle of Higher Education. Retrieved August 12, 2011.
  ^   "arXiv Scientific Advisory Board | arXiv e-print repository". arxiv.org. Retrieved October 10, 2020.
  ^   "About the Physics Archive | arXiv e-print repository". arxiv.org. Retrieved October 10, 2020.
^    a       b    McKinney, Michelle (2011), "ArXiv.org", Reference Reviews, 25 (7): 35–36, doi:10.1108/09504121111168622 
^    a       b    Merali, Zeeya (January 29, 2016). "ArXiv rejections lead to spat over screening process".  Nature . doi:10.1038/nature.2016.19267. S2CID 189061969. Retrieved December 14, 2017.
  ^  Computing Research Repository Subject Areas and Moderators; Statistics archive; Quantitative Biology archive; Physics archive
  ^  Ginsparg, Paul (2006), "As we may read", Journal of Neuroscience, 26 (38): 9606–9608, doi:10.1523/JNEUROSCI.3161-06.2006, PMC 6674456, PMID 16988030 
  ^  Greechie, Richard; Pulmannova, Sylvia; Svozil, Karl (July 2005), "Preface to the Proceedings of Quantum Structures 2002", International Journal of Theoretical Physics, 44 (7): 691–692, Bibcode:2005IJTP...44..691G, doi:10.1007/s10773-005-7053-z, S2CID 121442106, The new endorsement system may contribute to an effective barrier, a digital divide 
  ^  Josephson, Brian (February 23, 2005). "Vital resource should be open to all physicists". Nature. 433 (7028): 800. Bibcode:2005Natur.433..800J. doi:10.1038/433800a. PMID 15729314.
  ^  Perelman, Grisha (November 11, 2002). "The entropy formula for the Ricci flow and its geometric applications". arXiv:math.DG/0211159.
  ^  Lobastova, Nadejda; Hirst, Michael (August 21, 2006). "Maths genius living in poverty".  Sydney Morning Herald .
  ^  Kaufman, Marc (July 2, 2010), "Russian mathematician wins $1 million prize, but he appears to be happy with $0", Washington Post 
  ^  Jackson, Allyn (2002). "From Preprints to E-prints: The Rise of Electronic Preprint Servers in Mathematics" (PDF). Notices of the American Mathematical Society. 49 (1): 23–32.
  ^  Ginsparg, Paul (August 2011). "ArXiv at 20". Nature. 476 (7359): 145–147. Bibcode:2011Natur.476..145G. doi:10.1038/476145a. ISSN 0028-0836. PMID 21833066. S2CID 4421407.
  ^   "Open Archives Initiative (OAI)". arxiv.org. Retrieved April 25, 2020.
  ^   "arXiv License Information". Arxiv.org. Retrieved July 21, 2013.
 
General and cited sources[edit]
  Butler, Declan (2003). "Biologists Join Physics Preprint Club".  Nature . 425 (6958): 548. Bibcode:2003Natur.425..548B. doi:10.1038/425548b. PMID 14534551. S2CID 4374168. 
 Choi, Charles Q. (2003). "Biology's New Online Archive".  The Scientist . Archived from the original on March 13, 2005. Retrieved June 21, 2005. 
 Giles, Jim (2003). "Preprint Server Seeks Way to Halt Plagiarists".  Nature . 426 (6962): 7. Bibcode:2003Natur.426Q...7G. doi:10.1038/426007a. PMID 14603280. S2CID 29003994. 
 Ginsparg, Paul (1997). "Winners and Losers in the Global Research Village". The Serials Librarian. 30 (3–4): 83–95. doi:10.1300/J123v30n03_13. 
 Halpern, Joseph Y. (1998). "A Computing Research Repository".  D-Lib Magazine . 4 (11). doi:10.1045/november98-halpern. 
 Halpern, Joseph Y. (2000). "CoRR: A Computing Research Repository".  Journal of Computer Documentation . 24 (2): 41–48. arXiv:cs.DL/0005003. Bibcode:2000cs........5003H. doi:10.1145/337271.337274. S2CID 5453868. 
 Luce, Richard E. (2001). "E-Prints Intersect the Digital Library: Inside the Los Alamos arXiv". Issues in Science and Technology Librarianship (29). doi:10.5062/F44B2Z95. 
 McKiernan, Gerry (2000). "ArXiv.org: The los Alamos National Laboratory e‐print server" (PDF).  International Journal on Grey Literature . 1 (3): 127–138. doi:10.1108/14666180010345564. Archived from the original (PDF) on May 5, 2005. 
 Pinfield, Stephen (2001). "How Do Physicists Use an E-Print Archive? Implications for Institutional E-Print Services".  D-Lib Magazine . 7 (12). doi:10.1045/december2001-pinfield. 
 Quigley, Brian (2000). "Physics Databases and the Los Alamos e-Print Archive".  EContent . 23 (5): 22–26. 
 Taubes, Gary (1993). "Publication by Electronic Mail Takes Physics by Storm".  Science . 259 (5099): 1246–1248. Bibcode:1993Sci...259.1246T. doi:10.1126/science.259.5099.1246. PMID 17732237. 
 Warner, Simeon (2001). "Open Archives Initiative protocol development and implementation at arXiv". arXiv:cs/0101027. 
 "What Is q-bio?". Open Access Now. 2004.  
External links[edit]
 
Wikimedia Commons has media related to ArXiv.org. 
  Official website
arXiv - Wikipedia
https://en.wikipedia.org/wiki/ArXiv