Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks. Transformer networks (Vaswani et al., 2017) are sequence models that rely on the attention mechanism (Bahdanau et al., 2015) to capture long term dependencies. Since their introduction in the context of machine translation, they have been applied to many natural language processing tasks, such as language modeling (Al-Rfou et al., 2019) or sentence representation (Devlin et al., 2019) . On most of them, they are now surpassing the former state-of-the-art models based on recurrent (Hochreiter & Schmidhuber, 1997) or convolutional networks (Dauphin et al., 2017) . At their core, transformers use a self-attention layer that forms a representation of the current input by gathering the most relevant information from its context. This layer is repeated along the network depth, allowing for information to flow for long distances and to form rich sequence representations. The self-attention mechanism is often considered as the key component of their success and many have worked on improving transformers by increasing the size of the context captured by those layers (Wu et al., 2019; Dai et al., 2019; Sukhbaatar et al., 2019) . However, self-attention layers are not the only component of transformer networks and they do not explain the effectiveness of transformers by themselves. Each of these layers is followed by a feedforward layer. These feedforward layers contain most of the parameters of the model. This suggests that their role is probably as important as the self-attention mechanism. In fact, the transformer layer, i.e., the sequence of self-attention and feedforward sublayers, should be regarded as a single mechanism that gathers information from the context and transforms it into a rich representation. Having such two different layer types of at the core makes Transformer models harder to analyse and understand. In particular, there are not many works exploring the properties of feedforward layers. In this work, we simplify the transformer architecture by revisiting its mechanism, while keeping its properties. We introduce a new layer that merges the self-attention and feedforward sublayers into a single unified attention layer, as illustrated in Figure 1 . As opposed to the two-step mechanism of the transformer layer, it directly builds its representation from the context and a persistent memory block without going through a feedforward transformation. The additional persistent memory block stores, in the form of key-value vectors, information that does not depend on the context. In terms of parameters, these persistent key-value vectors replace the feedforward sublayer. This modification dramatically simplifies the structure of the network with no loss of performance. We evaluate the resulting architecture on standard word level and character level language modeling benchmarks and report performances that are competitive with transformers. Figure 1: On the left panel, the standard transformer layer is composed of a self-attention sublayer followed by a feedforward sublayer. On the right panel, our all-attention layer merges the weights of the feedforward sublayer with the self-attention sublayer. We represent both models in the case of a single head, but in the general case, both the self-attention sublayer and our all-attention layers have multiple heads. In this paper, we propose a novel attention layer that presents a unified mechanism to aggregate general and contextual information. It extends the self-attention layer of a transformer with a set of persistent vectors that are capable of storing information that is complementary to the short term information in contexts. We also show that these persistent vectors can replace the feedforward layers in a transformer network with no loss of performance. We think that this simplified layer can help better understand how information is processed and stored in transformer-like sequence models. A ATTENTION MAPS Figure 3 : Sample attention maps from our model that trained on the WikiText-103 dataset. The 4 plots correspond to 4 different attention heads in the model. The Y -axis is different samples from a short sequence, and the X-axis shows all the vectors in the attention. The first 2048 vectors come from the context, and the remaining 2048 are persistent vectors. In the top 2 heads, few persistent vectors are dominating the attention, although the 2nd head has some attention weights in the context part as well. The 3rd head has more diverse activations on the persistent vectors, while also attending to very recent context. The last head is mostly attending to about last 500 tokens in the context, but there are some activations in the persistent vectors.
A novel attention layer that combines self-attention and feed-forward sublayers of Transformer networks. This paper proposes a modification to the Transformer model by incorporating attention over "persistent" memory vectors into the self-attention layer, resulting in performance on par with existing models while using fewer parameters.