Gated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks
(RNNs) to reach and even surpass the performance of Transformers on certain
sequence modeling tasks. These modern RNNs feature a prominent design pattern:
linear recurrent layers interconnected by feedforward paths with multiplicative
gating. Here, we show how RNNs equipped with these two design elements can
exactly implement (linear) self-attention, the main building block of
Transformers. By reverse-engineering a set of trained RNNs, we find that
gradient descent in practice discovers our construction. In particular, we
examine RNNs trained to solve simple in-context learning tasks on which
Transformers are known to excel and find that gradient descent instills in our
RNNs the same attention-based in-context learning algorithm used by
Transformers. Our findings highlight the importance of multiplicative
interactions in neural networks and suggest that certain RNNs might be
unexpectedly implementing attention under the hood.
Nicolas Zucchet, Seijin Kobayashi, Yassir Akram, Johannes von Oswald, Maxime Larcher, Angelika Steger, João Sacramento
# Gated recurrent neural networks

###### Abstract

Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These modern RNNs feature a prominent design pattern: linear recurrent layers interconnected by feedforward paths with multiplicative gating. Here, we show how RNNs equipped with these two design elements can exactly implement (linear) self-attention, the main building block of Transformers. By reverse-engineering a set of trained RNNs, we find that gradient descent in practice discovers our construction. In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers. Our findings highlight the importance of multiplicative interactions in neural networks and suggest that certain RNNs might be unexpectedly implementing attention under the hood.

Attention-based neural networks, most notably Transformers (Vaswani et al., 2017), have rapidly become the state-of-the-art deep learning architecture, replacing traditional models such as multi-layer perceptrons, convolutional neural networks, and recurrent neural networks (RNNs). This is particularly true in the realm of sequence modeling, where once-dominating RNNs such as the long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) model and the related gated recurrent unit (GRU; Cho et al., 2014) have been mostly replaced by Transformers.

Nevertheless, RNNs remain actively researched for various reasons, such as their value as models in neuroscience (Dayan & Abbott, 2001), or simply out of genuine interest in their rich properties as a dynamical system and unconventional computer (Jaeger et al., 2023). Perhaps most importantly for applications, RNNs are able to perform inference for arbitrarily long sequences at a constant memory cost, unlike models based on conventional softmax-attention layers (Bahdanau et al., 2015). This ongoing research has led to a wave of recent developments. On the one hand, new deep linear RNN architectures (Gu et al., 2022; Orvieto et al., 2023b) have been shown to significantly outperform Transformers on challenging long-sequence tasks (e.g., Tay et al., 2020). On the other hand, efficient linearized attention models have been developed, whose forward pass can be executed in an RNN-like fashion at a constant inference memory cost (Tsai et al., 2019; Katharopoulos et al., 2020; Choromanski et al., 2021; Schlag et al., 2021).

We present a unifying perspective on these two seemingly unrelated lines of work by providing a set of parameters under which gated RNNs become equivalent to any linearized self-attention, without requiring infinite number of neurons or invoking a universality argument. Crucially, our construction makes use of gated linear units (GLUs; Dauphin et al., 2017), which are ostensibly featured in recent deep linear RNN models. Turning to LSTMs and GRUs, which also include multiplicative gating interactions, we find somewhat surprisingly that our results extend only to LSTMs. Moreover, the LSTM construction we provide hints that the inductive bias towards attention-compatible configurations might be weaker for this architecture than for deep gated linear RNNs.

We then demonstrate that GLU-equipped RNNs, but not LSTMs and GRUs, can effectively implement our construction once trained, thus behaving as attention layers. Moreover, we find that such
GLU-equipped RNNs trained to solve linear regression tasks acquire an attention-based in-context learning algorithm. Incidentally, it has been shown that the very same algorithm is typically used by Transformers trained on this problem class (von Oswald et al., 2023; Mahankali et al., 2023; Ahn et al., 2023; Zhang et al., 2023). Our results thus challenge the standard view of RNNs and Transformers as two mutually exclusive model classes and suggest that, through learning, RNNs with multiplicative interactions may end up encoding attention-based algorithms disguised in their weights.

## 1 Background

### Linear self-attention

We study causally-masked linear self-attention layers that process input sequences \((x_{t})_{t}\) with \(x_{t}\in\mathbb{R}^{d}\) as follows:

\[y_{t}=\left(\sum_{t^{\prime}\leq t}(W_{V}x_{t^{\prime}})(W_{K}x_{t^{\prime}})^ {\top}\right)(W_{Q}x_{t}) \tag{1}\]

In the previous equation, \(W_{V}\in\mathbb{R}^{d\times d}\) is the value matrix, \(W_{K}\in\mathbb{R}^{d\times d}\) the key matrix and \(W_{Q}\in\mathbb{R}^{d\times d}\) the query matrix. Note that we use square matrices throughout our paper for simplicity, but our findings hold for rectangular ones. As usually done, we call \(v_{t}:=W_{V}x_{t}\), \(k_{t}:=W_{K}x_{t}\) and \(q_{t}:=W_{Q}x_{t}\) the values, keys and queries. The output vector \(y_{t}\) has the same dimension as the input, that is \(d\). Such linear self-attention layers can be understood as a linearized version of the softmax attention mechanism (Bahdanau et al., 2015) in use within Transformers (Vaswani et al., 2017). Attention layers commonly combine different attention heads; we focus on a single one here for simplicity.

In a linear self-attention layer, information about the past is stored in an effective weight matrix \(W_{t}^{\text{ff}}:=\sum_{t^{\prime}}v_{t^{\prime}}k_{t^{\prime}}^{\top}\) that will later be used to process the current query \(q_{t}\) through \(y_{t}=W_{t}^{\text{ff}}q_{t}\). At every timestep, \(W_{t}^{\text{ff}}\) is updated through the rule \(W_{t}^{\text{ff}}=W_{t-1}^{\text{ff}}+v_{t}k_{t}^{\top}\), which is reminiscent of Hebbian learning (Schmithuber, 1992; Schlag et al., 2021) and leads to faster inference time (Katharopoulos et al., 2020; Choromanski et al., 2021; Shen et al., 2021; Peng et al., 2021) than softmax self-attention.

### Gated recurrent neural networks

In this paper, we focus our analysis on a simplified class of gated diagonal linear recurrent neural networks. They implement linear input and output gating that multiplies a linear transformation \(W_{\text{x}}x_{t}\) of the input with a linear gate \(W_{\text{m}}x_{t}\): \(g(x_{t})=W_{\text{m}}x_{t}\odot W_{\text{x}}x_{t}\). Here, \(\odot\) is the element-wise product. The class of gated networks we consider satisfies

\[h_{t+1}=\lambda\odot h_{t}+g^{\text{in}}(x_{t}),\ \ \ y_{t}=Dg^{\text{out}}(h_{t}). \tag{2}\]

In the previous equation, \(\lambda\) is a real vector, \(x_{t}\) is the input to the recurrent layer, \(h_{t}\) the hidden state, \(g^{\text{in}}\) the input gating, \(g^{\text{out}}\) the output gating, and \(D\) a linear readout. This simplified class makes connecting to attention easier while employing similar computational mechanisms as standard gated RNNs architectures.

This class is tightly linked to recent deep linear RNN architectures and shares most of its computational mechanisms with them. While linear diagonal recurrence might be seen as a very strong inductive bias, many of the recent powerful deep linear RNN models adopt a similar bias (Gupta et al., 2022; Smith et al., 2023), and it has been shown to facilitate gradient-based learning (Orvieto et al., 2023; Zucchet et al., 2023). Those architectures use complex-valued hidden states in the recurrence; we only use its real part here. Some of those works employ a GLU (Dauphin et al., 2017) after each recurrent layer, \(\operatorname{GLU}(x)=\sigma(W_{\text{m}}x_{t})\odot W_{\text{x}}x_{t}\) with \(\sigma\) the sigmoid function. The gating mechanism we consider can thus be interpreted as a linearized GLU. Finally, we can recover (2) by stacking two layers: the GLU in the first layer acts as our input gating, and the one in the second as output gating. We include a more detailed comparison in Appendix A. In the rest of the paper, we will use the LRU layer (Orvieto et al., 2023) as the representative of the deep linear RNN architectures because of its proximity with (2).
LSTMs can operate in the regime of Equation 2, but this requires more adaptation. First, the recurrent processing of these units is nonlinear and is more involved than a simple matrix multiplication followed by a nonlinearity. Second, gating occurs in different parts of the computation and depends on additional variables. We compare in more details this architecture and the one of Equation 2 in Appendix A, showing that LSTMs can implement (2) when stacking two layers on top of each other. We additionally show that GRUs cannot do so.

## 2 Theoretical construction

As highlighted in the previous section, our class of gated RNNs and linear self-attention have very different ways of storing past information and using it to modify the feedforward processing of the current input. The previous state \(h_{t}\) acts through a bias term \(\lambda\odot h_{t}\) that is added to the current input \(g^{\text{in}}(x_{t})\) in gated RNNs, whereas the linear self-attention recurrent state \(W_{t}^{\text{ff}}\) modifies the weights of the feedforward pathway. We reconcile these two mismatched views of neural computation in the following by showing that gated RNNs can implement linear self-attention.

We demonstrate in this section how a gated recurrent layer followed by a linear readout as in Equation 2 can implement any linear self-attention layer through a constructive proof. In particular, our construction only requires a finite number of neurons, therefore providing a much stronger equivalence result than more general universality of linear recurrent networks theorems (Grigoryeva and Ortega, 2018; Orvieto et al., 2023a).

### Key ideas

Our construction comprises three main components: Firstly, the input gating \(g^{\text{in}}\) is responsible for generating the element-wise products between the keys and values, as well as the queries. Then, recurrent units associated with key-values accumulate their inputs with \(\lambda=1\), whereas those receiving queries as inputs return the current value of the query, hence \(\lambda=0\). Lastly, the output gating \(g^{\text{out}}\) and the final readout layer \(D\) are in charge of multiplying the flattened key-value matrix

Figure 1: An example of a diagonal linear gated recurrent neural network that implements the same function as a linear self-attention layer with parameters \((W_{V},W_{K},W_{Q})\) and input dimension \(d\). Inputs are processed from the bottom to the top. We do not use biases so we append 1 to the input vector \(x_{t}\) to be able to send queries to the recurrent neurons. We use \(\operatorname{repeat}(A,n)\) to denote that the matrix \(A\) is repeated \(n\) times on the row axis and \(W_{V,i}\) is the \(i\)-th row of the \(W_{V}\) matrix. The bars in the matrices highlight the different kinds of inputs/outputs. Digits in matrices denote column vectors appropriately sized. The readout matrix \(D\) appropriately sums the element-wise products between key-values and queries computed after the output gating \(g^{\text{out}}\).
with the query vector. We visually illustrate our construction and provide a set of weights for which the functional equivalence holds in Figure 1. Crucially, the key-values in a linear self-attention layer are the sum of degree two polynomials of each previous input. Input gating mechanism and perfect memory units (\(\lambda=1\)) are needed to replicate this behavior within a gated recurrent layer. Similarly, output gating is required to multiply key-values with the queries.

### On the number of neurons required by the construction

The simple construction we illustrate in Figure 1 requires \(d^{2}\) hidden neurons for the key-value elements and \(d\) additional ones to represent the query. It is, however, possible to make the construction more compact by leveraging two additional insights: First, when the key and value matrices are equal, the key-value matrix is symmetric and, therefore, requires \(d(d+1)/2\) elements to be represented. Second, any combination of key and value matrices for which \((W_{K}^{\top}W_{Q})\) is fixed leads to the same function in the linear self-attention layer. This implies that, when the value matrix is invertible, we can reduce the number of hidden neurons storing key-values to \(d(d+1)/2\) by making the key matrix to be equal to \(W_{V}\), and changing the query matrix to be \(W_{V}^{\top}W_{K}^{\top}W_{Q}\).

Overall, the output gating requires \(\mathcal{O}(d^{2})\) input and output entries for the gated RNN to match a linear self-attention layer. The RNN thus requires \(\mathcal{O}(d^{4})\) parameters in total, with a lot of redundancy, significantly more than the \(3d^{2}\) parameters of the linear self-attention layer. It comes as no surprise that numerous equivalent configurations exist within the gated RNN we study. For instance, linear gating is invariant under permutations of rows between its two matrices and under multiplication-division of these two rows by a constant. Left-multiplying \(W_{Q}\) in the input gating by any invertible matrix \(P\), and subsequently reading out the hidden neurons with \(\lambda=0\) through \(\mathrm{repeat}(P^{-1},d)\), also does not alter the network's output. Several other invariances exist, making exact weight retrieval nearly impossible.

### Implications for other classes of gated RNNs

We conclude this section by commenting on whether similar insights hold for other gated RNNs architectures. The LRU architecture is close to (2) but only has output gating. Stacking two LRU layers on top of each other enables the output gating of the first layer to act as the input gating for the second layer, and, therefore, implement linear self-attention. As noted in Section 1.2, LSTMs and GRUs are further away from our simplified gated RNN model. However, one single LSTM layer can implement linear self-attention, but stacked GRU layers cannot do so. We detailed those considerations in Appendix A.

## 3 Gated RNNs learn to mimic attention

We now demonstrate that gated RNNs learn to implement linear self-attention and comprehend how they do so. In this section, a student RNN is tasked to reproduce the output of a linear self-attention layer. Appendix B contains detailed descriptions of all experiments performed in this section. Importantly, each sequence is only presented once to the network.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Loss & Score KV & Score Q & Polynomial distance \\ \hline \(4.97\times 10^{-8}\) & \(4.52\times 10^{-8}\) & \(2.06\times 10^{-10}\) & \(3.73\times 10^{-4}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Gated RNNs implement attention. The KV and Q scores are equal to one minus the \(R^{2}\) score of the linear regression that predicts key-values and queries from resp. the perfect memory neurons and perfect forget neurons. The polynomial distance is the L2 distance between the coefficients of the degree-4 polynomial that describes the instantaneous processing of the (optimal) linear self-attention layer and the trained RNN.
### Teacher identification

In our first experiment, we train a student RNN (\(|x|=4\), \(|h|=100\) and \(|y|=4\)) to emulate the behavior of a linear self-attention layer with weights sampled from a normal distribution. The low training loss, reported in Table 1, highlights that the student's in-distribution behavior aligns with the teacher's. However, this is insufficient to establish that the student implements the same function as the teacher. The strategy we adopt to show functional equivalence is as follows: First, we observe that only perfect memory neurons (\(\lambda=1\)) and perfect forget neurons (\(\lambda=0\)) influence the network output. Additionally, each of these groups of neurons receives all the information needed to linearly reconstruct resp. the key-values and the queries from the input. Finally, we show that the output gating and the decoder matrix accurately multiply accumulated key-values with current queries.

After the learning process, a significant proportion of the weights in the input and output gating and the readout becomes zeros. Consequently, we can eliminate neurons with input or output weights that are entirely zeros, thereby preserving the network's function. By doing so, we can remove \(86\) out of the \(100\) hidden neurons and \(87\) out of the \(100\) pre-readout neurons. After having permuted rows in the two gating mechanisms and reordered hidden neurons, we plot the resulting weights on Figure 2.A. Consistently with our construction, only recurrent neurons with \(\lambda=0\) or \(\lambda=1\) contribute to the network's output. The key-values neurons receive a polynomial of degree \(2\) without any term of degree \(1\) as the last column of \(W_{\rm m}^{\rm in}\) and \(W_{\rm x}^{\rm in}\) is equal to zero for those units, and the query neurons a polynomial of degree \(1\). The learning process discovers that it can only use \(d(d+1)/2=10\) neurons to store key-values, similar to our compact construction. We show in Table 1 that it is possible to linearly reconstruct the key-values from those \(10\) neurons perfectly, as well as the queries from the \(4\) query neurons. By combining this information with the fact that the \(\lambda\)s are zeros and ones, we deduce that the cumulative key-values \(\sum_{t^{\prime}\leq t}v_{t^{\prime}}k_{t^{\prime}}^{\rm in}\) can be obtained linearly from the key-values' hidden neurons, and the instantaneous queries \(q_{t}\) from the query neurons.

Additionally, the output gating combined with the linear readout can multiply the key-values with the queries. Since we have already confirmed that the temporal processing correctly accumulates

Figure 2: Teacher-student experiment (\(d=4\)). **(A)** Weights of the student RNN once post-processed. The predictions yielded by those weights closely match the ones of the linear self-attention teacher. **(B)** Value \(W_{V}\), key \(W_{K}\) and query \(W_{V}\) weight matrices of the teacher considered in this example; all are invertible. **(C)** For each output coordinate \(i\) of the network, the kernels generated by the last 3 lines (11, 12 and 13) of the output gating are linearly combined through the decoding matrix \(D\) are all proportional to the same kernel, which can be generated in a way that is coherent with our construction. In all the weight matrices displayed here, zero entries are shown in grey, blue denotes positive entries, and red negative ones.
key-values, our focus shifts to proving that the instantaneous processing of the gated RNN matches the one of the attention layer across the entire input domain. Given that both architectures solely employ linear combinations and multiplications, their instantaneous processing can be expressed as a polynomial of their input. The one of linear self-attention, \((W_{V}x)(W_{K}x)^{\top}(W_{Q}x)\), corresponds to a polynomial of degree \(3\), whereas the one of the gated RNN, \(g^{\mathrm{out}}(g^{\mathrm{in}}(x))\), corresponds to one of degree \(4\). By comparing these two polynomials, we can compare their functions beyond the training domain. For every one of the four network outputs, we compute the coefficients of terms of degree \(4\) or lower of their respective polynomials and store this information into a vector. We then calculate the normalized Euclidean distance between these coefficient vectors of the linear self-attention layer and the gated RNN, and report the average over all 4 output units in Table 1. The evidence presented thus far enables us to conclude that the student network has correctly identified the function of the teacher.

While the majority of the weights depicted in Figure 2.A conform to the block structure characteristic of our construction, the final three rows within the output gating matrices deviate from this trend. As shown in Figure 2, these three rows can be combined into a single row matching the desired structure. More details about this manipulation can be found in Appendix B.2.

### Identification requires mild overparametrization

The previous experiment shows that only a few neurons in a network of \(100\) hidden neurons are needed to replicate the behavior of a self-attention layer whose input size is \(d\). We therefore wonder if identification remains possible when decreasing the number of hidden and pre-output gating neurons the student has. We observe that mild overparametrization, around twice as many neurons as the actual number of neurons required, is needed to reach identification. We report the results in Figure 3.A.

### Nonlinearity makes identification harder

We now move from our simplified class of gated RNNs and seek to understand how our findings apply to LSTMs, GRUs, and LRUs. We use the following architecture for those three layers: a linear embedding layer projects the input to a latent representation, we then repeat the recurrent layer once or twice, and finally apply a linear readout. While those layers are often combined with layer normalization, dropout, or skip connections in modern deep learning experiments, we do not include any of those here to stay as close as possible to the teacher's specifications. In an LRU layer, the input/output dimension differs from the number of different neurons; we here set all those dimensions to the same value for a fair comparison with LSTMs and GRUs. We compare these

Figure 3: Gated RNNs learn compressed representation. **(A)** In the teacher-student experiment of Section 3 in which there is no clear structure within the attention mechanism that the RNN can extract, slight overparametrization is needed in order to identify the teacher. **(B)** In the linear regression task of Section 4, the linear attention mechanism needed to solve the task optimally has a sparse structure that the RNN leverages. Identification is thus possible with much smaller networks. The quantity reported in B is the difference between the prediction loss of the RNN and the optimal loss obtained after one step of gradient descent. We use the same input dimension \(d=6\) to make the two plots comparable.
methods to the performance of our simplified gated RNNs, with both diagonal (as in Equation 2) and dense linear recurrent connectivity.

We report the results in Figure 4 for \(d=6\). While diagonal connectivity provides a useful inductive bias to learn how to mimic linear self-attention, it is not absolutely needed as the performance of the dense counterpart of the construction highlights. It is theoretically possible to identify the teacher with one LSTM layer. However, gradient descent does not find such a solution and the performance of LSTMs is close to that of GRUs that cannot implement attention. Motivated by our construction, we slightly change the LRU architecture and add a nonlinear input gating to each layer. We find that it significantly improves its ability to mimic attention. We extensively compare different LRU layers in Appendix B.

## 4 Attention-based in-context learning emerges in trained RNNs

The previous section shows that gated RNNs learn to replicate a given linear self-attention teacher. We now demonstrate that they can find the same solution as linear self-attention when both are learned. To that end, we study an in-context regression task in which the network is shown a few input-output pairs and later has to predict the output value corresponding to an unseen input. Linear self-attention is a particularly beneficial inductive bias for solving this task. When the input-output mapping is linear, von Oswald et al. (2023) have shown that linear self-attention implement one step of gradient descent.

### In-context linear regression

Linear regression consists in estimating the parameters \(W^{*}\in R^{d_{y}\times d_{x}}\) of a linear model \(y=W^{*}x\) from a set of observations \(\{(x_{t},y_{t})\}_{t=1}^{T}\) that satisfy \(y_{t}=W^{*}x_{t}\). The objective consists in finding a parameter \(\hat{W}\) which minimizes the squared error loss \(L(W)=\frac{1}{2T}\sum_{t=1}^{T}\|y_{t}-Wx_{t}\|^{2}\). Given an initial estimate of the parameter \(W_{0}\), one step of gradient descent on \(L\) with learning rate \(T\eta\) yields the weight change

\[\Delta W_{0}=\eta\sum_{t=1}^{T}(y_{t}-W_{0}x_{t})x_{t}^{\top}. \tag{3}\]

In the in-context version of the task, the observations \((x_{t},y_{t})_{1\leq t\leq T}\) are provided one after the other to the network, and later, at time \(T+1\), the network is queried with \((x_{T+1},0)\) and its output regressed against \(y_{T+1}\). Under this setting, von Oswald et al. (2023) showed that if all bias terms are zero, a linear self-attention layer learns to implement one step of gradient descent starting from \(W_{0}=0\) and predict through

\[\hat{y}_{T+1}=(W_{0}+\Delta W_{0})x_{T+1}=\eta\sum_{t=1}^{T}y_{t}x_{t}^{\top} x_{T+1}. \tag{4}\]

Figure 4: Comparison of the validation loss obtained by different gated recurrent networks architectures in **(A)** the teacher-student task of Section 1.2 and **(B)** the in-context linear regression of Section 4. The construction baseline corresponds to the gated RNN of Eq. 2, with diagonal or dense connectivity. We use the default implementation of LSTMs and GRUs, and slightly modify the LRU architecture to reflect our construction better.
In the following, we show that gated RNNs also learn to implement the same algorithm and leverage the sparse structure of the different attention matrices corresponding to gradient descent to learn a more compressed representation than the construction one.

### Gated RNNs learn to implement gradient descent

We now train gated RNNs as in Equation 2 to solve the in-context linear regression task, see Appendix C.1 for more details. We set the number of observations to \(T=12\) and set the input and output dimensions to \(3\) so that \(d=6\). Once learned, the RNN implements one step of gradient descent with optimal learning rate, which is also the optimal solution one layer of linear self-attention can find (Mahankali et al., 2023). Several pieces of evidence back up this claim: the training loss of RNN after training (\(0.0945\)) is almost equal to the one of an optimal step of gradient descent (\(0.0947\)) and the trained RNN implements the same instantaneous function, as the polynomial analysis of Table 2 reveals.

Linear self-attention weights implementing gradient descent have a very specific sparse structure (von Oswald et al., 2023). In particular, many key-values entries are always 0, so the construction contains many dead neurons. This leads us to wonder whether gated RNNs would pick up this additional structure and learn compressed representations. To test that, we vary the gated RNN size and report in Figure 3.B the difference between the final training loss and the loss obtained after one optimal gradient descent step. We observe a similar phase transition than in the teacher-student experiment, this time happening for a much smaller number of neurons than our construction specifies. Gated RNNs thus learn a more compressed representation than the one naively mimicking self-attention. This result provides some hope regarding the poor \(\mathcal{O}(d^{4})\) scaling underlying our construction: in situations that require an attention mechanism with sparse \((W_{V},W_{K},W_{Q})\) matrices, gated RNNs can implement attention with far fewer neurons. A precise understanding of how much compression is possible in practical scenarios requires further investigation.

### Nonlinear gated RNNs are better in-context learners than one step gradient descent

Finally, as a side question we compare the ability to learn in context of the nonlinear gated RNN architectures that are LSTMs, GRUs and LRUs. Although not the main focus of our paper, this allows putting our previous results in perspective. In particular, we are interested in understanding if similarity with attention correlates with in-context learning performance, as attention has been hypothesized to be a key mechanism for in-context learning (Olsson et al., 2022; Garg et al., 2022; von Oswald et al., 2023). We report our comparison results in Figure 4, measuring the loss on weights \(W^{*}\) drawn from a distribution with double the variance of the one used to train the model.

Overall, we find that nonlinearity greatly helps and enables nonlinear gated RNN architectures to outperform one gradient descent step when given enough parameters, suggesting that they implement some more sophisticated mechanism than vanilla gradient descent. Surprisingly, while the GRU is the architecture that is the furthest away from attention, it performs the best in the task. Within the different LRU layers we compare, we find a high correlation between in-context learning abilities and closeness to attention, c.f. Figure 5 in the appendix. In particular, we observe a mas

\begin{table}
\begin{tabular}{c c c} \hline \hline Term & RNN & GD \\ \hline \(x_{1}^{2}y_{1}\) & \(6.81\times 10^{-2}\pm 8.52\times 10^{-5}\) & \(6.76\times 10^{-2}\) \\ \(x_{2}^{2}y_{1}\) & \(6.82\times 10^{-2}\pm 6.40\times 10^{-5}\) & \(6.76\times 10^{-2}\) \\ \(x_{3}^{2}y_{1}\) & \(6.82\times 10^{-2}\pm 5.56\times 10^{-5}\) & \(6.76\times 10^{-2}\) \\ residual norm & \(1.35\times 10^{-3}\pm 1.97\times 10^{-4}\) & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The coefficients of the instantaneous polynomial implemented by the first output unit of a trained RNN on the in-context linear regression task match one optimal step of gradient descent, averaged over 4 seeds. The ”residual norm” measures the norm of the polynomial coefficients, excluding the ones appearing in the table. Those coefficients are all vanishingly small. The optimal GD learning rate is obtained analytically (\(\eta^{*}=(T+d_{x}-1/5)^{-1}\)), c.f. Appendix C.2.
sive performance improvement from the vanilla LRU architecture to the ones additionally including input gating to match our construction more closely.

## 5 Discussion

Our study reveals a closer conceptual relationship between RNNs and Transformers than commonly assumed. We demonstrate that gated RNNs can theoretically and practically implement linear self-attention, bridging the gap between these two architectures. Moreover, while Transformers have been shown to be powerful in-context learners (Brown et al., 2020; Chan et al., 2022), we find that RNNs excel in toy in-context learning tasks and that this performance is partly uncorrelated with the architecture inductive bias toward attention. This highlights the needed for further investigations on the differences between RNNs and Transformers in controlled settings, as also advocated by Garg et al. (2022).

Our results partly serve as a negative result: implementation of attention is possible but requires squaring the number of parameters attention has. We have shown that gated RNNs can leverage possible compression, but understanding whether real-world attention mechanisms lie in this regime remains an open question. Yet, our work is of current practical relevance as it provides a framework that can guide future algorithmic developments. Bridging the gap between Transformers' computational power and RNNs' inference efficiency is a thriving research area (Fournier et al., 2023), and the link we made facilitates interpolation between those two model classes.

Finally, our work carries implications beyond deep learning. Inspired by evidence from neuroscience supporting the existence of synaptic plasticity at different timescales, previous work (Schmidhuber, 1992; Ba et al., 2016; Miconi et al., 2018) added a fast Hebbian learning rule, akin to linear self-attention, to slow synaptic plasticity with RNNs. We show that, to some extent, this mechanism already exists within the neural dynamics, provided that the response of neurons can be multiplicatively amplified or shut-off in an input-dependent manner. Interestingly, several single-neuron and circuit level mechanisms have been experimentally identified which could support this operation in biological neural networks (Silver, 2010). We speculate that such multiplicative mechanisms could be involved in implementing self-attention-like computations in biological circuitry.

### Acknowledgements

The authors thank Asier Mujika and Razvan Pascanu for invaluable discussions. This study was supported by an Ambizione grant (PZ00P3_186027) from the Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1).