Resurrecting Recurrent Neural Networks for Long Sequences
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but
are hard to optimize and slow to train. Deep state-space models (SSMs) have
recently been shown to perform remarkably well on long sequence modeling tasks,
and have the added benefits of fast parallelizable training and RNN-like fast
inference. However, while SSMs are superficially similar to RNNs, there are
important differences that make it unclear where their performance boost over
RNNs comes from. In this paper, we show that careful design of deep RNNs using
standard signal propagation arguments can recover the impressive performance of
deep SSMs on long-range reasoning tasks, while also matching their training
speed. To achieve this, we analyze and ablate a series of changes to standard
RNNs including linearizing and diagonalizing the recurrence, using better
parameterizations and initializations, and ensuring proper normalization of the
forward pass. Our results provide new insights on the origins of the impressive
performance of deep SSMs, while also introducing an RNN block called the Linear
Recurrent Unit that matches both their performance on the Long Range Arena
benchmark and their computational efficiency.
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, Soham De
# Resurrecting Recurrent Neural Networks for Long Sequences

###### Abstract

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. In this paper, we show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, while also introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the _Long Range Arena_ benchmark and their computational efficiency.

## 1 Introduction

Recurrent neural networks (RNNs) have played a central role since the early days of deep learning, and are a natural choice when modelling sequential data (Elman, 1990; Hopfield, 1982; McCulloch and Pitts, 1943; Rumelhart et al., 1985). However, while these networks have strong theoretical properties, such as Turing completeness (Chung and Siegelmann, 2021; Kilian and Siegelmann, 1996), it is well-known that they can be hard to train in practice. In particular, RNNs suffer from the vanishing and exploding gradient problem (Bengio et al., 1994; Hochreiter, 1991; Pascanu et al., 2013), which makes it difficult for these models to learn about the long-range dependencies in the data. Several techniques were developed that attempt to mitigate this issue, including orthogonal/unitary RNNs (Arjovsky et al., 2016; Helfrich et al., 2018), and gating mechanisms such as long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and gated recurrent units (GRUs) (Cho et al., 2014). Nonetheless, these models are still slow to optimize due to the inherently sequential nature of their computation (Kalchbrenner et al., 2016), and are therefore hard to scale.

In recent years, Transformers (Vaswani et al., 2017) have gained increasing prominence for sequence modelling tasks, achieving remarkable success in a wide range of applications (Brown et al., 2020; Dosovitskiy et al., 2020; Jumper et al., 2021). Compared to RNNs, attention layers are easier to scale and parallelize during training, and crucially they do not suffer from the vanishing gradient problem, since the interaction between any two tokens in the sequence is modeled by direct edges in the network. A key issue with attention layers however is that their computational and memory costs scale quadratically as \(O(L^{2})\) with the sequence length \(L\). Transformers can therefore be especially expensive to deploy on long sequences. RNNs, which scale linearly with the sequence length, are therefore typically faster than transformers at inference time even for modest sequence lengths (Liu et al., 2019).

Motivated by these problems, Gu et al. (2021) recently introduced the S4 model, a carefully designed deep state-space model (SSM) achieving remarkable performance on tasks from the Long Range Arena (LRA) (Tay et al., 2020), a benchmark explicitly designed to require very long-ranged reasoning. S4 is theoretically principled and inspired by continuous-time linear SSMs; well-established components of modern control systems. More importantly, the S4 layer and its variants (DSS, S4D, S5, etc) (Gu et al., 2022; Gupta et al., 2022; Smith et al., 2022) overcome the \(O(L^{2})\) bottleneck of attention layers by modeling interactions between
tokens using a hidden state (like RNNs) under proper discretization techniques. These models can be made very efficient at inference time by simply unrolling the layer like an RNN. Futhermore, since SSMs are linear in the temporal dimension, they are easily parallelizable during training, in contrast to the slow sequential nature of training a typical RNN. This makes them very computationally efficient on long sequences.

While the S4 model is equivalent to an RNN during inference, it has a number of unique characteristics during training. For example, S4 is parameterized as a discretization of a latent continuous-time system of differential equations. S4 also uses specific initializations of the state matrices motivated from the theory of polynomial projections (Gu et al., 2020). While these characteristics might seem to motivate the impressive performance of these models, later works (Gu et al., 2022; Gupta et al., 2022, 2022; Smith et al., 2022) have suggested that the specific initialization used by S4 is often not crucial for performance, and that the discretization rules which achieve best performance may deviate from theory (Smith et al., 2022). It is therefore unclear what these unique characteristics of the deep SSMs are doing mechanistically, and how they can be simplified.

Motivated by the striking similarities between RNNs and deep SSMs, and in an attempt to better understand the underlying mechanism driving the performance of these models, we study the power and limitations of RNNs when used as core components of deep architectures for long-range reasoning. Our main goal is to answer the question:

_"Can we match the performance and efficiency of deep continuous-time SSMs using deep RNNs?"_

We give a _positive answer_ to this question. We show that the performance boost provided by deep SSMs like S4 can also be achieved via a series of small changes to a vanilla deep RNN. With these changes, we can recover the performance and efficiency of these deep SSMs on the Long Range Arena (LRA) benchmark (Tay et al., 2020). We call this new RNN model the Linear Recurrent Unit (or LRU for short).

Main Steps.We outline here the main steps needed towards crafting performant and efficient RNN models. Note while some of these observations have been made in prior works (see SSB), we provide novel perspectives and careful ablations leading to new insights. Each step presented in this paper unveils a specific property of

Figure 1: **(Left)** _Deep Linear Recurrent Unit (LRU) architecture introduced in this paper, inspired by S4 (Gu et al., 2021). The model is a stack of LRU blocks, with nonlinear projections in between, and also uses skip connections and normalization methods like batch/layer normalization. We expand on the details in §D and provide pseudocode in §A. We also use the same architecture structure (Norm-Recurrence-GLU-Skip) for every variant of the recurrent module in our study (\(\tanh\) dense, linear dense, etc.). (Right) Summary of effects for the main steps outlined in the introduction towards designing LRUs starting from \(\tanh\) RNNs. Shown is the average performance (3 seeds) of the recurrent module at each step on the Long Range Arena (LRA), compared to average performance of deep SSMs. For all LRA tasks, we match the performance of deep SSMs like S4/S4D/S5 with LRUs. Detailed results in §3._
recurrent networks, and showcases the challenges and best practices in training and initializing deep RNNs.
* **Linear Recurrences.** When replacing SSM layers in a deep architecture with vanilla RNN layers using tanh or ReLU activations, the performance on Long Range Arena (LRA) drops significantly. Surprisingly, in SS3.1 we find that simply _removing the nonlinearities in the recurrence of the RNN_ (i.e., using linear recurrences) gives a substantial boost in test accuracy. We motivate this effect in SSE.1 by showing that stacking linear RNN layers and nonlinear MLP blocks (Fig.1) can indeed model complex nonlinear sequence-to-sequence maps without the need for nonlinearities in the recurrence. While dropping the nonlinearity does not seem to harm expressivity, it leads to several advantages, from the ability to directly control how quickly the gradients might vanish or explode, to allowing us to parallelize training. Our findings also partially motivate the success of deep SSMs, where the recurrence is also linear.
* **Complex Diagonal Recurrent Matrices.** Dense linear RNN layers can be reparameterized to a complex diagonal form without affecting the expressivity of the network or the features at initialization (SS3.2). Diagonal linear RNN layers additionally allow for a _highly parallelizable unrolling_ of the recurrence using parallel scans to substantially improve training speeds (Martin and Cundy, 2017). We validate that these observations, which have been leveraged by prior SSMs (Gupta et al., 2022; Smith et al., 2022), also provide important efficiency improvements for linear RNN layers.
* **Stable Exponential Parameterization.** In SS3.3 we show that using an exponential parameterization for the diagonal recurrent matrix has important benefits. Crucially, this enables us to easily enforce stability during training, which in turn allows us to modify the initialization distribution to facilitate long-range reasoning and improve performance. Our results indicate that rather than the specific deterministic initializations used by several recent SSMs, it is the eigenvalue distribution of the recurrent layer at initialization that determines if the model can capture long-range reasoning.
* **Normalization.** In SS3.4 we show that normalizing the hidden activations on the forward pass is important when learning tasks with very long-range dependencies. With this final modification, our RNNs can match the performance of deep SSMs on all tasks in the LRA benchmark. Connecting back to state-space models, we show in SS4 how our normalization can be linked to the discretization structure in S4.

We summarize the deep Linear Recurrent Unit (LRU) architecture used in this paper, and the effect of each of the above steps on performance in Fig.1. We emphasize that the main purpose of our work is not to surpass the performance of S4-based models, but rather to demonstrate that simple RNNs can also achieve strong performance on long range reasoning tasks when properly initialized and parameterized. We believe the insights derived in this paper can be useful to design future architectures, and to simplify existing ones.

## 2 Preliminaries

In this section, we compare the key architectural components (RNNs and SSMs) studied in this work, and also describe our methodology and experimental setup. For a more thorough discussion or related architectures, the reader can check our related work section SSB.

### Recap of recurrent block structures

We give an overview of the main architectural components considered in this paper, focusing on the major difference between Vanilla RNNs and recent S4-like deep SSMs (Gu et al., 2021, 2022; Gupta et al., 2022; Smith et al., 2022).

RNN Layer.Let \((u_{1},u_{2},\ldots,u_{L})\) be a sequence of \(H_{\text{in}}\)-dimensional inputs, which can be thought of as either the result of intermediate layer computations (which keep the sequential structure) or as the initial input. An RNN layer with \(N\)-dimensional hidden state computes a sequence of \(H_{\text{out}}\)-dimensional outputs \((y_{1},y_{2},\ldots,y_{L})\) through a recurrent computation1 using learnable parameters \(A\in\mathbb{R}^{N\times N},B\in\mathbb{R}^{N\times H_{\text{in}}},C\in\mathbb{ R}^{H_{\text{out}}\times N},D\in\mathbb{R}^{H_{\text{out}}\times H_{\text{in}}}\):

Footnote 1: We do not use bias parameters as they can be incorporated into the MLP blocks preceding and following the RNN block. Classical RNNs also included a nonlinearity on the output \(y_{k}=a_{\text{out}}(Cx_{k}+b)\) with \(D=0\). Having \(D\neq 0\) basically introduces a skip connection (standard in modern architectures), and the \(\sigma_{\text{out}}\) can be thought of as part of the MLP following the RNN.

\[x_{k}=\sigma(Ax_{k-1}+Bu_{k}),\quad y_{k}=Cx_{k}+Du_{k}, \tag{1}\]
starting from \(x_{0}=0\in\mathbb{R}^{N}\). \(\sigma\) here denotes a nonlinearity, often chosen to be a \(\tanh\) or sigmoid activation. If \(\sigma\) is the identity function, then we say the RNN layer is _linear_.

S4-like recurrent layer.We present a simplified2 version of the S4 recurrence introduced in Gu et al. (2021). The input \((u_{0},u_{1},\ldots,u_{L-1})\) is now seen as the result of sampling a latent continuous-time signal \(u_{\text{ct}}:\mathbb{R}_{\geq 0}\rightarrow\mathbb{R}^{H_{\text{lin}}}\) at multiples of a stepsize \(\Delta>0\): i.e. \(u_{\text{ct}}(\Delta k):=u_{k}\) for all \(k\in 0,\ldots,L-1\). The output sequence \((y_{0},y_{1},\ldots,y_{L-1})\) is then sampled, again with stepsize \(\Delta\), from the signal \(y_{\text{ct}}:\mathbb{R}_{\geq 0}\rightarrow\mathbb{R}^{H_{\text{test}}}\) computed by the following continuous-time state-space model, initialized at \(x_{\text{ct}}(0)=0\):

Footnote 2: This version is most similar to SS (Smith et al., 2022), but is here presented for ease of reasoning for a single discretization parameter \(\Delta\), shared across input dimensions. For more details, see §B.

\[\frac{d}{dt}x_{\text{ct}}(t)=\tilde{A}x_{\text{ct}}(t)+\tilde{B}u _{\text{ct}}(t),\] \[y_{\text{ct}}(t)=\Re\left[\tilde{C}x_{\text{ct}}(t)\right]+ \tilde{B}u_{\text{ct}}(t), \tag{2}\]

where \(\Re\left(p\right)\) denotes the real part of a complex-valued vector \(p\), \(\tilde{A}=\text{diag}(\tilde{a})\) with \(\tilde{a}\in\mathbb{C}^{N}\), \(\tilde{B}\in\mathbb{C}^{N\times H_{\text{lin}}}\), \(\tilde{C}\in\mathbb{C}^{H_{\text{test}}\times N}\) and \(\tilde{D}\in\mathbb{R}^{H_{\text{test}}\times H_{\text{lin}}}\). Ignoring the continuous-time nature of this model, the most striking differences compared to Eq.(1) are that (a) the computation on the right-hand-side is _linear_ in the hidden state and in the input, and (b) most parameters are _complex_ valued, with \(\tilde{A}\) being diagonal. While \(\tilde{B},\tilde{C},\tilde{D}\) follow complex random or uniform initialization, the transition matrix \(\tilde{A}\) is _structured_, i.e., initialized _deterministically_ through HiPPO theory (Gu et al., 2020) in diagonal form. Common choices (Gu et al., 2022) are \(\tilde{a}_{n}=-\frac{1}{2}+i\pi n\) (S4D-Lin) and \(\tilde{a}_{n}=-\frac{1}{2}+i\frac{N}{\pi}\left(\frac{N}{n+1}-1\right)\) (S4D-Inv), for \(n=1,2,\ldots,N\).

For training and inference, the continuous-time system in Eq.(2) is discretized at stepsize \(\Delta\) through a high-accuracy Zero-Order-Hold (ZOH) or Bilinear method. The ZOH method gives

\[x_{k}=Ax_{k-1}+Bu_{k},\quad y_{k}=Cx_{k}+Du_{k}, \tag{3}\]

where \(x_{-1}=0\), \(A=\exp(\Delta\tilde{A})\), \(B=(A-I)\tilde{A}^{-1}\tilde{B}\), \(C=\tilde{C}\) and \(D=\tilde{D}\), and \(\exp\) denotes the matrix exponential. Under the assumption that \(u_{\text{ct}}\) is constant in between timestamps (which can be thought of as a modeling assumption), this numerical integration is _exact_(Jacquot, 2019). Moreover, note that all these discretization operations can be quickly performed element-wise since \(\tilde{A}\) is diagonal.

Some key differences.It is worth pointing out a few structural and computational properties, to highlight some crucial differences between RNNs and SSMs:

* Since Eq.(3) is linear, it can be efficiently parallelized until \(k=L-1\) using parallel scans (Martin and Cundy, 2017; Smith et al., 2022), unlike a nonlinear RNN where the computation has to be performed sequentially.
* While Eq.(3) is similar to the linear RNN computation, it is crucial to note that (a) \(A\) and \(B\) are parameterized in a peculiar way, prescribed by discretization, and (b) these matrices share parameters; in particular \(\Delta\) affects both \(A\) and \(B\). These differences are critical as in SSMs learning is performed on the continuous-time parameters \(\tilde{A},\tilde{B},\tilde{C},\tilde{D},\Delta\); hence parameterization choices directly affect optimization.
* Unlike vanilla RNNs, most SSMs use complex-valued diagonal recurrent matrices that are initialized deterministically using HiPPO theory, and the literature attributes much of the success of SSMs to the specific initialized used (Gu et al., 2021, 2022; Gupta et al., 2022).

The points above motivate our investigation: in this paper we consider the same architecture as Gu et al. (2021, 2022); Smith et al. (2022), but replace the SSM layer in the recurrent core by an RNN. We then study which steps need to be taken to gradually retrieve 54-like performance on LRA (Tay et al., 2020) tasks. The effectiveness of each of our steps is supported by empirical evidence and theoretical considerations, and leads to the architecture presented in Fig.1.

### Experimental setup

In this paper, we consider the Long Range Arena benchmark (Tay et al., 2020), a set of tasks designed to test the ability of models to do long-range sequence modelling (except we use coloured images instead of grayscale images for the sequential CIFAR-10 classification task). Transformers fail to perform well on most of these tasks,
while deep SSMs have shown remarkable performance on these tasks (Dao et al., 2022; Gu et al., 2021). This makes it an appropriate benchmark to explore the long-range modelling capabilities of deep RNNs.

For all our experiments, we use a network of 6 layers with residual connections and layer/batch normalization (Ba et al., 2016; Ioffe and Szegedy, 2015) similar to Gu et al. (2021) (Fig.1), and we replace the SSM layers with RNN layers, building up to our LRU recurrence in a sequence of steps (see SS3). All experiments are repeated three times, and we report the mean and standard error. Networks are trained using the AdamW optimizer (Loshchilov and Hutter, 2017). We use a smaller learning rate and no weight decay on the recurrent parameters, as suggested by Gu et al. (2021); Steil (2004). We tune hyperparameters such as learning rates for all models on a logarithmic grid for best accuracy. See SSD for more details on our experimental setup.

## 3 Designing Performant Deep RNNs

In this section, we discuss the fundamental steps needed for designing RNNs to reach the impressive performance of deep SSMs on the LRA benchmark. We present these steps, already outlined in the introduction, in logical order, and support each claim with experimental evidence and theoretical considerations, expanded in SSE.

We consider the architecture of Fig.1, where the recurrent computation is gradually modified starting from a vanilla RNN. We start by showcasing the advantage of using linear recurrences in SS3.1; then, in SS3.2, we show how to speed-up training and inference without affecting expressivity and initialization distribution. In SS3.3, we discuss how (and why) changing the parameterization and initialization distribution enables us to make the RNN stable and improve long-range modeling. Finally, in SS3.4, we finalize the LRU architecture by proposing a normalization strategy for the hidden activations that results in a close match in performance with deep SSMs.

### Linear RNN layers are performant

One of the main findings of our work is that linear RNN layers can be surprisingly expressive when coupled with nonlinear MLP or GLU (Dauphin et al., 2017) blocks, outperforming tuned nonlinear RNN variants in the same architecture. In Tb.1, we show that simply removing3 the nonlinearity, and therefore computing the next state as \(x_{k}=Ax_{k-1}+Bu_{k}\), is able to improve test accuracy on most LRA tasks. While the boost provided by vanilla linear RNN blocks leads to performance which is still far behind S4 on some tasks (sCIFAR, PathFinder and PathX), this first finding motivates us to drop nonlinearities in the recurrence for the rest of this paper. In later sections, we leverage the linearity of the recurrence to significantly speed up training as well as derive principled initialization and normalization principles to learn long-range dependencies. We note that, on the Text and Retrieval tasks, performance using vanilla RNNs already matches performance of deep SSMs (see Tb.3 for the performance of S4D/S5 on these tasks).

Footnote 3: All other settings in the recurrent block are kept the same as in the Vanilla RNN module of Haiku (Hennigan et al., 2020). That is, all matrices have Glorot (Glorot and Bengio, 2010) initialization. The rest of the architecture is kept as in Fig.1, where the LRU block is replaced by an RNN.

Table 1 \(|\)_The effect of removing the nonlinearity from the recurrent unit on test accuracy (SS3.1). We show here results only for the sCIFAR, ListOps, Text and Retrieval tasks in LRA as these models did not exceed random guessing on PathFinder/PathX (further improvements in Tb.2 and 3). Performance of deep SSMs shown in Tb.3._

The empirical result in Tb.1 is _surprising_, since recurrent nonlinearities are believed to be a key component for the success of RNNs -- both in the theory and in practice (Erichson et al., 2021; Pascanu et al., 2013; Siegelmann, 2012). Indeed, a strong property of single-layer sigmoidal and tanh RNNs is Turing completeness, which cannot be achieved by the linear variant (Chung and Siegelmann, 2021). However, the architecture we use (Fig.1) is deeper than a standard RNN and includes nonlinearities, placed position-wise _after_ each RNN block. In SSE.1, we investigate how the expressivity and trainability of deep models is affected by recurrent
nonlinearities. Leveraging a spectral analysis and Koopman operator theory (Koopman and Neumann, 1932), we discuss how interleaving linear RNN layers with nonlinear feedforward blocks is sufficient to approximate highly nonlinear systems. A key observation in our analysis is that position-wise nonlinearities effectively transfer signal information to higher frequencies, enabling the system to go beyond linearity in the spectral domain and increasing the layer capacity. To further strengthen our claim on the advantage of linear recurrences, in SSE.2 we show that, while linear and nonlinear RNNs share an important class of approximating functionals (linear operators, see Wang et al. (2022)), nonlinear activations can potentially slow down training.

### Using complex diagonal recurrent matrices is efficient

We now show that we can significantly speed up training and inference for deep linear RNNs without losing performance by using complex-valued diagonal recurrent matrices. While the idea of diagonalizing linear systems for computational efficiency is a dominating feature of all deep SSMs since the introduction of DSS by Gupta et al. (2022), in this section we construct our diagonalized version to exactly match the initialization spectrum (see SS3.2.1) of the Glorot-initialized deep linear RNN in Tb.1. Our main purpose with this approach is to _disentangle the effects of initialization and diagonalization on performance_ (cf. Tb.2 and Tb.3).

We start in SS3.2.1 by recalling some useful linear algebra elements, and then proceed in SS3.2.2 with a discussion on how to diagonalize the recurrence while preserving the eigenvalue spectrum at initialization.

#### 3.2.1 Linear RNN eigendecomposition

The recurrence \(x_{k}=Ax_{k-1}+Bu_{k}\) can be unrolled easily using the assumption that \(x_{-1}=0\in\mathbb{R}^{N}\):

\[x_{0}=Bu_{0},\quad x_{1}=ABu_{0}+Bu_{1},\quad x_{2}=A^{2}Bu_{0}+ABu_{1}+Bu_{2}, \quad\ldots\quad\Longrightarrow\quad x_{k}=\sum_{j=0}^{k-1}A^{j}Bu_{k-j}. \tag{4}\]

Exponentiations of the matrix \(A\) in the equation above are the source of the well-known _vanishing/exploding gradient_ issue in RNNs (Bengio et al., 1994; Pascanu et al., 2013). While in nonlinear RNNs the state \(x_{k}\) is forced to live on the compact image of the activation function, the hidden-state of our linear variant can potentially explode or vanish exponentially as \(k\) increases. This phenomenon can be better understood by leveraging an eigenvalue (a.k.a. spectral) analysis: up to an arbitrarily small perturbation of the entries, every matrix \(A\in\mathbb{R}^{N\times N}\) is diagonalizable4(Axler, 1997), i.e. one can write \(A=P\Lambda P^{-1}\), where \(P\in\mathbb{C}^{N\times N}\) is an invertible matrix and \(\Lambda=\text{diag}(\lambda_{1},\lambda_{2},\ldots,\lambda_{N})\in\mathbb{C}^{ N\times N}\). It is essential to note that, unlike the symmetric setting where eigenvalues and eigenvectors are real, in the non-symmetric case5 one has to allow for _complex_ entries to achieve full equivalence. Plugging the decomposition \(A=P\Lambda P^{-1}\) into Eq.(4) and multiplying both sides by \(P^{-1}\), we get \(\tilde{x}_{k}=\sum_{j=0}^{k-1}\Lambda^{j}\tilde{B}u_{k-j}\), where \(\tilde{x}_{k}:=P^{-1}x_{k}\), \(\tilde{B}:=P^{-1}B\). The output can then be computed as \(y_{k}=\Re[\tilde{C}\tilde{x}_{k}]+Du_{k}\in\mathbb{R}^{H}\), where \(\tilde{C}=CP^{-1}\), and we take the real part of \(\tilde{C}\tilde{x}_{k}\). Therefore, instead of learning \((A,B,C,D)\), one can equivalently learn \((\Lambda,\tilde{B},\tilde{C},D)\), where \(\Lambda,\tilde{B},\tilde{C}\) are complex valued, and \(\Lambda\) is a diagonal matrix.

Footnote 4: In other words, the set of non-diagonalizable matrices has measure zero, see e.g. Zhinan (2002) for a proof idea.

Footnote 5: Take e.g. \(A=((0,1)(-1,0))\). The solution to the standard eigenvalue equation gives \(\lambda=\pm i\), where \(i\) is the imaginary unit.

Are complex numbers really necessary?We adopt complex numbers since they provide a convenient and compact representation of non-symmetric matrices in diagonal form. However this is not the only option - one could work (almost) as efficiently using real numbers. We discuss how this can be achieved in SSE.3.

Stability.Since \(\tilde{x}_{k}=\sum_{j=0}^{k-1}\Lambda^{j}\tilde{B}u_{k-j}\), the norm of component \(j\) of \(\tilde{x}\) at timestamp \(k\) evolves such that \(|x_{k,j}|=O(|\tilde{x}_{k,j}|)=O(|\lambda_{j}|^{k})\). Therefore, a sufficient condition to ensure stability (i.e. \(x_{k}\) does not explode) is therefore \(|\lambda_{j}|<1\) for all \(j\)(Gu et al., 2021).

#### 3.2.2 Learning in the diagonalized space

Learning recurrent linear systems in diagonal form provides substantial computational speedups both for training and inference. For example, in our implementation of sCIFAR, we found diagonal linear RNNs to be \(\sim\)8 times faster to train than a dense RNN with ReLUs, matching the speed of our implementations of S4D and S5. The main reasons for this computational benefit are that (a) taking powers of diagonal matrices is
trivial (speeding up both training and inference), while exponentiating dense matrices is computationally expensive, and (b) while nonlinear recurrences must be computed sequentially, unrolling a linear recurrence can be parallelized using associative scans resulting in faster training (Gupta et al., 2022; Smith et al., 2022).

Equivalent initialization.To disentangle the benefits of diagonal linear systems from the role of initialization, we seek an initialization for the diagonal system which keeps the eigenvalue spectrum of the recurrence unchanged when comparing our diagonal system with the dense linear RNN in SS3.1, where \(A\) followed Glorot initialization. Fortunately, we can use a classical result from random matrix theory (Ginibre, 1965).

**Theorem 3.1** (Strong circular law).: _Let \(\mu_{N}\) be the empirical spectral measure of \(A_{N}\), where \(A_{N}\) is a real \(N\times N\) matrix with i.i.d. Gaussian entries, each with zero mean and variance \(1/N\). Then, \(\mu_{N}\) converges weakly almost surely as \(N\to\infty\) to the uniform probability measure on \(\{|z|\leq 1\}\subseteq\mathbb{C}\)._

The theorem above, illustrated in Fig.2, shows that under Glorot initialization the spectrum of \(A\) is _de-facto_ sampled from the unit disk in \(\mathbb{C}\). This result motivates the strong performance of linear RNNs in SS3.1, since it implies Glorot initialization provides an approximately stable initialization (see definition in SS3.2.1).6 Moreover, from Theorem 3.1, an _equivalent spectral initialization_ follows for the diagonal system, which holds exactly for the large width limit: \(\Lambda\) should be diagonal with entries sampled uniformly on the unit disk. Using the definition of exponential of a complex number: \(\exp(-\nu+i\theta):=e^{-\nu}(\cos(\theta)+i\sin(\theta))\), we adopt a simple scheme for sampling uniformly on a ring in between circles with radii \(r_{\min}\) and \(r_{\max}\)in \(\mathbb{C}\).

Footnote 6: Later in training, the system is less likely to become unstable if the learning rate is small enough.

**Lemma 3.2**.: _Let \(u_{1},u_{2}\) be independent uniform random variables on the interval \([0,1]\). Let \(0\leq r_{\min}\leq r_{\max}\leq 1\). Compute \(\nu=-\frac{1}{2}\log\left(u_{1}(r_{\max}^{2}-r_{\min}^{2})+r_{\min}^{2}\right)\) and \(\theta=2\pi u_{2}\). Then \(\exp(-\nu+i\theta)\) is uniformly distributed on the ring in \(\mathbb{C}\) between circles of radii \(r_{\min}\) and \(r_{\max}\)._

We recover the spectrum of Glorot-initialization (in the limit of infinite width) by setting \(r_{\min}=0\) and \(r_{\max}=1\) (we will explore tuning these hyper-parameters in SS3.3). Tb.2 (first two rows) shows the results of learning deep linear RNNs in complex diagonal form,7 where each diagonal entry of \(\Lambda\) is initialized uniformly on unit disk in \(\mathbb{C}\) using Lemma 3.2 with \([r_{\min},r_{\max}]=[0,1]\). In our experiments, \(\bar{B},\bar{C}\) (which we rename for convenience back to \(B\) and \(C\)) follow Glorot initialization for both real and imaginary parts (parameterized separately), with halved variance in each component to preserve lengths on the input-output projections (Glorot and Bengio, 2010). Finally, after the SSM computation, the real part of the signal is kept and the imaginary discarded (as in Gu et al. (2022); Gupta et al. (2022)).

Footnote 7: To avoid issues with backpropagation on complex variables, each complex parameter in the network is stored and learned as a pair of floats encoding real and imaginary parts.

Our results in Tb.2 show that diagonalizing the recurrence surprisingly improves accuracy on tasks like ListOps and sCIFAR. More importantly, it drastically reduces training and inference time on all LRA tasks (see Tb.4 in SC.1 for training speed comparisons), and makes the RNN just as fast to train as deep SSMs like 54D and S5.
### Benefits of stable exponential parameterization

In SS3.2 we showed that moving to complex diagonal recurrences is computationally efficient. However we also observed that learning the diagonal model can be more unstable than learning the dense model in some experiments. To learn long-range dependencies and avoid quickly vanishing gradients, eigenvalues in the recurrence need to have magnitude close to 1 (Gu et al., 2022; Gupta et al., 2022); however, these eigenvalues are also likely to make the system unstable during training. In this section, we show the benefits of a stable parameterization of the RNN, and of tuning \(r_{\text{min}}\) and \(r_{\text{max}}\) (see Lemma 3.2).

Optimization under exponential parameterization.Lemma 3.2 suggests a natural parameterization of the diagonalized RNN as \(\Lambda=\text{diag}(\exp(-\nu+i\theta))\) with \(\nu\in\mathbb{R}^{N}\) and \(\theta\in\mathbb{R}^{N}\) as the learnable parameters (instead of the real and imaginary parts of \(\Lambda\)). As we explain in SSE.2 leveraging an easy-to-visualize 2-dimensional example (see Fig.8), this choice decouples magnitude and oscillation frequencies, making optimization with Adam easier. The positive effects of this exponential parametrization, which resembles some features of ZOH discretization (see SS2 and SS4) and notably takes the performance of PathFinder above random chance, can be observed in the third row of Tb.2.

Enforcing stability.An important benefit of the exponential parameterization is that it makes it simple to enforce stability on the eigenvalues. To see this, note that at initialization, \(|\lambda_{j}|=|\exp(-\nu_{j})|\leq 1\) since \(\nu_{j}>0\). Therefore, to preserve stability during training, we can use an exponential or another positive nonlinearity: \(\lambda_{j}:=\exp(-\exp(\nu_{j}^{\text{log}})+i\theta_{j})\), where \(\nu^{\text{log}}\in\mathbb{R}^{N}\) is the parameter we optimize, and we set \(\nu_{j}^{\text{log}}:=\log(\nu)\) at initialization. Note that a similar idea is used in deep SSMs (Gu et al., 2021) in the context of discretization. We choose an exponential non-linearity over a simple ReLU nonlinearity to increase granularity around \(|\lambda|=1\), achieved at \(\nu^{\text{log}}=-\infty\) (while \(|\lambda|=0\) is achieved at \(\nu^{\text{log}}=\infty\)). Stable parameterization helps on most LRA tasks. In the fourth row of Tb.2, we show its effects on sCIFAR, ListOps and Pathfinder. We observe the most drastic improvement on Pathfinder, one of the harder long-range dependency tasks in LRA, where performance now reaches above 93%.

The benefits of the stable parameterization becomes more apparent when we explore the idea of initializing the eigenvalues of \(\Lambda\) on a ring closer to the unit disk (increasing \(r_{\text{min}}\) closer to 1 in Lemma 3.2) to bias the network towards longer-range interactions and avoid vanishing gradients. Indeed, as discussed in detail in Gu et al. (2022); Gupta et al. (2022), for reasonings requiring consideration of interactions between distant tokens, eigenvalues in the recurrence need to have magnitude close to 1. Otherwise, as clear from the diagonal version of Eq.(4), when taking powers of eigenvalues close to the origin, the signal from past tokens quickly dies out (see SS3.2.1). As we show in the last row of Tb.5 in SSC, without enforcing stability, performance starts to degrade as we increase \(r_{\text{max}}\) past 0.9 in the sCIFAR task. With stability enforced, we can increase \(r_{\text{max}}\) up to 0.99 and improve performance. We see similar benefits on the other tasks where we sweep different values of \(r_{\text{min}}\) and \(r_{\text{max}}\) (Tbs.7 & 8 have more details). Finally, note that while here we explore changing the magnitude of the eigenvalues of \(\Lambda\), in SS3.4 we also show the benefits of initializing the eigenvalues to have a small phase to learn more global patterns, useful for particularly long-range reasoning tasks.

\begin{table}
\begin{tabular}{|l||c|c|c|} \hline  & sCIFAR & ListOps & Pathfinder \\ \hline Dense \(A\) & 72.2 (0.2) & 50.4 (0.2) & ✗ \\ \(\Lambda\) Real + Im & 86.5 (0.1) & 58.8 (0.3) & ✗ \\ \(\Lambda\) Exp & 85.4 (0.7) & **60.5** (0.3) & 65.4 (9.0) \\ \(\Lambda\) Stable Exp & 87.2 (0.4) & 59.4 (0.3) & 93.5 (0.5) \\ + Ring Init & **88.1** (0.0) & 59.4 (0.3) & **94.4** (0.3) \\ \hline \end{tabular}
\end{table}
Table 2: _Test accuracy of a linear diagonal complex RNNs under different parametrizations of the transition matrix (see §3.2). Performance directly improves the results in Tb.1, and showcases the advantage of exponential (polar) representation of \(\Lambda\). In bold font is the best parametrization option for linear RNN blocks. Ring Init denotes a changed initialization where \(r_{\text{min}}\) and \(r_{\text{max}}\) are tuned. Performance on the Text and Retrieval tasks is not shown as linear RNNs already align with S4 results (cf. Tb.1 with Tb.3). These models cannot solve PathX yet, and requires normalizing the hidden activations and initializing the eigenvalues of \(\Lambda\) with small phase (see Tb.3)._
### Additional considerations for long-range reasoning tasks

Up to this point, our model did not succeed in learning PathX -- the hardest dataset in our benchmark, with a sequence length of 16k tokens. In this section, we discuss the additional modifications we need to make to improve our model's ability to learn very long-range dependencies and finalize our LRU model.

Normalization.In SS3.3, we initialized the eigenvalues of \(\Lambda\) close to the unit disk for better performance on long-range tasks. However, we observed that as we moved \(r_{\text{min}}\) and \(r_{\text{max}}\) closer to 1, the training loss also started to blow up at initialization (see Fig.5). In this section, we first present a result explaining this phenomenon, before deriving a practical normalization scheme for the hidden activations to tackle this problem and further improve performance.

**Proposition 3.3** (Forward-pass blow-up).: _Let \(\Lambda\) be diagonal with eigenvalues sampled uniformly on the ring in \(\mathbb{C}\) between circles of radii \(r_{\text{min}}<r_{\text{max}}<1\). Then, under constant or white-noise input and Glorot input projection, we have that the squared norm of the state \(x_{k}\) converges as \(k\to\infty\) to the following quantity._

\[\mathbb{E}[\|x_{\infty}\|_{2}^{2}]=\frac{1}{r_{\text{max}}^{2}-r_{\text{min}}^ {2}}\log\left(\frac{1-r_{\text{min}}^{2}}{1-r_{\text{max}}^{2}}\right)\mathbb{ E}[\|Bu\|_{2}^{2}].\]

This result has the following intuitive form if \(r_{\text{min}}=r_{\text{max}}=r\): if we initialize \(\rho\)-close to the unit disk, the forward pass blows up by a factor \(1/\rho\) (since the contributions from previous states take longer to decay): let \(\epsilon=r_{\text{max}}^{2}-r_{\text{min}}^{2}\) and \(\rho=1-r_{\text{max}}^{2}\), then:

\[\lim_{\epsilon\to 0}\frac{\mathbb{E}[\|x_{\infty}\|_{2}^{2}]}{\mathbb{E}[\|Bu\|_{ 2}^{2}]}=\lim_{\epsilon\to 0}\left[\frac{1}{\epsilon}\log\left(1+\frac{ \epsilon}{\rho}\right)\right]=\lim_{\epsilon\to 0}\left[\frac{1}{\epsilon} \left(\frac{\epsilon}{\rho}+O(\epsilon^{2})\right)\right]=\frac{1}{\rho}= \frac{1}{1-r^{2}}. \tag{5}\]

Towards the derivation of an effective normalization scheme for the forward pass, we present a simplified derivation of the \(1/\rho\) gain formula for the one-dimensional setting under white-noise input8: let \(\Lambda=\lambda\in\mathbb{C}\), and \(B=1\). Let \(p^{*}\) denote the conjugate of \(p\in\mathbb{C}\), we have that \(|p|^{2}=p^{*}p\) and in expectation over the input, using Eq.(4) and the fact that \(\mathbb{E}[u_{k-i}u_{k-j}]=0\) for \(i\neq j\):

Footnote 8: We use the random input assumption for our normalization scheme as we found it to work well in practice.

\[\mathbb{E}|x_{k}|^{2}=\left(\sum_{i=0}^{k-1}\lambda^{i}\mathbb{E}[u_{k-i}] \right)\left(\sum_{j=0}^{k-1}\lambda^{j}\mathbb{E}[u_{k-j}]\right)^{*}=\sum_{ i,j=0}^{k-1}\lambda^{i}(\lambda^{j})^{*}\mathbb{E}[u_{k-i}u_{k-j}]=\sum_{i=0}^{k-1} \lambda|2^{i}\overset{\infty}{\to}\frac{1}{1-|\lambda|^{2}}. \tag{6}\]

Since the formula above holds for every Euclidean direction in our recurrence (\(\Lambda\) is diagonal), we can add a normalization parameter that is initialized element-wise. Additionally, note that as \(\lambda\) approaches \(1\), \(1-|\lambda|^{2}\)
approaches \(0\), making further adaptations with SGD of this parameter hard. Therefore, we use normalization parameter \(y^{\log}\in\mathbb{R}^{N}\), initialized element-wise as \(Y_{i}^{\log}\leftarrow\log(\sqrt{1-|A_{i}|^{2}})\),9 and modify the recurrence as:

Footnote 9: We also tried setting \(y_{i}\) to \(\sqrt{1-|A_{i}|^{2}}\) in each training iteration, and found it to work similarly in practice to a trainable \(y\).

\[x_{k}=\Lambda x_{k-1}+\exp(y^{\log})\odot(Bu_{k}), \tag{7}\]

where \(\odot\) denotes the element-wise product. The \(y\) parameter allows the RNN to adaptively scale the input fed into the corresponding eigendirection. We found the \(y\) normalization to consistently improve performance on tasks that benefit from initializing close to the unit disk, such as sCIFAR and Pathfinder, as shown in Tb.3.

Reducing Eigenvalue Phase at Initialization.In the context of the diagonalized recurrence, we have \(\Lambda=\operatorname{diag}(\exp(-\exp(\nu^{\log})+\theta))\), where \(\nu^{\log}\in\mathbb{R}^{N}\) is the vector of log eigenvalue magnitudes and \(\theta\in\mathbb{R}^{N}\) the vector of eigenvalue _phases_. While \(\nu^{\log}\) encodes the distance to the origin, \(\theta\) is the angle from the vector \(1+0i\). _For long sequences_, initializing uniformly \(\theta\sim[0,2\pi]\) implies that most state entries will exhibit an overall large number of oscillations at initialization, see upper panel in Fig.4. Equivalently, in this setting, most state dimensions are the result of _convolutions_10 capturing an average of _local oscillation patterns_. This behavior is independent from the ability of capturing long-range dependencies (controlled by \(\nu^{\log}\)), but pertains to the nature of the information stored by the RNN. Therefore, we claim that initializing \(A\) with uniform phase on long sequence data inherently biases the network towards learning spurious features in the input sequence. The model cannot recover from this suboptimal initialization: we indeed observe that, for our best to far model on PathX, the

Figure 4: _Evolution of \(x\in\mathbb{R}^{3}\) under impulse input \(u=(1,0,0,\ldots,0)\in\mathbb{R}^{16k}\). Plotted in different colors are the 3 components of \(x\). \(\Lambda\) has parameters \(\nu_{j}=0.00005\) and \(\theta_{j}\) sampled uniformly in \([0,2\pi]\) or with small phase \([0,\pi/50]\). For small sequences, such as \(L=1024\) (PathFinder, sCIFAR), \([0,2\pi]\) produces kernels with acceptable overall number of oscillations: information about \(u_{0}\) is recalled only a few times in the overall state history. Instead, for high \(L\), the range of the imaginary part at initialization has to be smaller to obtain a similar effect._

Figure 5: _Efficet of normalization and using a small phase at initialization on the PathX task. For each setting, we show mean and standard errors over three independent runs for 100k iterations. Without normalization, the model presents higher loss values at initialization and quickly converges to a suboptimal value, where train and test accuracy are both at random chance. Adding normalization helps: the train loss is lower at initialization, and the optimizer is able to escape the suboptimal region and train accuracy also increases. Interestingly, this model still fails to generalize at all. Finally, reducing initialization phase (i.e. tuning the range of \(\theta\)) dramatically improves convergence on the training set, while also generalizing to the test set._
training loss after a few iterations converges to a highly suboptimal minimizer which leads to random chance test performance (see Fig.5). To fix this issue, we found it sufficient to restrict the range of \(\theta\) to a thin slice around \(0\), biasing the model towards learning more global features. Since the optimal values of \(\theta\) are small, we parameterize the phase logarithmically: \(\theta=\exp(\theta^{\log})\), where \(\theta^{\log}\) is optimized, to aid optimization.

Restricting the range of the phase at initialization to be \([0,\pi/10]\), our LRU achieved 94.2% _on PathX_, aligning with state-of-the-art deep SSMs. We did not explore using a smaller phase at initialization for the other LRA tasks, although we believe this might further improve performance on other tasks as well. Note that using both \(\gamma\) normalization and restricting the eigenvalue phase at initialization were crucial to solving PathX. We were unable to learn when using restricted phase at initialization without also introducing \(\gamma\) normalization.

With all the components of SS3 taken together, we name this new model the **Linear Recurrent Unit** (or **LRU** for short). It provides a flexible, interpretable, and principled framework for initializing and learning deep RNNs efficiently, and matches performance and efficiency of deep SSMs across all LRA tasks as shown in Tb.3.

## 4 Insights on S4 and Variants

We believe our ablations in SS3 explain the underlying mechanisms driving the success of deep SSMs. Hence, to conclude the paper, in this section, we inspect in detail the main similarities and differences between our LRU model and diagonal SSMs, and elaborate a few insights. As in SS2, to avoid technicalities, we provide a simplified discussion capturing the main features of models stemming from the original S4 paper. For a comparison of different models, we defer the reader to SSB.

As detailed in SS2, diagonal SSMs (DSS, S4D, S5) are instantiated and parameterized through _discretization_ of a latent continuous-time model \(\hat{x}_{\mathrm{ct}}(t)=\hat{A}x_{\mathrm{ct}}(t)+\hat{B}u_{\mathrm{ct}}(t)\), where \(A=\mathrm{diag}(\bar{a})\) is initialized with complex entries, often prescribed or inspired by HiPPO theory (Gu et al., 2020). Zero-Order-Hold (ZOH) discretization with stepsize \(\Delta\) leads to the recurrence \(x_{k}=\exp(\Delta\bar{A})x_{k-1}+(\exp(\Delta\bar{A})-I)\bar{A}^{-1}\bar{B}u_{k}\). This formula, while arguably complex compared to our Eq.(7), relates to it as outlined in the next paragraphs.

Matrix exponentials make training easier.The exponential in the ZOH formula is due to exact integration of \(\hat{x}_{\mathrm{ct}}(t)=\bar{A}x_{\mathrm{ct}}(t)\), which leads to \(x_{\mathrm{ct}}(\Delta k)=\exp(\Delta\bar{A})x_{\mathrm{ct}}(\Delta(k-1))\). In addition, to enforce stability, in models inspired by S4 the real part of \(A\) is often fed into a positive nonlinearity, as we also do in SS3.3. From our results SS3.3 and our discussion on optimization advantages (see also SSE.2), we claim that the power of exponential parameterization is not necessarily attributable to accurate integration (which is not present in our system), but is more fundamentally rooted in a magnitude-phase decoupling on the recurrence (this makes training with Adam easier, see Fig.8), as well as in the overall advantage of learning in diagonalized space (see Tb.2). We also note that stabilizing the recurrence by adding a nonlinearity was beneficial also in our experiments, although this is not prescribed by the theory underlying S4.

Structured initialization is not necessary.While Gu et al. (2022); Gupta et al. (2022); Smith et al. (2022) also discuss initializations for \(A\) deviating from the HiPPO structure (see SS2 and SSB), to the best of our knowledge we are the first to show that simple uniform initialization on a slice of the unit disk, combined with proper normalization, is able to also solve the hardest task in LRA: PathX.11 We also show (Tb.2) that uniform initialization on the disk, which is simply the diagonalized version of Glorot initialization (Thm. 3.1), is sufficient to achieve performance close to more complex deep state-space models on the remaining LRA tasks. Our results ultimately suggest that HiPPO theory, while fundamental for the development of this field, should not be thought of as the main source of S4 success.

Footnote 11: Among the models in (Gu et al., 2022), only S4D-inv and S4D-LegS (options heavily inspired by the HiPPO theory) perform beyond random guessing on PathX. In S5, the skew-symmetric component of the HiPPO matrix is used for initialization.

Discretization changes initialization spectrum.For simplicity, let us restrict our attention to S4D-Lin, for which \(A=\mathrm{diag}(\bar{a})\) with \(\bar{a}_{n}=-\frac{1}{2}+i\pi n\), yielding a diagonal transition matrix with elements (i.e. eigenvalues) initialized at \(\exp(-\Delta/2+i\pi\Delta n)\). Under typical choices e.g. \(\Delta=1e{-3},N=128\), the SSM eigenvalues have magnitude \(\exp(-\Delta/2)\approx 0.9995\), and phase \(\theta=\pi\Delta n\stackrel{{\leftarrow}}{{\in}}[0,\pi/8]\) -- i.e. initialization is performed on a ring12 close to the unit circle in \(C\), with restricted phase connected to the eigenvalues magnitude. As is clear from
the results in SS3.3 and SS3.4, linking the eigenvalues phase and magnitude is not necessary to achieve good performance: indeed, as it can be seen in Tb.3, test accuracy on the Long Range Arena (except PathX) can be recovered by using a more natural magnitude-independent initialization on the complete ring. As we discussed in SS3.4, changing the initialization phase to a small range around 0 can be motivated by first principles, yet is only needed for extremely long sequences: this modification is already hard-coded in S4, where choosing a small \(\Delta\) also shrinks the phase.13 However, our results clearly show that connecting real and imaginary parts during training through the \(\Delta\) parameter is not necessary to achieve good performance, even on PathX.

Footnote 13: This is a useful effect of having a latent continuous-time model: choosing eigenvalues close to the unit circle (i.e. small \(\Delta\)) changes the oscillation frequencies in the discretized system.

Discretization performs normalization.The most striking visual difference between ours and ZOH-discretized S4 recurrence is in the matrix multiplier for \(u_{k}\): \((\exp(\Delta\bar{A})-I)\bar{A}^{-1}\bar{B}\). After conducting experiments on S4D, we found that simply replacing this multiplier with its first-order expansion in \(\Delta\), i.e. \(\Delta B\), yields a close match in performance. For input dimension \(H=1\) and unit \(B\in\mathbb{R}^{N\times 1}\) (to keep reasoning simple), the corresponding recurrence is \(x_{k}=\exp(\Delta\bar{a})+\Delta 1_{N}u_{k}\). Elementwise unrolling of this recurrence - without the \(\Delta\) in front of \(u\) - yields \(|x_{k,i}|\leq\sum_{j=0}^{k}|\exp(\Delta\bar{a}_{i})|u_{k-j,i}\), which in the limit \(k\to\infty\) gives \(O(\Delta^{-1})\). Therefore, the \(\Delta\) multiplier in front of \(B\) effectively scales the recurrence to avoid blow-ups -- similar to our \(y\) normalization factor.

Parameter sharing is not necessary.As a result of discretization, the \(\Delta\) parameter multiplying both \(\bar{A}\) and \(\bar{B}\) couples the recurrence formula with the input projection during training. In our S4 ablations, we found that decoupling these in two separate parameters -- keeping the same initialization to guarantee no blow-ups (see last paragraph) -- does not decrease performance, suggesting that the ODE discretization viewpoint (which induces parameter sharing) is not necessary to achieve S4 performance.

From this discussion, we conclude that the success of (diagonal) state-space models is attributable to the use of linear recurrences and complex diagonal exponential matrices, combined with the normalization and initialization induced by discretization. On the other hand, other artifacts of discretization such as parameter sharing or the continuous-time interpretation do not necessarily contribute to its performance.

## 5 Conclusion

In this paper, we introduce a new RNN layer called the Linear Recurrent Unit or LRU and show how it can be effectively and efficiently used as core layers of deep sequence models. We provide theoretical insights and extensive ablations on a series of step-by-step modifications of a vanilla RNN--linearization, diagonalization, stable exponential parameterization and normalization--that substantially improve performance, especially on tasks requiring long range reasoning. While our recurrence shares similarities with modern deep SSMs, our design does not rely on discretization of a latent continous-time system or on structured transition matrices. Instead our improvements directly follow from initialization and forward pass analysis arguments standard in the deep learning community, starting from a Glorot-initialized RNNs. Our final model matches the performance of modern deep state-space models (e.g. S4 or S5) on all LRA tasks.

## Acknowledgements

The authors would like to thank Michalis Titsias, Aleksandar Botev, James Martens and Yee Whye Teh for the interesting discussions and perspectives on our work.