Episodic Memory Theory for the Mechanistic Interpretation of Recurrent
  Neural Networks
Understanding the intricate operations of Recurrent Neural Networks (RNNs)
mechanistically is pivotal for advancing their capabilities and applications.
In this pursuit, we propose the Episodic Memory Theory (EMT), illustrating that
RNNs can be conceptualized as discrete-time analogs of the recently proposed
General Sequential Episodic Memory Model. To substantiate EMT, we introduce a
novel set of algorithmic tasks tailored to probe the variable binding behavior
in RNNs. Utilizing the EMT, we formulate a mathematically rigorous circuit that
facilitates variable binding in these tasks. Our empirical investigations
reveal that trained RNNs consistently converge to the variable binding circuit,
thus indicating universality in the dynamics of RNNs. Building on these
findings, we devise an algorithm to define a privileged basis, which reveals
hidden neurons instrumental in the temporal storage and composition of
variables, a mechanism vital for the successful generalization in these tasks.
We show that the privileged basis enhances the interpretability of the learned
parameters and hidden states of RNNs. Our work represents a step toward
demystifying the internal mechanisms of RNNs and, for computational
neuroscience, serves to bridge the gap between artificial neural networks and
neural memory models.
Arjun Karuvally, Peter Delmastro, Hava T. Siegelmann
# Episodic Memory Theory for the Mechanistic Interpretation of Recurrent Neural Networks

###### Abstract

Understanding the intricate operations of Recurrent Neural Networks (RNNs) mechanistically is pivotal for advancing their capabilities and applications. In this pursuit, we propose the Episodic Memory Theory (EMT), illustrating that RNNs can be conceptualized as discrete-time analogs of the recently proposed General Sequential Episodic Memory Model. To substantiate EMT, we introduce a novel set of algorithmic tasks tailored to probe the variable binding behavior in RNNs. Utilizing the EMT, we formulate a mathematically rigorous circuit that facilitates variable binding in these tasks. Our empirical investigations reveal that trained RNNs consistently converge to the variable binding circuit, thus indicating universality in the dynamics of RNNs. Building on these findings, we devise an algorithm to define a _privileged basis_, which reveals hidden neurons instrumental in the temporal storage and composition of variables -- a mechanism vital for the successful generalization in these tasks. We show that the privileged basis enhances the interpretability of the learned parameters and hidden states of RNNs. Our work represents a step toward demystifying the internal mechanisms of RNNs and, for computational neuroscience, serves to bridge the gap between artificial neural networks and neural memory models.

Recurrent Neural Networks Mechanistic Interpretability Memory Models

## 1 Introduction

Mechanistic interpretability aims to reverse engineer the intricate workings of neural networks that drive their behavior (Olah, 2022). At the core of its significance lies the pressing need for transparency and comprehension in an era where AI-driven systems have become ubiquitous in real-world applications (Christian, 2021; Sears, 2021; Bostrom, 2014; Muller and Bostrom, 2013). While these systems demonstrate remarkable proficiency, their inherently black-box nature often renders them inscrutable (Alishahi et al., 2019; Buhrmester et al., 2019; Fong and Vedaldi, 2017). Gaining a mechanistic understanding not only builds trust in such systems but also provides insights that can lead to refinement and innovation (Raukur et al., 2022). In essence, mechanistic interpretability is not just about demystifying AI; it's about harnessing its potential responsibly and efficiently. Current approaches to mechanistic interpretability focus on neural networks without any long-term temporal behavior. On the other hand, Recurrent Neural Networks (RNNs) pose a unique challenge - the task relevant information is stored in a hidden state that evolves over time. This raises the
question: _How is information reliably stored and processed in an evolving hidden state and how is the hidden state dynamics connected to the computations performed?_

To answer this question, we draw inspiration from memory models in computational neuroscience. First, we show how autonomously evolving RNNs can be interpreted as performing episodic memory retrieval in Section 3. This establishes the connection between existing RNN architectures and neurocomputational memory models, and forms the foundational principle for the Episodic Memory Theory (EMT). In Section 4, we formulate a class of algorithmic tasks to probe the variable binding behavior of RNNs. Section 5 introduces _variable memories_, which are linear subspaces capable of symbolically binding and recursively composing information, and use them to reverse engineer variable binding mechanisms in RNNs. The experimental results reveal consistent convergence to the proposed mechanisms adding evidence to the _universality hypothesis_ in mechanistic interpretability (Olah et al., 2020; Li et al., 2015). In Section 6, we build on the empirical results and propose an algorithm to construct a privileged basis grounded in the _variable memories_. For the first time, this basis unveils _hidden neurons_, which exist in superposition within the RNN hidden state, and actively participate in the storage and composition of variables.

## 2 Related Works

**Dynamical Systems Interpretation of RNNs**: Current approaches to interpret RNNs consider them as non-linear dynamical systems and apply linearization around fixed or slow-changing points to reveal their behavior (Marschall and Savin, 2023; Sussillo and Barak, 2013). The preliminary step in this analysis involves linearization around fixed points and slow-changing points found using optimization algorithms. The phase space flow is assembled piecemeal from each linearized region. The exploration of the long-term behavior of these regions is undertaken through the eigen-spectrum analysis of the corresponding linearized dynamical systems (Strogatz, 1994), providing insights into the dynamics of convergence, divergence, stability, or spiraling (Rowley et al., 2009; Kim, 1996). However, this method becomes intractable when there are many dimensions exhibiting non-convergent behaviors. The proposed EMT generalizes this approach and enables interpretation even when the number of non-converging dimensions is arbitrarily large.

**Mechanistic Interpretability**: Mechanistic interpretability seeks to reverse-engineer neural networks to expose the underlying mechanisms enabling them to learn and adapt to previously unconcountered conditions. The prevailing strategy involves examining the networks' internal "circuits" (Comny et al., 2023; Wang et al., 2022; Cammarata et al., 2020). Researchers have found that applying these interpretability methods to large networks, such as transformers (Vaswani et al., 2017) handling complex tasks in natural language processing and vision, faces the challenge of unclear features to be modeled in internal circuits. To address this, they create toy models with clearly defined features essential for task resolution. Probing models trained on toy tasks has resulted in supporting evidence for prevalent hypotheses. Some of the notable hypotheses are _universality_(Chughtai et al., 2023; Li et al., 2015) - models learn similar features and circuits across different models when trained on similar tasks, _bottleneck superposition_(Elhage et al., 2022) - a mechanism for storing more information than the available dimensions, and _composable linear representations_(Cheung et al., 2019) - the use of linear spaces in feature representation. Despite these advancements, current approaches remain confined to networks without a recurrent state like MLPs and transformers. Recurrent architectures, which maintain and recursively update a hidden state for task-related information processing, present a unique challenge - information needs to be stored and processed over time (Cruse, 1996; Hochreiter and Schmidhuber, 1997).

Figure 1: **Equivalence between Episodic Memory Models and Variable Binding**: **A**. Episodic Memory models aim to uncover the cognitive processes involved in the retrieval of subjective past experiences often stored as a temporal sequence of memory items. The illustration shows the retrieval of a personal experience when an apple is observed. **B**. The illustration shows the application of the Episodic Memory Theory, which poses that learning the addition operation over arbitrary numbers, a task involving variable binding, can be considered equivalent to episodic memory retrieval where the computations are performed over variables instead of predetermined memories. The abstract addition operation is stored in the synapses in the form of _how_ the variables interact with each other to produce the desired result.
**Neural Memory Models**: Developments in memory modeling have revealed links between deep neural networks and memory models. The first investigation of this link explored the relationship between Dense Associative Memory and Multi-Layer Perceptron (MLP) with various activation functions (Krotov and Hopfield, 2016). Later studies extended this connection to explain the practical computational benefits observed in neural architectures like transformers (Ramsauer et al., 2020). Recently, the traditional memory models capable of associative recall of single memories were expanded to retrieving memory sequences (Karuvally et al., 2022; Chaudhry et al., 2023). This expansion allows memories that previously did not interact in the single memory retrieval context to interact and produce complex temporal behavior (Kleinfeld, 1986; Kleinfeld and Sompolinsky, 1988). A fundamental assumption in memory modeling (in both single and sequence retrieval) is that the network's memories are predetermined and stored in the synapses. This assumption limits the models' applicability to mechanistic interpretability, which requires the symbolic binding of memories typically available only during inference.

In EMT, we will demonstrate that by lifting the fixed memory assumption in memory modeling, these memory models can be utilized to show how binding of external information happens in RNNs, revealing the synergistic relationship between the three fields - memory modeling, recurrent neural networks and mechanistic interpretability.

## 3 RNN as Episodic Memory

We show that RNNs can be viewed as a discrete-time analog of a memory model called General Sequential Episodic Memory Model (GSEMM) (Karuvally et al., 2022). To be applicable for the more general setting of RNNs, we slightly modify the GSEMM formulation with a pseudoinverse learning rule instead of the Hebbian learning rule for the synapses. This modification allows us to deal with more general (linearly independent vectors) memories than orthogonal vectors (Chaudhry et al., 2023; Personnaz et al., 1986). We discretize the continuous time GSEMM model using forward Euler approximation under the conditions that the timescale parameters are \(\mathcal{T}_{f}=1,\mathcal{T}_{h}=0,\) and \(\mathcal{T}_{d}=0\) (See Appendix A.2 for details). The final discrete system we obtain is \(V_{f}(t+1)=\Xi\,(I+\Phi^{\top})\,\Xi^{\dagger}\,\sigma_{f}(V_{f}(t))\). The columns of \(\Xi\) are the _stored memories_ of the model, and \((I+\Phi^{\top})=\Phi^{\top}\) is the matrix representing sequential memory interactions between these stored memories. The neural state variable is \(V_{f}\in\mathbb{R}^{d\times 1}\). The discrete system we derived is topologically conjugate to the update equations of an Elman RNN under the homeomorphic transformation if the norm of the matrix is bounded by 1. That is, if \(||\Xi\,\Phi^{\top}\,\Xi^{\dagger}||\leq 1\), we can consider a new state variable \(h=\sigma_{f}(V_{f})\) such that

\[h(t)=\sigma_{f}(\Xi\,\Phi^{\top}\,\Xi^{\dagger}h(t-1))\,. \tag{1}\]

This conjugate system has equations that are equivalent to an Elman RNN hidden state update equation without bias \(h(t+1)=\sigma_{f}(W_{hh}h(t))\).

A corollary to the equivalence between the sequence memory model and Elman RNNs is that if we decompose the weight matrix of the RNN in terms of the memories such that \(W_{hh}=\Xi\,\Phi^{\top}\,\Xi^{\dagger}\), the RNN computations can be interpreted as the retrieval of episodic memories temporally transitioning according to the rules encoded in \(\Phi\). This result along with the previous results (Krotov and Hopfield, 2016; Ramsauer et al., 2020) connecting memory models with feedforward neural networks forms the foundation of the Episodic Memory Theory of learned neural networks:

Figure 2: **Circuit of variable binding in an illustrative task of four variables, each with five dimensions**: **A**. The hidden state at time \(t\) has subspaces capable of storing external information in their activities. The colors depict the vector components (or activity) of the hidden state in the variable memory basis. The linear operator \(\Phi\) acts on the hidden state such that these activities are copied between variable memories except for \(\Psi_{4}\), which implements the linear operation \(f\). **B**. The \(N^{\text{th}}\) variable contents are read during the output phase using the appropriate linear operator \(W_{r}=\Psi_{N}^{\star}\).
```
\(0\leq\alpha\leq 1\)\(\triangleright\) number of time-steps in the input phase \(W_{hh},W_{uh},W_{r}\)\(\triangleright\) learned parameters of the RNN \(\Psi_{s}\leftarrow\alpha W_{uh}+(1-\alpha)W_{r}^{*}\) for\(k\in\{s-1,s-2,\ldots 1\}\)do \(\Psi_{k}\leftarrow\alpha W_{hh}^{s-k}W_{uh}+(1-\alpha)\left(\left(W_{hh}^{\top} \right)^{k}W_{r}^{*}\right)\) \(\Psi_{k}\leftarrow\Psi_{k}-EE^{*}\Psi_{k}\quad\forall E:\lambda(E)<1\)\(\triangleright\) Remove the components along transient directions. endfor \(\Psi\leftarrow[\Psi_{1};\ldots;\Psi_{s}]\) \(\Psi^{\downarrow}\leftarrow\text{PC}(\{h(t)\}-\Psi^{*}\,\{\tilde{h}(t)\})\)\(\triangleright\) Principle Components of \(\tilde{h}\) from simulations
```

**Algorithm 1** Algorithm for computing variable memories of trained linear RNNs

**The Episodic Memory Theory (EMT) poses that the inner workings of learned neural networks can be revealed by analyzing the learned inter-memory interactions and its effect in the space of stored memories (Figure 1).**

## 4 Variable Binding Tasks

Simple algorithmic tasks are very useful for mechanistic interpretability as they provide a controllable empirical setup compared to complex and noisy real-world scenarios. We formulate a class of tasks with input and output phases. At each timestep of the input phase, external information is provided _to_ the RNN. During the output phase, the RNN needs to utilize this external information to synthesize novel outputs at each time step that are subsequently read _from_ the network as output. This simple two-phase setup closely matches the behavior of NLP tasks like translation and conditional generative modeling where the noisy real-world features are abstracted out. Formally, the input phase consists of \(s\) total timesteps where at each timestep \(t\), a vector of \(d\) dimensions \(u(t)=\left(u^{1}(t),u^{2}(t),\ldots,u^{d}(t)\right)^{\top}\) is input to the model. We call the vector components \(u^{i}(t)\) the external information that needs to be _stored_ in the RNN hidden state. After the input phase is complete, the zero vector is continually passed as input to the model, so we say the RNN evolves autonomously (without any external input) during the output phase. The future states of the system during output phase evolve according to the following equation.

\[u(t)=f(u(t-1),u(t-2),\ldots u(t-s)),\quad t>s. \tag{2}\]

For analytical tractability, we add two restrictions to the variable binding tasks: (1) the composition function \(f\) is a linear function of its inputs. (2) the codomain of \(f\) and the domain of the inputs is binary \(\in\{-1,1\}\).

## 5 Variable Binding Circuit in RNN

Linearization approaches have shown promise in the analysis of RNNs in the neighborhood of fixed points (Sussillo and Barak, 2013). We build our model of variable binding on a linearized RNN defined by the following equations.

\[\begin{cases}h(t)=W_{hh}h(t-1)+W_{uh}u(t)\,,\\ y(t)=W_{r}\,h(t)\,.\end{cases} \tag{3}\]

We envision that any non-linear RNN can be converted to this form by finding fixed points and subsequent linearization (See Appendix A.4 for details). Here, \(W_{hh},W_{uh},W_{r}\) are linear operators, \(h(t)\) is the hidden state, \(u(t)\) is the input, and \(y(t)\) is the output. We use a simplifying assumption that \(W_{hh}\) has sufficient capacity to represent all the variables required for variable binding tasks (no requirement for bottleneck superposition (Elhage et al., 2022)). We further assume that \(h(0)\) is the zero vector. To handle the basis change suggested by the EMT view of RNNs, we use Dirac and Sinstein notations from abstract algebra (Appendix A.1). This new notation has two benefits - (1) We are able to formalize variable binding mechanisms independent of the basis, and (2) The notation enables a very clean and concise description of all the components of the circuit. Formally, we write any vector \(v\) as \(\ket{v}=\bra{\epsilon^{i}}\ket{v}\ket{\epsilon_{j}}=v^{i}\ket{\epsilon_{j}}\). \(\ket{\epsilon_{j}}\) is the basis in which the vector has vector components \(v^{i}=\bra{\epsilon^{i}}\ket{v}\cdot\bra{\epsilon^{i}}\) are the basis _covectors_ defined such that \(\bra{\epsilon^{i}}\ket{\epsilon_{j}}=\delta_{ij}\), and \(\delta_{ij}\) is the Dirac delta function.

In the new notation, the linear RNN of Equation 3 evolving autonomously is reformulated as follows.

\[\ket{h(t)}=\left(\xi_{\mu}^{i}\,\Phi_{\nu}^{\mu}\,(\xi^{\dagger})_{j}^{\nu}\, \ket{e_{i}}\bra{e^{j}}\right)\ket{h(t-1)} \tag{4}\]
Here \(\left|e_{i}\right\rangle\) is the standard basis vector which is typically used in simulations. To simplify this system further, we use a new basis \(\left|\psi_{\mu}\right\rangle=\xi_{\mu}^{i}\left|e_{i}\right\rangle\).

\[\left|h(t+1)\right\rangle=\left(\Phi_{\nu}^{\mu}\left|\psi_{\mu}\right\rangle \left\langle\psi^{\nu}\right|\right)\left|h(t)\right\rangle \tag{5}\]

This new basis allows us to treat the linearized RNN as applying a _single_ linear operator as opposed to three in the original formulation. In the new basis, the hidden state vector is \(\left|h(t)\right\rangle=h^{\psi_{\mu}}\left|\psi_{\mu}\right\rangle\). We pose that these components \(h^{\psi_{\mu}}\) (also called subspace _activity_) can be set to any external information for solving the task by appropriately interacting with the hidden state, thus behaving like variables in computation. The collection of vectors \(\{\Psi_{i}\}=\{\left|\psi_{\mu}\right\rangle:\mu\in\{(i-1)d,\ldots,id\}\}\) that defines a subspace where the variable is stored is called the \(i^{\text{th}}\)_variable memory_. The newly introduced concept of variable memory enables treating activity in the space collectively, simplifying reasoning about their behavior. In order to retain information, the \(\Phi\) operator must have necessary mechanisms to handle the storage of information over time. One possibility for this is the following linear operator \(\Phi\).

\[\Phi=\sum_{\mu=1}^{(N-1)d}\left|\psi_{\mu}\right\rangle\left\langle\psi^{\mu+d }\right|+\underbrace{\sum_{\mu=(N-1)d}^{Nd}\Phi_{\nu}^{\mu}\left|\psi_{\mu} \right\rangle\left\langle\psi^{\nu}\right|}_{f(u(t-1),u(t-2),\ldots,u(t-N))}. \tag{6}\]

The action of the operator on the hidden state is illustrated in Figure 1A. For all variable memories with index \(i\in\{2,3,4,\ldots N\}\), the information contents are copied to the variable memory with index \(i-1\). The operator then implements the function \(f\) defined in Equation 2 and stores the result in the \(N^{\text{th}}\) subspace. Any _linear_ function \(f\) of the history can be represented in this framework. This specific setting of the linear operator allows information to be stored in the \(N^{\text{th}}\) variable over \(N\) timesteps reliably for computing the solution to the problem.

**Reading Variables**: Once RNN has performed its computation, the computed information needs to be extracted. RNNs have a linear operator \(W_{r}\), which facilitates the reading of information from \(\left|h(t)\right\rangle\) at consecutive time steps. We propose that \(W_{r}\) has the following equation.

\[W_{r}=\Psi_{N}^{*}=\sum_{\mu=(N-1)d+1}^{Nd}\left|e_{\mu-(N-1)d}\right\rangle \left\langle\psi^{\mu}\right| \tag{7}\]

The reading operation reads the activity of the \(N^{th}\) subspace and outputs them in the standard basis (Figure 2B) for the output to be read out of RNNs.

We do not propose any form for \(W_{uh}\). This is because the operator has two roles to play in the RNN behavior. One role is to add the input information in the \(N^{th}\) subspace, and two, suppress the effect of \(\Phi\) operator during the input phase - \(\Phi\) actively computes \(f\) even though all the variables are not filled during the input phase. These conflicting roles make proposing a _linear_ form for \(W_{uh}\) intractable. In our experiments, we find that this is not an issue for describing the long term behavior of the RNN as \(W_{uh}\) only influences the input phase.

**Optimization**: One point to note is that our definition of variable memory till now is not optimized - there are dimensions in certain variables that are irrelevant for future computations and hence uncessary to be stored. This

\begin{table}
\begin{tabular}{l|l l l|l l l} \hline \hline Task & \multicolumn{4}{c}{hidden size: 64} & \multicolumn{4}{c}{hidden size: 128} \\  & L2: 0.0 & L2: 0.001 & L2: 0.1 & L2: 0.0 & L2: 0.001 & L2: 0.1 \\ \hline \(\mathcal{T}_{1}\) & — (0.97) & — (0.88) & — (0.50) & 0.0005 (1.00) & — (0.93) & — (0.50) \\ \(\mathcal{T}_{2}\) & 0.0075 (1.00) & — (0.85) & — (0.50) & 0.0055 (0.98) & 0.0031 (0.98) & — (0.50) \\ \(\mathcal{T}_{3}\) & 0.0026 (1.00) & 0.0010 (0.97) & — (0.50) & 0.0031 (0.98) & 0.0005 (1.00) & — (0.50) \\ \(\mathcal{T}_{4}\) & 0.0112 (0.94) & 0.0011 (1.00) & — (0.50) & 0.0022 (1.00) & 0.0006 (1.00) & — (0.50) \\ \hline \hline \end{tabular}
\end{table}
Table 1: **RNNs consistently converge to the variable binding model**: The table shows the MAE in the complex argument between the eigenspectrum of the predicted \(\Phi\) from the variable binding circuit and the empirically learned \(W_{hh}\) in \(4\) tasks across \(20\) seeds under different RNN configurations. This average error is indeterminate if the rank of the theoretical \(\Phi\) is different from the empirical \(W_{hh}\). Values in the brackets show the average test accuracy of the trained model. For models that have high test accuracy (\(>0.94\)), the error in the theoretically predicted spectrum is very low indicating consistent convergence to the theoretical circuit. A notable exception of this behavior is \(\mathcal{T}_{1}\) with hidden size\(=64\) and \(L2=0\), where the restricted availability of dimensions forces the network to encode variables in bottleneck superposition resulting in a low-rank representation of the solution.
redundant information can be safely discarded without any effect to the network behavior. Concretely, we propose that RNNs learn an optimized representation of the variables such that for a basis transform \(M\) where each standard basis vector is masked by mask \(m^{i}\in\{0,1\}\). \(M=\left|m_{i}e_{i}\right\rangle\left\langle m^{j}e^{j}\right|\):

\[\arg\min_{m^{i}}\sum_{i}m^{i}\,,\text{such that}\,,\,rank(M^{T}\Phi M)=rank(\Phi) \tag{8}\]

The final optimized basis is the minimum amount of information that needs to be retained for computing the function \(f\) safely over time. For the rest of the paper, we will always consider this optimized basis, unless explicitly specified.

## 6 Algorithm for Computing Variable Memories

The mechanisms we elucidate suggest that _variable memories_ can be treated as a privileged and interpretable basis - a special basis where RNN behavior is interpreted as forming variables that are stored and processed. To compute variable memories, we compare the learned hidden weights \(W_{hh}\) to the interaction matrix \(\Phi\) of the theoretical model in experimental settings. We view the hidden state space in the new basis consisting of the \(\Psi=[\Psi_{1};...;\Psi_{s}]\). We consider also an additional orthogonal basis \(\Psi^{\perp}\) for the remaining dimensions of the space not currently explained by the variable binding circuit in Section 5. Once this basis is defined, the action of the learned weights on the variable memories can be extracted from the learned \(W_{hh}\) using the basis transformation \(\Phi_{\text{learned}}^{\top}=\Psi^{*}W_{hh}\Psi\). Any interaction between the variable memories and the non-memory space will be encoded in the matrices \(\Psi^{*}W_{hh}\Psi^{\perp}\) and \((\Psi^{\perp})^{*}W_{hh}\Psi\).

Building on the intuition from the linear model, we use the learned weights \(W_{hh}\), and \(W_{r}\) to estimate the \(\Psi_{k}\) for the variable memories of the trained RNN. The read weights \(\Psi_{s}=W_{r}^{*}\) define one of the variable memory, and all other subspaces can be found simply by propagating these dimensions _forward_ in time: \(\Psi_{k}=\Phi^{s-k}\Psi_{s}=W_{hh}^{s-k}W_{r}^{*}\). Although the variable memories are defined based on a linear evolution assumption, we found that the method of power iterating \(W_{hh}\) was effective in defining a variable memory space for even nonlinear RNNs. We specifically selected to define the variable memory dimensions \(\Psi_{k}\) by propagating \(W_{r}^{*}\) forward in time:

\[\Psi_{s}=W_{r}^{*},\qquad\Psi_{k}=W_{hh}^{s-k}W_{r}^{*}\quad\text{for}\,\,k<s \tag{9}\]

We also removed the projection of each \(\Psi_{k}\) onto the eigenvectors of \(W_{hh}\) whose eigenvalues were less than 1 in magnitude since they do not contribute to the long-term behavior but may interfere with the basis definition. The theorized interaction matrix \(\Phi\) has eigenvalues that sit on the unit circle in the complex plane. This ensures that iterating \(\Phi\) does not cause the hidden state to contract or expand in the linear model. We will expect that any eigenvectors of the learned \(W_{hh}\) with eigenvalue inside the unit circle corresponding to _transient_ behavior associated with dimensions whose activity dies out at later timesteps during inference. To obtain the orthogonal basis \(\Psi^{\perp}\) for the rest of the hidden space, we applied principal component analysis (PCA) to hidden state dynamics during inference after removing the projection onto the variable memories. The algorithm to compute variable memories from a linearized RNN is summarized in Algorithm 1.

## 7 Results

### Rnns consistently converge to the variable binding model

In Section 5, we proposed an exact, mathematically rigorous circuit for variable binding capable of storing and processing information over time. To substantiate that this mechanism is indeed learned by RNNs, we trained various RNN configurations, differing in hidden sizes and regularization penalties. After training, the RNNs were linearized, and the eigen-spectrum of the learned \(W_{hh}\) matrix is compared with the theoretical \(\Phi\), as defined in Equation 6. If RNNs learn a representation in alignment with our model, both operators, i.e., the learned \(W_{hh}\) and theoretical \(\Phi\), are expected to share a portion of their spectrum as they are similar matrices (i.e they differ by only a basis change). In this comparison, it's pertinent to evaluate solely the arguments of the spectrum, disregarding the magnitude. The rationale behind this exclusion lies in what the magnitude tells about the dynamical behavior, which portray whether a linear dynamical system is diverging, converging, or maintaining consistency along the eigenvector directions. RNNs typically incorporate a squashing non-linearity, such as the tanh activation function, which restricts trajectories that divergence to infinity. Essentially, provided the eigenvalue magnitude remains \(\geq 1\), the complex argument solely determines the overall dynamical behavior of the RNN. Table 1 depicts the average absolute error when various RNN models are trained across \(4\) distinct tasks. The RNNs exhibiting robust generalization tend to consistently converge towards the circuit mechanisms detailed in Section 5. The results also highlight a particular setup - \(7_{1}\) with hidden size\(=64\) and \(L2=0.0\), which, while not converging to the theoretical mechanism, still achieves high generalization accuracy. \(7_{1}\) stands out as it necessitates exactly \(64\) dimensions per the optimized theoretical model, matching the exact
dimensionality available in the RNN. In this instance, RNNs exhibit a form of bottleneck superposition, a scenario not accommodated by the variable binding circuit yet. Nevertheless, provided there are sufficient dimensions to encapsulate the variable binding circuit, the RNNs tend to converge to it.

#### Variable memory reveals hidden neurons storing variable information

Variable memories define bases for the storage and processing of variable information within RNNs if the RNNs follow the variable binding circuit. Building on the empirical results we obtained on the consistent convergence to the variable binding circuit, we compute variable memories using Algorithm 1 for models trained in Repeat Copy (\(\mathcal{T}_{1}\)). In the Repeat Copy task, the RNN must repeatedly output a stream of inputs provided during the input phase. The simulated

Figure 4: **EMT enables human interpretability of learned RNN parameters**: The learned weights when visualized in the variable memories result in a form that is human-interpretable. For RNNs trained on two sample tasks \(\mathcal{T}_{1}\) (**A** left) and \(\mathcal{T}_{2}\) (**B** right), the weight matrix \(W_{hh}\) converts into a form that reveals internal mechanisms of how RNNs solve the task. For both tasks, the variables with index \(<8\) copies its contents to the preceding variable. Variable \(8\) actively computes the function \(f\) applied on all the variables stored in the hidden state using the variable binding circuit. For \(\mathcal{T}_{1}\), it is a simple copy of the \(1^{\text{st}}\) variable, and for \(\mathcal{T}_{2}\), it is a linear composition of all the variables Notably, the circuit for \(\mathcal{T}_{2}\) shows an optimized basis where all the irrelevant dimensions are absent.

Figure 3: **EMT reveals hidden neurons storing task relevant information over time**: **A**. In the repeat copy task (\(\mathcal{T}_{1}\)), the RNN needs to repeatedly produce an input sequence that is presented. A typical trained hidden state after providing the input does not show any meaningful patterns connected to the input. **B**. The same hidden states when visualized in the variable memories reveal the input information being stored as variables and processed according to the variable binding circuit. The actual hidden state is in a superposition of these _hidden_ variable memory activities.
hidden states of learned RNNs are visualized by projecting the hidden state in the variable memories: \(\tilde{h}=\Psi^{*}h\). The results shown in Figure 3 reveal that the hidden state is in a superposition of hidden neurons that actively store each variable required to compute the function \(f\) at all points in time. The basis transformation helps to disentangle these superposed variables from the hidden state so that they are easily visualized.

#### Variable memories enable human interpretability of learned weights

In addition to revealing hidden neurons that store and process information over time, variable memories can also be used as bases to view the operations of the learned matrices. The variable memories are carefully constructed such that \(W_{hh}\) converts to the underlying \(\Phi\) when viewed in the basis. As observed in the Figure 4, viewing the operations of \(W_{hh}\) in the new basis enables human-interpretability in terms of the variable binding circuit and provides a way to influence or "fix" RNN behavior after training. This "fixing" operation can be imagined as changing specific weights of the extracted \(\Phi\) to improve either variable storage properties, or problems computing \(f\).

One practical consideration when computing variable memories is the sensitivity of these results to minor changes in basis definition. It may be found that in some practical cases, the basis transformation does not reveal these interpretable representations even though the spectrum converges to the theoretical circuit. This deviation from the theory is a result of the sensitivity of the basis definition to minor errors in the pseudo-inverse required to compute the dual. It is possible that new empirical procedures may be developed to improve this computation in the future.

## 8 Discussion

We presented a novel perspective on Recurrent Neural Networks (RNNs), framing them as dynamical systems performing sequence memory retrieval. We introduced the concept of "variable memories," linear subspaces capable of symbolically binding and recursively composing information. Our approach addresses the limitations of current methods in understanding RNNs, particularly their inscrutability as 'black boxes' and the complexity of spectral analysis in high-dimensional task spaces. We presented a new class of algorithmic tasks that are designed to probe the variable binding behavior of RNNs. We presented a circuit mechanism that is capable of recursively storing and composing variables and show that trained RNNs consistently converge to this circuit pointing towards universality in the learned models. Building on the empirical evidence, we used variable memories to derive a privileged basis that, for the first time, revealed hidden neurons actively involved in information processing in RNNs. Further, using variable memories, we viewed the learned parameters of an RNN in a human-interpretable manner, enabling reasoning about RNN behavior as repeatedly copying and composing variables.

The Episodic Memory Theory and variable memories are versatile enough to be broadly applicable in various scenarios, offering valuable insights for researchers designing new algorithms. One practical application of this analysis can be in the development of continual learning algorithms, which can restrict gradients to pre-existing variable memory spaces to minimize catastrophic forgetting of prior tasks. Additionally, in task composition--where a new task is a combination of two existing tasks--the linear spaces of each task can be linearly combined to efficiently solve the composite problem. Another possible application is transfer learning, where task knowledge is shared between networks. The Episodic Memory Theory suggests that variable memories and their interactions are the essential components for knowledge transfer, allowing the remaining network dynamics to be disregarded or easily relearned, streamlining the transfer process.

With these diverse applications, it is also important to recognize certain inherent limitations to the approach. One of the limitation is that the analysis is primarily restricted to linear dynamical systems. Although an accurate representation of the qualitative behavior within small neighborhoods of fixed points can be found for non-linear dynamical systems, the RNNs have to be confined to these linear regions for the approach to be applicable. It is an interesting behavior that models consistently converge to these linearized regime, at least for the tasks outlined in Section 4. The second limitation of the variable binding model is that external information is stored as a _linear_ superposition of variable memories in the hidden state. Our results indicates that the role of non-linearity in encoding external information may be minimal for the toy tasks. However, we have observed that when the number of dimensions of the linear operator \(W_{hh}\) is not substantially large compared to the task's dimensionality requirements (bottleneck superposition) or when the regularization penalty is high, the RNN can effectively resort to non-linear encoding mechanisms to store external information (Appendix B.3). Overcoming these limitations of non-linearity will be an interesting direction to pursue in future research.