How noise affects memory in linear recurrent networks
JingChuan Guan Tomoyuki Kubota Yasuo Kuniyoshi Kohei Nakajima


APS/123-QED













kan@isi.imi.i.u-tokyo.ac.jp








Intelligent Systems and Informatics Laboratory, Graduate School of Information Science and Technology, The University of Tokyo, 7-3-1 Hongo, Bunkyo-Ku, Tokyo, Japan.
 



             

§ ABSTRACT 
The effects of noise on memory in a linear recurrent network are theoretically investigated.
Memory is characterized by its ability to store previous inputs in its instantaneous state of network, which receives a correlated or uncorrelated noise.
Two major properties are revealed:
First, the memory reduced by noise is uniquely determined by the noise's power spectral density (PSD). 
Second, the memory will not decrease regardless of noise intensity if the PSD is in a certain class of distribution (including power law).

The results are verified using the human brain signals, showing good agreement.




  Keywords
noise, memory, information processing, RNN, autocorrelation.




                              
How noise affects memory in linear recurrent networks
    Kohei Nakajima
    September 9, 2024
=====================================================



Introduction.—Understanding the effects of noise on information processing is a crucial problem in comprehending any physical system.
For instance, in the field of quantum computation, 
the interaction between a quantum device and environment occurs with noise in the device, which impairs the accuracy of quantum computation <cit.>. 
In nature, living organisms process information from the external environment
by extracting the necessary inputs from a large amount of signals containing noise, where noise works as a type of disturbance.

Short-term memory plays an essential role among various types of information processing,
which requires past input history.
This includes various tasks required in daily lives:
mental calculation <cit.>, recalling brief number of items <cit.>, and motor controls involving precise time perception <cit.>.
Additionally, many recent studies have reported that various types of physical systems can be utilized as computational resources <cit.>
where their short-term memories are exploited to solve tasks <cit.>.
In the theoretical studies exploring recurrent neural networks (RNNs),
the memory has been characterized by memory function (MF) <cit.>
and information processing capacity <cit.>, which can comprehensively reveal the memory in the network
<cit.>.
Using these measures, 
the dependency of memory on parameters has been investigated. 
Some studies <cit.>
numerically revealed that its network topology <cit.> affects the memory.
From the perspective of noise, other researches <cit.> have reported that random noise reduces the past inputs held in the network and that,
as the noise-to-signal ratio (NSR) increases,
the reduction becomes more critical.
Therefore, the random noise dominantly has the negative impact on the information processing based on the short-term memory.

Those researches <cit.> have focused on the case of random noise,
which is termed independent and identically distributed (i.i.d.) noise;
however, real-world systems receive not only i.i.d. noise but also correlated noise. 
For example, 1/f-like noise <cit.>,
whose power spectral density (PSD) follows 1/f^β, can be ubiquitously observed.
Accordingly,
it is imperative to analyze the effects of general noises on information processing besides uncorrelated noise.

In this paper, we theoretically reveal properties of general noise regarding its influence on information processing in a linear RNN that receives input and noise.
We derive an analytical solution of MF to investigate the effects of noise on memory.
Based on the analytical solution, we show two properties. 
First, we derive a simplified representation of the total memory by taking sufficiently large number of nodes to reveal the effect induced by noise correlation.
Second, we introduce a novel way to express MF, and clarify the impact of noise intensity on MF.
We demonstrate these effects of noise using experimental data obtained from the human brain. 


Methods.—We consider an RNN with a fixed internal weight, which is called an echo state network (ESN) <cit.>.
A discrete-time RNN updates the state as follows:
x_t+1 = f(Wx_t + w_1u_t+1 + w_2v_t+1),
where x_t∈ℝ^N, u_t∈ℝ,
and v_t∈ℝ denote the state, input, and noise at t-th step, respectively,
while N is the number of nodes; f(·) is the activation function and W∈ℝ^N× N is the internal weight matrix; and 
w_1, w_2∈ℝ^N are the weight vectors of input and noise, respectively. 
In the present paper, we have adopted the uniformly random input u_t∈[-1, 1] and the linear activation function f(·). 

We evaluate the short-term memory in the RNN using the MF <cit.>,
which represents how well the past injected input u_t-τ, could be emulated by a linear approximation with the network state:
û_t-τ = w_out^⊤x_t,
where τ is the delay from the current time.
After determining the readout weight vector w_out by minimizing a loss function of the mean squared error (MSE) 1/T∑_t=1^T (û_t-τ-u_t-τ)^2,
the MF is obtained as follows:

    M[u_t-τ] = 1 -
    min_w_out⟨ (û_t-τ-u_t-τ)^2 ⟩/⟨ u_t-τ^2⟩ (≤ 1),

where T is the sampled time length and 
⟨·⟩ denotes the time average.
The upper bound 1 is satisfied when the system has fully memorized the input required to reconstruct the target. 
The sum of the MF with respect to all τ represents the memory capacity (MC) of the full system, 
=∑_τ=0^∞ M[u_t-τ].
The upper bound of  is the number of linearly independent time series in the state, which is called rank and ideally N.

To demonstrate an applicable range of our results, 
we have utilized not only noise models <cit.> but also experimental data of human brain activities that show a 1/f-like property <cit.>.
We adopted the EEG signals <cit.> measured from three brain areas:
midline frontal (Fz), vertex (Cz), midline parietal (Pz). 
Signals in brain are expected to have a large amount of information, which include not only memory but irrelevant signals coming from different regions of the brain.
In the current study, we have injected both input and EEG signal to the RNN and regard the EEG as noise. 

Results.—In the present study, we derive the MF and MC of RNNs that receive input and noise by direct substitution. 
We begin the derivation from the following MF <cit.>, which is equivalent to Eq. (<ref>):

    M[u_t-τ]    =
    U_τ^⊤X(X^⊤X)^-1X^⊤U_τ/U_τ^⊤U_τ,

where X∈ℝ^T× N is a matrix whose column represents the states time series and U_τ is the delayed input series.
In this derivation, we impose two assumptions on the system:
(i) the input u_t and the noise v_t are uncorrelated;
(ii) input and noise share the same weight vector, that is w = w_1 = w_2. 
Under the conditions, we derived analytical solutions of MF and MC using a matix H and an autocorrelation matrix of u+v, C_uv = E + r C_v, where r = ⟨ v^2 ⟩ / ⟨ u^2 ⟩ is NSR and C_v is the autocorrelation matrix of v. 
H is defined as
H = 
[ H_K-1 H_K-2     ⋯   H_0 ], where 
H_τ = 
[ λ_1^τ λ_2^τ     ⋯ λ_N^τ ]^⊤, 
K is the time length and should be sufficiently large,
and λ_i is the i-th eigenvalue of W.
The (i,j) components of C_v is defined as (C_v)_ij=C(|i-j|), 1≤ i,j ≤ K, i,j ∈ℕ, 
where C(τ) is the autocorrelation function of v normalized by the variance.
We call this an analytical solution involving inverse matrix (ASI):

    M[u_t-τ]    =
    H_τ^⊤
    (HC_uvH^⊤ )^-1H_τ, 
       = ∑_τ=0^K-1 M[u_t-τ] =
    tr[
    H^⊤
    ( HC_uvH^⊤ )^-1H],

which are derived in Sec. 1 of Supplementary material.
From ASI, we could confirm that
the MF and MC only depend on the eigenvalues λ_i and the autocorrelation of v 
(detailed investigation in Section 2 of Supplementary material).
Based on ASI, we derived the following analytical results. 


First, we focus on the MC of a sufficiently large RNN (i.e., the number of nodes and time-series length are infinite N=K→∞). 
Under this assumption, we can derive the following formula of C_ sum, u from Eq. (<ref>) under the assumption that the rank of system is full:

    = ∑_i=1^N1/ 1 + r λ[C_v]_i,

where λ[C_v]_i is the eigenvalues of C_v (derived in Section 4 of Supplementary material).
This formula yields two important properties of MC.
(i) The MC becomes independent of λ_i and is determined only by C_v, which are equivalent to the PSD of v <cit.>.
The result suggests that, in addition to NSR, the PSD is also crucial in determining the effect of noise, which are numerically verified in Section 4 of Supplementary material.
(ii) The MC of the infinite-dimensional RNN with an arbitrary autocorrelated noise is greater than that with random noise.
We can explain this property by introducing the minimum value of :

    ≥1/1+rN

which is proven under two conditions that tr[ C_v ]=K and the function 1/1+x is downward convex
(Theorem 4. 1 in Supplementary material).
The minimum value is the MC with any type of random noise,
meaning that equality holds if the PSD of v is flat and that  becomes larger if the PSD is not flat. 





Second, we focus on MF of both the input and noise.
The preceding result has focused on the memory of input held in RNNs, which also keeps noise as memory; 
however, the MF is conventionally defined with i.i.d. signals and that,
for an autocorrelated noise, has not been derived thus far. 
To address this problem, 
we define the MF from another perspective <cit.>, 
which is the square norm of the coefficient in the state expanded by orthonormal bases and is equivalent to the definition of Eq. (<ref>). 
We use an autocorrelated noise model v_t represented by a sum of two factors: i.i.d. noise ñ_t and time-dependent function a_t.
A delayed noise of v_t includes delayed series of ñ_t and orthogonal basis ã_t obtained by decomposing a_t.
Subsequently, k-th delayed noise v_t-k can be decomposed into 
v_t-k = c^n_kñ_t-k + ∑_i=0^k c^a_kiã_t-i,
where the two elements have time averages < ñ_t> = < 
ã_t> = 0.
The bases {ñ_t} are innately defined in the noise and 
{ã_t-i} are defined using the Gram–Schmidt orthogonalization:

    â_t-k = a_t-k - 
        ∑_i=0^k-1< ã_t-ia_t-k> ã_t-i, 
        ã_t-k = â_t-k/||â_t-k||.

In accordance with the polynomial expansion of state, we have introduced the MFs about the system with an i.i.d. input u_t and an autocorrelated noise v_t.
Because the RNN includes only linear terms, the state time series is expanded by three types of time-series bases: {u_t-τ}, {ñ_t-τ}, and {ã_t-τ}. 
The delayed input time-series {u_t-τ} organize linearly independent bases because the input at each step is i.i.d.
Additionally, because of the orthogonalization, {ñ_t-τ} and {ã_t-τ} are also appended to the bases of the orthogonal system. 
As a result, {u_t-τ}, {ñ_t-τ}, and {ã_t-τ} span the complete orthonormal system for the linear RNN.
Using these bases, we can perfectly expand the state,
and define the MFs on the bases u_t-τ, ñ_t-τ, and ã_t-τ 
as M[u_t-τ], M[n_t-τ], and M[a_t-τ],
respectively (see Sec. 5 of Supplementary material for derivation).
We can define  and  as the MC of each signal, and  as the total MC of the system, which can be computed as
= ∑_τ=0^∞ M[u_t-τ],
= ∑_τ=0^∞ M[n_t-τ] + M[a_t-τ],
and =  +.
According to the completeness property <cit.>, 
= N holds because the system depends only on the past input and noise series that span the complete system.


This definition of MF enables us to elucidate the limitations of autocorrelated noise effect. 
Comparing the MF of i.i.d. elements and that of time-dependent function,
their difference can be characterized by the number of bases.
In a system where infinite time has passed, the number of bases u_t-τ would be infinite because the delayed time series would be linearly independent.
For the same reason, the number of bases {n_t-τ} is also infinite,
while that of ã_t-τ can be both finite (e.g., sinusoidal curve) and infinite (e.g., 1/f noise).
Here, we begin with the assumption that the number N_a of bases ã_t-τ is finite,
showing that, with a sufficiently large N, the sum of MFs about a_t-τ would become 0:

    lim_N→∞1/N∑_τ=0^∞ M[a_t-τ] = 0

which is derived based on the following generating procedure of the base ã_t-τ.
A base of ã_t-τ is newly generated when τ increments (Eq. <ref>). 
Subsequently, the new base is removed if the current input can be expressed by linear combination of existing orthogonal polynomials. This procedure is repeated until the new base does not appear, and we finally obtain the finite number N_a of bases in some cases.
For example, if a_t = cos(ω t), the bases are cos(ω t) and sin(ω t), indicating N_a=2. 
This mechanism suggests that RNNs integrate the existing memory of the past inputs and overlapped information of the current input due to autocorrelation.
With a sufficiently large N, we obtain
∑_τ=0^∞ M[a_t-τ] ≤ N_a,
which produces
Eq. (<ref>)
(MFs of a in Fig. <ref>a, b).
Accordingly, combined with the completeness property, we derive lim_N→∞ 1/N ∑_τ=0^∞( M[u_t-τ] + M[n_t-τ] ) = 1.
In addition, the MFs of these i.i.d. elements can be characterized by Eq. (<ref>),
such that the variances ⟨ u^2 ⟩ and ⟨ n^2 ⟩ determine the ratio between the MFs of i.i.d. elements (see Sec. 5 of Supplementary material for derivation),
indicating that they are just scaled (MFs of u and n in Fig. <ref>a–c).
As the MC of a_t-τ gradually becomes 0,
increase in N leads to the enhancement of /N , 
which suggests that the inhibitory effect of noise becomes smaller.
In the infinite-dimensional RNN, 
/N (/N) will converge to a certain value determined by the ratio of ⟨ u^2 ⟩ and ⟨ n^2 ⟩ (Fig. <ref>e): 

    lim_N→∞/N = ⟨ u^2 ⟩/⟨ u^2 ⟩+⟨ n^2 ⟩.

If the noise v is composed only of random components n_t-τ,
the ratio between the  and  is independent of N
and there is no enhancement of  caused by increasing N (Fig. <ref>c, d). 
In a system with finite N_a (Fig. <ref>d),
we can confirm that, as the proportion of n_t-τ within v decreases,
the disturbance effect of noise becomes smaller, which elucidates the result of Eq. (<ref>). 
Conversely, if v is composed only of a_t-τ, the noise has a little inhibitory effect independent of r:

    lim_N→∞/N=1 
         (
        lim_N→∞/N=0
        ) .






Furthermore, even if N_a is infinite,  could fulfill Eq. (<ref>). 
We proved Eq. (<ref>) in two cases of
(i) lim_N→∞ < ∞ and
(ii) lim_N→∞ = ∞ under conditions.
(i) Under the d'Alembert's test condition of lim_n→∞λ̂[C_v]_n+1/λ̂[C_v]_n<1, Eq. (<ref>) holds, 
where λ̂[C_v]_n expresses a sorted version of λ[C_v]_n in descending order.
(ii) In addition, even if lim_N→∞ = ∞,
 could hold Eq. (<ref>).
For example, we have proven the case of 1/f^β noise (β≥ 1) (see proofs in Sec. 6 of Supplementary material).
Both the examples of PSDs shown here are characterized by the skewed distribution.
We say that a PSD is skewed if the distribution of λ̂[C_v]_n satisfies Eq. (<ref>). 
Note that, since PSD represents the magnitude of coefficients of the Fourier series, in which the state is expanded by linearly independent bases of sinusoidal wave,
these results show that the distribution of the magnitude determine the noise effects.
To demonstrate cases in which a very large noise does not hinder information processing in RNN, 
we examined the dependency of normalized MC (/N) on the parameters β and r.
Counterintuitively, even if the NSR is large (e.g., r = 100), 
the RNN keeps /N ≈ 1 with a large β (>2.0). 
Note that a blue region with β≥ 1 does not satisfy Eq. (<ref>) because /N = o(log(log N)/log N)
slowly converges to 0, implying that N ∼ 10^4 is not sufficiently large.
To mitigate the disturbance, the system requires N∼ 10^10^2,
implying that the real-world systems (e.g., the number of neurons in brain N∼ 10^7 <cit.>) cannot fully hold the MC.




Finally, we numerically verified the effects of noise on the memory using EEG signals from three human brain areas <cit.>.
Comparing the MF and MC of the systems using the autocorrelated noise with those using noises randomly shuffled in time direction (Fig. <ref>),
the former consistently have significantly higher values than the latter. 
Even though the noise intensity is much larger than the input intensity (r=100), the MC keeps / > 0.7 (“Fz” in Fig. <ref>, right), where  is numerically normalized by .
As N increases, / seems to converge to a fixed value,
which may indicate the ratio of ⟨ u^2 ⟩ and ⟨ n^2 ⟩ according to Eq. (<ref>).
The three different convergent values may imply that the ratio of random and autocorrelated elements varies among the regions of the brain.
In addition, by incorporating a threshold to determine where the convergence occurs, we can find a sufficiently large N to bring out the maximum MC under the noise.
For example, under the threshold in which / perturbed smaller than 5× 10^-3, those N of “Fz”, “Cz”, and “Pz”are computed as 40, 39, and 37, respectively.
Overall, we could confirm that the autocorrelated noise in the real-world always has smaller disturbance than random noises and that the memory of the input is retained even with the very strong irrelevant signals.


There are two limitations in our study.
We assumed that the input weights were shared between the input and noise.
This can be interpreted as a setting in which we inject the input containing noise to the RNN and ignore other noises. 
This enables us to analytically evaluate the effects of the initially added noise that entails any intensity and any correlation. 
We need to consider the noises coming from other input pathways in the future.
Next, we have only focused on linear RNNs in the current study.
However, previous researches have confirmed the presence of both linear and nonlinear information processing in real-world systems <cit.>,
indicating that nonlinear cases should be investigated in future.



Summary.—In the present study,
we used a linear RNN with random input and noise,
including correlated ones, to investigate the effects of noise on memory in general.
We derived an analytical solution of MF and MC dependent only on the autocorrelation of noise and the eigenvalues of internal weight,
which revealed the following three results:
First, in infinite-dimensional systems, 
the MC becomes independent of the internal weight and is determined only by the PSD of noise.
Therefore, our results hold for any type of linear RNNs regardless of whether its internal weights are random or trained. 
By using this solution, we proved that the MC with autocorrelated noise is larger than that with a random noise. 
Second, noise has little inhibitory effects in a sufficiently large system, regardless of its intensity,
if the intensities of linearly independent bases in the noise satisfy a certain condition.
In a general case that the number of base is infinite,
this result is satisfied when the intensities have a skewed distribution that decays at a sufficiently fast rate such as 1/f noise.
Actually, this condition is also effective under the case with a finite number of base because its distribution always fulfill the skewed condition such as sinusoidal noise.
Third, we used EEG data to verify the above analytical results.
We demonstrated that, as a form of noise, the EEG series had a small inhibitory effect,
despite its strong intensity.
Moreover, different brain regions showed different ratio of random component and system size to bring out the maximum MC.
From these results, our research has clarified the effects of general noises on information processing,
providing analytical explanation.



