Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs. iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD). We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation. We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.
 Recurrent neural networks (RNNs) in each round store a hidden state vector, h m ∈ R D , and upon receiving the input vector, x m+1 ∈ R d , linearly transform the tuple (h m , x m+1 ) and pass it through a memoryless non-linearity to update the state over T rounds. Subsequently, RNNs output an affine function of the hidden states as its prediction. The model parameters (state/input/prediction parameters) are learnt by minimizing an empirical loss. This seemingly simple update rule has had significant success in learning complex patterns for sequential input data. Nevertheless, that training RNNs can be challenging, and that performance can be uneven on tasks that require long-term-dependency (LTD), was first noted by Hochreiter (1991) , Bengio et al. (1994) and later by other researchers. Pascanu et al. (2013b) attributed this to the fact that the error gradient back-propagated in time (BPTT), for the time-step m, is dominated by product of partials of hiddenstate vectors, T −1 j=m ∂hj+1 ∂hj , and these products typically exhibit exponentially vanishing decay or explosion, resulting in incorrect credit assignment during training and test-time. Rosenblatt (1962) , on whose work we draw inspiration from, introduced continuous-time RNN (CTRNN) to mimic activation propagation in neural circuitry. CTRNN dynamics evolves as follows: τġ(t) = −αg(t) + φ(U g(t) + W x(t) + b), t ≥ t 0 . ( Here, x(t) ∈ R d is the input signal, g(t) ∈ R D is the hidden state vector of D neurons,ġ i (t) is the rate of change of the i-th state component; τ, α ∈ R + , referred to as the post-synaptic time-constant, impacts the rate of a neuron's response to the instantaneous activation φ(U g(t) + W x(t) + b); and U ∈ R D×D , W ∈ R D×d , b ∈ R D are model parameters. In passing, note that recent RNN works that draw inspiration from ODE's (Chang et al., 2019) are special cases of CTRNN (τ = 1, α = 0). Vanishing Gradients. The qualitative aspects of the CTRNN dynamics is transparent in its integral form: This integral form reveals that the partials of hidden-state vector with respect to the initial condition, ∂g(t) ∂g(t0) , gets attenuated rapidly (first term in RHS), and so we face a vanishing gradient problem. We will address this issue later but we note that this is not an artifact of CTRNN but is exhibited by ODEs that have motivated other RNNs (see Sec. 2). Shannon-Nyquist Sampling. A key property of CTRNN is that the time-constant τ together with the first term −g(t), is in effect a low-pass filter with bandwidth ατ −1 suppressing high frequency components of the activation signal, φ((U g(s )) + (W x(s )) + b). This is good, because, by virtue of the Shannon-Nyquist sampling theorem, we can now maintain fidelity of discrete samples with respect to continuous time dynamics, in contrast to conventional ODEs (α = 0). Additionally , since high-frequencies are already suppressed, in effect we may assume that the input signal x(t) is slowly varying relative to the post-synaptic time constant τ . Equilibrium. The combination of low pass filtering and slowly time varying input has a significant bearing. The state vector as well as the discrete samples evolve close to the equilibrium state, i.e., g(t) ≈ φ(U g(t) + W x(t) + b) under general conditions (Sec. 3). Incremental Updates. Whether or not system is in equilibrium, the integral form in Eq. 2 points to gradient attenuation as a fundamental issue. To overcome this situation, we store and process increments rather than the cumulative values g(t) and propose dynamic evolution in terms of increments. Let us denote hidden state sequence as h m ∈ R D and input sequence x m ∈ R d . For m = 1, 2, . . . , T , and a suitable β > 0 τġ(t) = −α(g(t) ± h m−1 ) + φ(U (g(t) ± h m−1 ) + W x m + b), g(0) = 0, t ≥ 0 (3) Intuitively, say system is in equilibrium and −α(µ(x m , h m−1 ))+φ(U µ(x m , h m−1 )+W x m +b) = 0. We note state transitions are marginal changes from previous states, namely, h m = µ(x m , h m−1 ) − h m−1 . Now for a fixed input x m , as to which equilibrium is reached depends on h m−1 , but are nevertheless finitely many. So encoding marginal changes as states leads to "identity" gradient. Incremental RNN (iRNN) achieves Identity Gradient. We propose to discretize Eq. 3 to realize iRNN (see Sec. 3). At time m, it takes the previous state h m−1 ∈ R D and input x m ∈ R d and outputs h m ∈ R D after simulating the CTRNN evolution in discrete-time, for a suitable number of discrete steps. We show that the proposed RNN approximates the continuous dynamics and solves the vanishing/exploding gradient issue by ensuring identity gradientIn general, we consider two options, SiRNN, whose state is updated with a single CTRNN sample, similar to vanilla RNNs, and, iRNN, with many intermediate samples. SiRNN is well-suited for slowly varying inputs. Contributions. To summarize, we list our main contributions: (A) iRNN converges to equilibrium for typical activation functions. The partial gradients of hiddenstate vectors for iRNNs converge to identity, thus solving vanishing/exploding gradient problem! (B) iRNN converges rapidly, at an exponential rate in the number of discrete samplings of Eq. 1. SiRNN, the single-step iRNN, is efficient and can be leveraged for slowly varying input sequences. It exhibits fast training time, has fewer parameters and better accuracy relative to standard LSTMs. (C) Extensive experiments on LTD datasets show that we improve upon standard LSTM accuracy as well as other recent proposals that are based on designing transition matrices and/or skip connections. iRNNs/SiRNNs are robust to time-series distortions such as noise paddings (D) While our method extends directly (see Appendix A.1) to Deep RNNs, we deem these extensions complementary, and focus on single-layer to highlight our incremental perspective. Gated Architectures. Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997 ) is widely used in RNNs to model long-term dependency in sequential data. Gated recurrent unit (GRU) (Cho et al., 2014 ) is another gating mechanism that has been demonstrated to achieve similar performance of LSTM with fewer parameters. Some recent gated RNNs include UGRNN (Collins et al., 2016) , and FastGRNN (Kusupati et al., 2018) . While mitigating vanishing/exploding gradients, they do not eliminate it. Often, these models incur increased inference, training costs, and model size. Unitary RNNs. Arjovsky et al. (2016); Jing et al. (2017) ; ; Mhammedi et al. (2016) focus on designing well-conditioned state transition matrices, attempting to enforce unitary-property, during training. Unitary property does not generally circumvent vanishing gradient (Pennington et al. (2017) ). Also, it limits expressive power and prediction accuracy while also increasing training time. Deep RNNs. These are nonlinear transition functions incorporated into RNNs for performance improvement. For instance, Pascanu et al. (2013a) empirically analyzed the problem of how to construct deep RNNs. Zilly et al. (2017) proposed extending the LSTM architecture to allow stepto-step transition depths larger than one. Mujika et al. (2017) proposed incorporating the strengths of both multiscale RNNs and deep transition RNNs to learn complex transition functions from one timestep to the next. While Deep RNNs offer richer representations relative to single-layers, it is complementary to iRNNs. Residual/Skip Connections. Jaeger et al. (2007) ; Bengio et al. (2013); Campos et al. (2017) ; Kusupati et al. (2018) feed-forward state vectors to induce skip or residual connections, to serve as a middle ground between feed-forward and recurrent models, and to mitigate gradient decay. Nevertheless, these connections, cannot entirely eliminate gradient explosion/decay. For instance, Kusupati et al. (2018) suggest h m = α m h m−1 + β m φ(U h m−1 + W x m + b), and learn parameters so that α m ≈ 1 and β m ≈ 0. Evidently, this setting can lead to identity gradient, observe that setting β m ≈ 0, implies little contribution from the inputs and can conflict with good accuracy, as also observed in our experiments. Linear RNNs. (Bradbury et al., 2016; Balduzzi & Ghifary, 2016) have focused on speeding up recurrent neural networks by replacing recurrent connections, such as hidden-to-hidden interactions, with light weight linear components. While this has led to reduced training time, it has resulted in significantly increasing model size. For example, typically requires twice the number of cells for LSTM level performance. ODE/Dynamical Perspective. There are a few works that are inspired by ODEs, and attempt to address stability, but do not end up eliminating vanishing/exploding gradients. Talathi & Vartak (2015) proposed a modified weight initialization strategy based on a dynamical system perspective on weight initialization process that leads to successfully training RNNs composed of ReLUs. Niu et al. (2019) analyzed RNN architectures using numerical methods of ODE and propose a family of ODE-RNNs. Chang et al. (2019) , propose Antisymmetric-RNN. Their key idea is to express the transition matrix in Eq. 1, for the special case α = 0, τ = 1, as a difference: U = V − V T and note that the eigenspectrum is imaginary. Nevertheless, Euler discretization, in this context leads to instability, necessitating damping of the system. As such vanishing gradient cannot be completely eliminated. Its behavior is analogous to FastRNN Kusupati et al. (2018) , in that, identity gradient conflicts with high accuracy. In summary, we are the first to propose evolution over the equilibrium manifold, and demonstrating identity gradients. Neural ODEs (Chen et al., 2018; Rubanova et al., 2019) have also been proposed for time-series prediction to deal with irregularly sampled inputs. To do this they parameterize the derivative of the hidden-state in terms of an autonomous differential equation and let the ODE evolve in continuous time until the next input arrives. As such, this is not our goal, our ODE explicitly depends on the input, and evolves until equilibrium for that input is reached. We introduce incremental updates to bypass vanishing/exploding gradient issues, which is not of specific concern for these works.
Incremental-RNNs resolves exploding/vanishing gradient problem by updating state vectors based on difference between previous state and that predicted by an ODE. The authors address the problem of signal propagation in recurrent neural networks by building an attractor system for the signal transition and checking whether it converges to an equilibrium. 