Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system.
 We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction.
We classify the the dynamical features one and two GRU cells can and cannot capture in continuous time, and verify our findings experimentally with k-step time series prediction.  The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset This paper analyzes GRUs from a dynamical systems perspective, and shows that 2d GRUs can be trained to adopt a variety of fixed points and can approximate line attractors, but cannot mimic a ring attractor. Converts GRU equations into continuous time and uses theory and experiemnts to study 1- and 2-dimensional GRU networks and showcase every variety of dynamical topology available in these systems