Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability. At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics. What determines the dimensionality of activity in neural circuits? What is the functional role of dimensionality in behavior and task learning? In this work we address these questions using recurrent neural network (RNN) models. We find that, depending on the dynamics of the initial network, RNNs learn to increase and reduce dimensionality in a way that matches task demands. These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases. Dynamics shape computation in brain circuits. Due to the limitations in our ability to record every neuron in a circuit, it can be difficult to characterize these dynamics through direct observation alone. Bridging between machine learning and neuroscience, artificial recurrent neural networks (RNNs) are powerful tools for investigating dynamical representations in controlled settings, and enable tests of theoretical hypotheses that can be leveraged to formulate experimental predictions (reviewed in [2] ). Thinking of artificial networks as dynamical brain circuits is likewise a useful way of understanding their power and flexibility. Since RNNs give rise to well-defined dynamical systems, the neural representation of the recurrent units is governed by the system's dynamical response to inputs. In this work we task a network with classifying inputs into one of two classes (binary classification). We treat each input as an impulse delivered at an initial time t 0 , and allow the RNN a delay period to process this input before querying the network to output the class label (Fig. 1) . To reveal the essential dynamical elements at play in high-dimensional systems such as RNNs, dimensionality reduction is routinely employed [4] . These approaches reveal a surprising fact: rather than scaling with the number of neurons in the circuit, dynamics are often effectively constrained to regions whose dimensionality seems to be intimately linked to the complexity of the function, or behavior, that the neural circuit fulfills or produces [11] . This link between task and representation dimension is especially intriguing in light of fundamental ideas in learning theory. On one hand, high-dimensional representations can subserve complex and general computations that nonlinearly combine many features of inputs [6, 12] . On the other, low-dimensional representations that preserve only essential features needed for specific tasks can allow learning based on fewer parameters and examples, and hence with better generalization [6, 15] . Here we ask how an RNN balances reducing and increasing dimensionality of input data, and link this behavior to network dynamics. We find that the answer can depend on initialization; in particular, networks that are initially more chaotic have a tendency to expand the dimensionality of low-dimensional inputs. Frequently encountered in network models of brain function, dynamical chaos (whereby tiny changes in internal states are amplified by unstable, but deterministic, dynamics) provides a parsimonious explanation for both repeatable structure as well as internally generated variability seen in highly recurrent brain networks such as cortical circuits [14, 7] . While chaos-driven dimensionality expansion with fixed recurrent weights has previously been explored [8] , the attributes of this phenomenon as recurrent weights evolve through training are less understood. We find that in tasks where inputs are linearly separable by class label, RNNs generically reduce the dimension of their inputs over the delay period, in the process forming a representation that lends itself to good generalization. Next, we find that in harder tasks where inputs are low-dimensional and class separation boundaries are highly nonlinear, only networks with sufficiently chaotic initializations are successful. We explain this by showing that chaos-driven dimensionality expansion results in representations with linear separation boundaries. Taken together, we find evidence that RNNs learn representations that have the minimal dimensionality needed to support relatively simple class separation boundaries, provided that the initialization is sufficiently chaotic. These findings invite further exploration of learning strategies through the lens of modulating dimensionality and suggest functional roles for variability found in brain circuits. While effective dimensionality was chosen in this study because it is able to capture the distribution of disjoint manifolds and related coding properties, in general it is of interest to study nonlinear measures of dimensionality (i.e. intrinsic dimension). Recent work has explored this direction in the context of deep feedforward neural networks [10, 1] , but connection between nonlinear dimension and RNN dynamics have still not been explored as far as we are aware. In addition, it is of interest to track the dimension of individual input or class manifolds, as is done in [3] for deep feedforward networks. It is also of interest to see if the phenomena of dimension compression and expansion can be captured by mathematical analysis. See [13, 5] for work in the direction of demonstrating how compression can be driven by stochastic gradient descent. While this study suggests roles for dimensionality modulation and (chaotic) variability in biological neural circuits, it would be interesting to look for this explicitly in experiments. Finally, it is of interest to see if these phenomena extend to (recurrent) network models that achieve state-of-the-art performance, and to see if the principles explored here can be used to improve the functioning of such networks.
Recurrent Neural Networks learn to  increase and reduce the dimensionality of their internal representation in a way that matches the task, depending on the dynamics of the initial network.