Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime,we introduce the Unrestricted Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent. We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of differ-ent depths and widths) can emerge dynamically from the same URN.These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network. We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data. A remarkable property of biological neural netowrks (BNNs) is their adaptability in the face of new environments, different tasks and when coping with structure damage [1] . In contrast, despite their successes, artificial neural networks (ANNs) are limited in their applicability, and structures need to be designed for each particular task. Inspired by the genetic evolution of BNNs, an active field of neural architecture search has emerged leading to specialized networks which excel at specific tasks [3] . However, there is no ANN analog of the lifetime evolution of the structure of BNNs which provide flexibility in the face of new challenges or damage. The question therefore naturally arises of 1. whether there exist flexible ANNs which can adapt their connectivity structure to the task they are trained on during their lifetime and 2. whether there exists a new machine learning paradigm based on these flexible networks which can compete with the highly specialized networks in use today. The present work is a small step towards answering some of these questions. In particular, we introduce the Unrestricted Recursive Network (URN), and show that when trained end to end via stochastic gradient descent, a URN dynamically chooses its structure. Specifically, depending on the geometric structure of the data and the choice of regulator hyperparameters, the same URN can turn into networks which are recursive or feedforward, fully connected or locally connected (as in CNNs), and can choose whether or not to have residual skip connections. We also show that the specific form of the URN and the loss function used is mostly determined by various symmetry arguments. In this paper we have an empirical demonstration that many of the neural network structures in use today can dynamically emerge from the same general framework of the URN. We showed in examples that the final topology of these networks is easily interpretable as feed-forward MLPs with number of layers and number of neurons per layer determined during training. Furthermore, we showed that given input data with proximity information (e.g. a metric), we can naturally extend the URN loss function such that we can derive locally connected networks whose generalization performance is considerably improved. These demonstrations, however, are only the first stages of this project and much work remains to be done. For example, one can ask, how does the emergent network topology vary with task difficulty. This question is currently under study and beyond the scope of the demonstrations in this paper. One can also ask many other questions: e.g. under what circumstances, if any, recurrent neural NNs emerge from a URN or if it is possible to somehow naturally incorporate weight sharing such that we can arrive at a convolutional network. Finally, a theoretical understanding of why we generically arrive at feed-forward networks beyond simple intuitive arguments is still needed. Never-Ending Structure Accumulation. In light of the recent works in continual and never-ending learning [6] , and to circle back to the points raised in the introduction, we propose the following alternative learning scheme. Let us assume that we are given a series of related tasks of gradually increasing difficulty. For example, in vision, these can start from simple edge detection and end with image classification. We can intuitively predict what will happen if we train a network with dynamically chosen architecture on these tasks consecutively using a compatible lifelong learning algorithm which minimizes performance loss on prior tasks. When trained on the simple tasks, the emergent network would be shallow with few layers. However, as more complex tasks are trained, the depth of the network would grow and each consecutive tasks would naturally build on top of the structures already present in the architecture. Preliminary results show that this expectation is borne out when training a URN in conjunction with the lifelong learning algorithm from Golkar et al. [5] on a series of simple to difficult image tasks. This line of argument and experiments suggest an alternative learning paradigm to today's highly specialized networks specifically built for each task. In this learning paradigm, which we dub Never-Ending Structure Accumulation or NESA, the structure is simply determined by the series of simple to difficult tasks which culminate in the final ML problem of interest. The responsibility of the ML practitioner in NESA would then be to design this series tasks. While this is not a trivial undertaking for many ML problems, it brings the problem of training ANNs much closer to how BNNs learn to perform new tasks during their lifetime.
We introduce a network framework which can modify its structure during training and show that it can converge to various ML network archetypes such as MLPs and LCNs. 