Building deep neural networks to control autonomous agents which have to interact in real-time with the physical world, such as robots or automotive vehicles, requires a seamless integration of time into a network’s architecture. The central question of this work is, how the temporal nature of reality should be reflected in the execution of a deep neural network and its components. Most artificial deep neural networks are partitioned into a directed graph of connected modules or layers and the layers themselves consist of elemental building blocks, such as single units. For most deep neural networks, all units of a layer are processed synchronously and in parallel, but layers themselves are processed in a sequential manner. In contrast, all elements of a biological neural network are processed in parallel. In this paper, we define a class of networks between these two extreme cases. These networks are executed in a streaming or synchronous layerwise-parallel manner, unlocking the layers of such networks for parallel processing. Compared to the standard layerwise-sequential deep networks, these new layerwise-parallel networks show a fundamentally different temporal behavior and flow of information, especially for networks with skip or recurrent connections. We argue that layerwise-parallel deep networks are better suited for future challenges of deep neural network design, such as large functional modularized and/or recurrent architectures as well as networks allocating different network capacities dependent on current stimulus and/or task complexity. We layout basic properties and discuss major challenges for layerwise-parallel networks. Additionally, we provide a toolbox to design, train, evaluate, and online-interact with layerwise-parallel networks. Over the last years, the combination of newly available large datasets, parallel computing power, and new techniques to design, implement, and train deep neural networks has led to significant improvements and numerous newly enabled applications in various fields including vision, speech, and reinforcement learning. Considering applications for which a neural network controls a system that interacts in real-time with the physical world, ranging from robots and autonomous vehicles to chat-bots and networks playing computer games, renders it essential to integrate time into the network's design.In recent deep learning literature, enabling networks to learn and represent temporal features has gained interest. Methods were presented leveraging short-term dynamic features to build temporal consistent network responses (e.g. BID9 , BID15 ) as well as networks learning to store and utilize information over longer time periods (e.g. BID17 , BID3 ).Two major aspects considering the role of time in neural networks can be distinguished: First, the way neural networks and their components such as layers or single units, are implemented. For example, network components could operate sequentially or in parallel, and in case of parallel evaluation, synchronous and asynchronous implementations can be distinguished. Second , the extent to which the network through its architecture can form representations of temporal features. For example , if the network has no mechanisms to integrate information over time, such as recurrent connections, the network will not be able to represent temporal features, such as optic-flow. In this work , we focus on the implementation aspect but highly emphasise that our approach fundamentally influences the network's temporal behavior and the way information is integrated over time.Whereas, biological neural networks and some realizations of neural networks in silicon (reviewed in BID10 ), comparison in Farabet et al. (2012 ) can operate on a continuous temporal dimension, we will assume a discrete (frame-based) temporal domain throughout this paper. In this paper, we defined and discussed layerwise-parallel deep neural networks, by which layerwise model-parallelism is realized for deep networks independently of their architecture. We argued that layerwise-parallel networks are beneficial for future trends in deep network design, such as large functional modularized or recurrent architectures as well as for networks allocating different network capacities dependent on stimulus and/or task complexity. Due to their biologically inspired increased parallelizability, layerwise-parallel networks can be distributed across several processes or GPUs natively without the need to explicitly specifying the network parts which should be parallelized. Finally, we presented an open source toolbox to explore layerwise-parallel networks providing design, training, evaluation, and interaction mechanisms.We would like to think of this work as a step towards native model-parallel deep networks, connecting the networks architecture directly to the temporal domain. For this, major challenges for the future remain, such as a more general formulation of neuron and synapse-pools than the one used in the provided toolbox, the design of new local plasticities, or designing more adequate tasks which take the temporal domain into account.
We define a concept of layerwise model-parallel deep neural networks, for which layers operate in parallel, and provide a toolbox to design, train, evaluate, and on-line interact with these networks. A GPU-accelerated toolbox for parallel neuron updating, written in Theano, that supports different update orders in recurrent networks and networks with connections that skip layers.  A new toolbox for deep neural networks learning and evaluation, and proposal for a paradigm switch from layerwise-sequential networks to layer-wise parallel networks.