Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs). Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy. Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences. In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks. This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT. Recurrent neural networks (RNNs) model sequential data by observing one sequence element at a time and updating their internal (hidden) state towards being useful for making future predictions. RNNs are theoretically appealing due to their Turing-completeness Siegelmann and Sontag (1995) , and, crucially, have been tremendously successful in complex real-world tasks, including machine translation Cho et al. (2014) ; Sutskever et al. (2014) , language modelling Mikolov et al. (2010) , and reinforcement learning Mnih et al. (2016) . Still, training RNNs in practice is one of the main open problems in deep learning, as the following issues prevail. (1) Learning long-term dependencies is extremely difficult because it requires that the gradients (i.e. the error signal) have to be propagated over many steps, which easily causes them to vanish or explode Hochreiter (1991) ; Bengio et al. (1994) ; Hochreiter (1998) (2) Truncated Backpropagation Through Time (TBPTT) Williams and Peng (1990) , the standard training algorithm for RNNs, requires memory that grows linearly in the length of the sequences on which the network is trained. This is because all past hidden states must be stored. Therefore, the memory requirements of training RNNs with large hidden states on long sequences become prohibitively large. (3) In TBPTT, parameters cannot be updated until the full forward and backward passes have been completed. This phenomenon is known as the parameter update lock Jaderberg et al. (2017) . As a consequence, the frequency at which parameters can be updated is inversely proportional to the length of the time-dependencies that can be learned, which makes learning exceedingly slow for long sequences. The problem of vanishing/exploding gradients has been alleviated by a plethora of approaches ranging from specific RNN architectures Hochreiter and Schmidhuber (1997) ; Cho et al. (2014) to optimization techniques aiming at easing gradient flow Martens and Sutskever (2011) ; Pascanu et al. (2013) . A candidate for effectively resolving the vanishing/exploding gradient problem is hierarchical RNNs (HRNNs) Schmidhuber (1992) ; El Hihi and Bengio (1996) ; Koutnik et al. (2014) ; Sordoni et al. (2015) ; Chung et al. (2016) . In HRNNs, the network itself is split into a hierarchy of levels, which are updated at decreasing frequencies. As higher levels of the hierarchy are updated less frequently, these architectures have short (potentially logarithmic) gradient paths that greatly reduce the vanishing/exploding gradients issue. In this paper, we show that in HRNNs, the lower levels of the hierarchy can be decoupled from the higher levels, in the sense that the gradient flow from higher to lower levels can effectively be replaced by locally computable losses. Also, we demonstrate that in consequence, the decoupled HRNNs admit training with memory decreased by a factor exponentially in the depth of the hierarchy compared to HRNNs with standard TBPTT. The local losses stem from decoder networks which are trained to decode past inputs to each level from the hidden state that is sent up the hierarchy, thereby forcing this hidden state to contain all relevant information. We experimentally show that in a diverse set of tasks which rely on long-term dependencies and include deep hierarchies, the performance of the decoupled HRNN with local losses is indistinguishable from the standard HRNN. In summary, we introduce a RNN architecture with short gradient paths that can be trained memoryefficiently, thereby addressing issues (1) and (2). In the bigger picture, we believe that our approach of replacing gradient flow in HRNNs by locally computable losses may eventually help to attempt solving issue (3) as well. In this paper, we have shown that in hierarchical RNNs the gradient flow from higher to lower levels can be effectively replaced by locally computable losses. This allows memory savings up to an exponential factor in the depth of the hierarchy. In particular, we first explained how not propagating gradients from higher to lower levels permits these memory savings. Then, we introduced auxiliary losses that encourage information to flow up the hierarchy. Finally, we demonstrated experimentally that the memory-efficient HRNNs with our auxiliary loss perform on par with the memory-heavy HRNNs and strongly outperform HRNNs given the same memory budget on a wide range of tasks, including deeper hierarchies. High capacity RNNs, like Differentiable Plasticity Miconi et al. (2018) , or Neural Turing Machines Graves et al. (2014) have been shown to be useful and even achieve state-of-the-art in many tasks. However, due to the memory cost of TBPTT, training such models is often impractical for long sequences. We think that combining these models with our techniques in future work could open the possibility for using high capacity RNNs for tasks involving long-term dependencies that have been out of reach so far. Still, the problem of the parameter update lock remains. While this is the most under-explored of the three big problems when training RNNs (vanishing/exploding gradients and memory requirements being the other two), resolving it is just as important in order to be able to learn long-term dependencies. We believe that the techniques laid out in this work (i.e. replacing gradients in HRNNs by locally computable losses) can be a stepping stone towards solving the parameter update lock. We leave this for future work.
We replace some gradients paths in hierarchical RNN's by an auxiliary loss. We show that this can reduce the memory cost while preserving performance. The paper introduces a hierarchical RNN architecture that could be trained more memory efficiently. The proposed paper suggests to decouple the different layers of hierarchy in RNN using auxiliary losses.