The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency. To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time. To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL). We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference. Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions. Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives. We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting. Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines. While recent successes in deep (reinforcement) learning for computer games (Atari (Mnih et al., 2013) , StarCraft (Vinyals et al., 2019) ), Go (Silver et al., 2017) and other high-throughput domains, e.g. (OpenAI et al., 2018) , have demonstrated the potential of these methods in the big data regime, the high cost of data acquisition has so far limited progress in many tasks of real-world relevance. Data efficiency in machine learning generally relies on inductive biases to guide and accelerate the learning process; e.g. by including expert domain knowledge of varying granularity. Incorporating such knowledge can accelerate learning -but when inaccurate it can also inappropriately bias the space of solutions and lead to sub-optimal results. Robotics represents a domain in which data efficiency is critical, and human prior knowledge is commonly provided. However, for scalability and reduced dependency on human accuracy, we can instead utilise an agent's permanent embodiment and shared environment across tasks. Intuitively, such a scenario suggests the natural strategy of focusing on inductive biases that facilitate the sharing and reuse of experience and knowledge across tasks while other aspects of the domain can be learned. As a general principle this relieves us from the need to inject detailed knowledge about the domain, instead we can focus on general principles that facilitate reuse (Caruana, 1997) . Successes for transfer learning have, for example, built on optimizing initial parameters (e.g. Finn et al., 2017) , sharing models and parameters across tasks either in the form of policies or value functions (e.g. Rusu et al., 2016; Teh et al., 2017; Galashov et al., 2018) , data-sharing across tasks (e.g. Riedmiller et al., 2018; Andrychowicz et al., 2017) , or through the use of task-related auxiliary objectives (Jaderberg et al., 2016; Wulfmeier et al., 2017) . Transfer between tasks can, however, lead to either constructive or destructive transfer for humans (Singley and Anderson, 1989) as well as for machines (Pan and Yang, 2010; Torrey and Shavlik, 2010) . That is, jointly learning to solve different tasks can provide both benefits and disadvantages for individual tasks, depending on their similarity. Finding a mechanism that enables transfer where possible but avoids interference is one of the long-standing research challenges. In this paper we explore the benefits and limitations of hierarchical policies in single and multitask reinforcement learning. Similar to Mixture Density Networks (Bishop, 1994) our models represent policies as state-conditional Gaussian mixture distributions, with separate Gaussian mixture components as low-level policies which can be selected by the high-level controller via a categorical action choice. In the multitask setting, to obtain more robust and versatile low-level behaviors, we additionally shield the mixture components from information about the task at hand. In this case, task information is only communicated through the choice of mixture component by the high-level controller, and the mixture components can be seen as domain-dependant, task-independent skills although the nature of these skills is not predefined and emerges during end-to-end training. We implement this idea by building on three forms of transfer: targeted exploration via the concatenation of tasks within one episode (Riedmiller et al., 2018) , sharing transition data across tasks (Andrychowicz et al., 2017; Riedmiller et al., 2018 ), and reusing low-level components of the aforementioned policy class. To this end we develop a novel robust and data-efficient multitask actor-critic algorithm, Regularized Hierarchical Policy Optimization (RHPO). Our algorithm uses the multitask learning aspects of SAC (Riedmiller et al., 2018) to improve data-efficiency and robust policy optimization properties of MPO (Abdolmaleki et al., 2018a) in order to optimize hierarchical policies. We furthermore demonstrate the generality of hierarchical policies for multitask learning via improving results also after replacing MPO as policy optimizer with another gradient-based, entropy-regularized policy optimizer (Heess et al., 2015) (see Appendix A.10). We demonstrate that compositional, hierarchical policies -while strongly reducing training time in multitask domains -can fail to improve performance in single task domains if no additional inductive biases are given. While multitask domains provide sufficient pressure for component specialization, and the related possibility for composition, we are required to introduce additional incentives to encourage similar developments for single task domains. In the multitask setting, we demonstrate considerably improved performance, robustness and learning speed compared to competitive continuous control baselines demonstrating the relevance of hierarchy for data-efficiency and transfer. We finally evaluate our approach on a physical robot for robotic manipulation tasks where RHPO leads to a significant speed up in training, enabling it to solve challenging stacking tasks on a single robot 1 . We introduce a novel framework to enable robust training and investigation of hierarchical, compositional policies in complex simulated and real-world tasks as well as provide insights into the learning process and its stability. In simulation as well as on real robots, RHPO outperforms baseline methods which either handle tasks independently or utilize implicit sharing. Especially with increasingly complex tasks or limited data rate, as given in real-world applications, we demonstrate hierarchical inductive biases to provide a compelling foundation for transfer learning, reducing the number of environment interactions significantly and often leading to more robust learning as well as improved final performance. For single tasks with a single training objective all components can remain aligned, preventing problem decomposition and the hierarchical policy replicates a flat policy. Performance improvements appear only when the individual components specialize, either via variety in the training objectives or additional incentives. Furthermore, as demonstrated in Appendix A.9, a pre-trained set of specialized components can notably improve performance when learning new tasks. One important next step is identifying how to optimize a basis set of components which transfers well to a wide range of tasks Since with mixture distributions, we are able to marginalize over components when optimizing the weighted likelihood over action samples in Equation 6, the extension towards multiple levels of hierarchy is trivial but can provide a valuable direction for practical future work. While this approach partially mitigates negative interference between tasks in a parallel multitask learning scenario, addressing catastrophic inference in sequential settings remains a challenge. We believe that especially in domains with consistent agent embodiment and high costs for data generation learning tasks jointly and information sharing is imperative. RHPO combines several ideas that we believe will be important: multitask learning with hierarchical and compositional policy representations, robust optimization, and efficient off-policy learning. Although we have found this particular combination of components to be very effective we believe it is just one instance of -and step towards -a spectrum of efficient learning architectures that will unlock further applications of RL both in simulation and, importantly, on physical hardware.
We develop a hierarchical, actor-critic algorithm for compositional transfer by sharing policy components and demonstrate component specialization and related direct benefits in multitask domains as well as its adaptation for single tasks. A combination of different learning techniques for acquiring structure and learning with asymmetric data, used to train an HRL policy. The authors introduce a hierarchical policy structure for use in both single task and multitask reinforcement learning, and assess the structure's usefulness on complex robotic tasks.