Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills. Deep reinforcement learning has demonstrated success in policy search for tasks in domains like game playing BID12 BID7 BID11 and robotic control BID9 b; BID16 . However, it is very difficult to accumulate multiple skills using just one policy network BID24 . Knowledge transfer techniques like distillation BID3 BID18 BID15 BID24 have been applied to train a policy network both to learn new skills while preserving previously learned skill as well as to combine single-task policies into a multi-task policy. Existing approaches usually treat all tasks independently. This often prevents full exploration of the underlying relations between different tasks. They also typically assume that all policies share the same state space and action space. This precludes transfer of previously learned simple skills to a new policy defined over a space with differing states or actions. When humans learn new skills, we often take advantage of our existing skills and build new capacities by composing or combining simpler ones. For instance, learning multi-digit multiplication relies on the knowledge of single-digit multiplication; learning how to properly prepare individual ingredients facilitates cooking dishes based on complex recipes. Inspired by this observation, we propose a hierarchical policy network which can reuse previously learned skills alongside and as subcomponents of new skills. It achieves this by discovering the underlying relations between skills. To represent the skills and their relations in an interpretable way, we also encode all tasks using human instructions such as "put down." This allows the agent to communicate its policy and generate plans using human language. Figure 1 illustrates an example: given the instruction "Stack blue," our hierarchical policy learns to compose instructions and take multiple actions through a multi-level hierarchy in order to stack two blue blocks together. Steps from the top-level policy π 3 (i.e., the red Figure 1: Example of our multi-level hierarchical policy for a given task -stacking two blue blocks. Each arrow represents one step generated by a certain policy and the colors of arrows indicate the source policies. Note that at each step, a policy either utters an instruction for the lower-level policy or directly takes an action.branches) outline a learned high-level plan -"Get blue → Find blue → Put blue. " In addition, from lower level policies, we may also clearly see composed plans for other tasks. Based on policy π 2 , for instance, the task "Get blue" has two steps -"Find blue → action: turn left," whereas "Put blue" can be executed by a single action "put down" according to π 3 . Through this hierarchical model, we may i) accumulate tasks progressively from a terminal policy to a top-level policy and ii) unfold the global policy from top-level to basic actions. In order to better track temporal relationships between tasks, we train a stochastic temporal grammar (STG) model on the sequence of policy selections (previously learned skill or new skill) for positive episodes. The STG focuses on modeling priorities of tasks: for example, it is necessary to obtain an object before putting it down. Integrating the STG into the hierarchical policy boosts efficiency and accuracy by explicitly modeling such commonsense world knowledge. We validated our approach by testing it on object manipulation tasks implemented in a Minecraft world. Our experimental results demonstrate that this framework can (i) efficiently learn hierarchical policies and representations for multi-task RL; (ii) learn to utter human instructions to deploy pretrained policies, improve their explainability and reuse skills; and (iii) learn a stochastic temporal grammar via self-supervision to predict future actions. Multi-task Reinforcement Learning. Previous work on multi-task reinforcement learning mainly falls into two families: knowledge transfer through distillation BID18 BID15 BID24 BID25 or modular policy design through 2-layer hierarchical policy BID0 . Our multi-level policy is more similar to the latter approach. The main differences between our model and the one in BID0 are two-fold: i) we do not assume that a global task can be executed by only performing predefined sub-tasks; ii) in our multi-level policy, global tasks at a lower-level layer may also be used as sub-tasks by global tasks carried out at higher-levels. Hierarchical Reinforcement Learning. Complex policies often require the modeling of longer temporal dependencies than what standard Markov decision processes (MDPs) can capture. To combat this, hierarchical reinforcement learning was introduced to extend MDPs to semi-MDPs BID23 , where options (or macro actions) are introduced on top of primitive actions to decompose the goal of a task into multiple subgoals. In hierarchical RL, two sets of policies are trained: local policies that map states to primitive actions for achieving subgoals, and a global policy that initiates suitable subgoals in a sequence to achieve the final goal of a task BID1 BID8 BID27 BID25 BID0 . This two-layer hierarchical policy design significantly improves the ability of discovering complex policies which can not be learned by flat policies. However, it also often makes some strict assumptions that limit its flexibility: i) a task's global policy cannot use a simpler task's policy as part of its base policies; ii) a global policy is assumed to be executable by only using local policies over specific options, e.g., BID8 BID0 . In this work, we aim to learn a multi-level global policy which does not have these two assumptions. In addition, previous work usually uses a latent variable to represent a task. In our work, we encode a task by a human instruction to learn a task-oriented language grounding as well as to improve the interpretability of plans composed by our hierarchical policies. Language grounding via reinforcement learning. Recently, there has been work on grounding human language in 3D game environments BID5 BID4 or in text-based games BID14 via reinforcement learning. In these games agents are instructed to pick up an item described by a sentence. Besides visual grounding, BID0 grounded instructions (not necessarily using human language) to local policies in hierarchical reinforcement learning. Our approach not only learns the language grounding for both visual knowledge and policies, but is also trained to utter human instructions as an explicit explanation of its decisions to humans. To our knowledge, this is the first model that learns to compose plans for complex tasks based on simpler ones which have human descriptions. In this section, we discuss our multi-task RL setting, hierarchical policy, stochastic temporal grammar, and how interaction of these components can achieve plan composition. Let G be a task set, where each task g is uniquely described by a human instruction. For simplicity, we assume a two-word tuple template consisting of a skill and an item for such a phrase, i.e., u skill , u item . Each tuple describes an object manipulation task. In this paper, we define g = u skill , u item by default, thus tasks and instructions are treated as interchangeable concepts. For each task, we define a Markov decision process (MDP) represented by states s ∈ S and primitive actions a ∈ A. Rewards are specified for goals of different tasks, thus we use a function R(s, g) to signal the reward when performing any given task g. We assume that as a starting point, we have a terminal policy π 0 (as shown in FIG2 ) trained for a set of basic tasks (i.e., a terminal task set G 0 ). The task set is then progressively increased as the agent is instructed to do more tasks by humans at multiple stages, such that G 0 ⊂ G 1 ⊂ · · · ⊂ G K , which results in life-long learning of polices from π 0 for G 0 to π K for G K as illustrated by the "task accumulation" direction in Figure 1 . At stage k > 0, G k−1 is defined as the base task set of G k . The tasks in G k−1 are named as base tasks at this stage and π k−1 becomes the base policy of π k . Here, we utilize weak supervision from humans to define what tasks shall be augmented to the previous task set at each new stage. One of our key ideas is that a new task in current task set G k may be decomposed into several simpler subtasks, some of which can be base tasks in G k−1 executable by base policy π k−1 . Therefore, instead of using a flat policy ( FIG2 ) as π 0 that directly maps state and human instruction to a primitive action, we propose a hierarchical design FIG2 ) with the ability to reuse the base policy (i.e., π k−1 ) for performing base tasks as subtasks. Namely, at stage k, the global policy π k is defined by a hierarchical policy. This hierarchy consists of four sub-policies: a base policy for executing previously learned tasks, an instruction policy that manages communication between the global policy and the base policy, an augmented flat policy which allows the global policy to directly execute actions, and a switch policy that decides whether the global policy will primarily rely on the base policy or the augmented flat policy. The base policy is defined to be the global policy at the previous stage k − 1. The instruction policy maps state s and task g ∈ G k to a base task g ∈ G k−1 . The purpose of this policy is to inform base policy π k−1 which base tasks it needs to execute. Since an instruction is represented by two words, we define the instruction policy using two conditionally independent distributions, i.e., π DISPLAYFORM0 ). An augmented flat policy, π aug k (a|s, g), maps state s and task g to a primitive action a for ensuring that the global policy is able to perform novel tasks in G k that can not be achieved by only reusing the base policy. To determine whether to perform a base task or directly perform a primitive action at each step, the global policy further includes a switch policy, π sw k (e|s, g), where e is a binary variable indicating the selection of the branches, π inst k (e = 0) or π aug k (e = 1). Note that the above description of the hierarchical policy does not account for an STG. The instruction policy and switch policy introduced here are simplified from the ones in the full model (see Section 3.3).At each time step, we first sample e t from our switch policy π sw k to decide whether the global policy π k will rely on the base policy π k−1 or the augmented flat policy π aug k . We also sample a new instruction g t from our instruction policy π inst k in order to sample actions from the base policy. This can be summarized as: DISPLAYFORM1 DISPLAYFORM2 and finally DISPLAYFORM3 where π k and π k−1 are the global policies at stage k and k − 1 respectively. After each step, we will also obtain a reward r t = R(s t , g). Different tasks may have temporal relations. For instance, to move an object, one needs to first find and pick up that object. There has been previous research BID19 BID17 using stochastic grammar models to capture such temporal relations. Inspired by this, we summarize temporal transitions between various tasks with a stochastic temporal grammar (STG). In our full model, the STG interacts with the hierarchical policy described above through the modified switch policy and instruction policy by using the STG as a prior. This amounts to treating the past history of switches and instructions in positive episodes as a guidance on whether the hierarchical policy should defer to the base policy to execute a specific base task or employ its own augmented flat policy to take a primitive action. In an episode, the temporal sequence of e t and g t , i.e., { e t , g t ; t ≥ 0}, can be seen as a finite state Markov chain BID2 . Note that the state here is referred to the tuple e t , g t , which is not the state of the game s t ∈ S defined in Section 3.1. Consequently, at each level k > 0, we may define an STG of a task g by i) transition probabilities, ρ k (e t , g t |e t−1 , g t−1 , g), and ii) the distribution of e 0 , g 0 , q k (e 0 , g 0 |g), all of which follow categorical distributions. With the estimated probabilities, we sample e t and g t in an episode at level k > 0 w.r.t. to reshaped policies π sw k and π inst k respectively: DISPLAYFORM0 DISPLAYFORM1 • Otherwise, DISPLAYFORM2 Note that primitive action sampling is not affected by the STG. Combined with our hierarchical policy and STG defined above, we are able to run an episode to compose a plan for a task specified by a human instruction. Algorithm 1 in Appendix A summarized this procedure with respect to the policy and STG at level k. Note that to fully utilize the base policy, we assume that once triggered, a base policy will play to the end before the global policy considers the next move. The learning algorithm is outlined in Algorithm 2 in Appendix A. We learn our final hierarchical policy through k stages of skill acquisition. Each of these stages is broken down into a base skill acquisition phase and a novel skill acquisition phase in a 2-phase curriculum learning. In the base skill acquisition phase, we only sample tasks from the base task set G k−1 . This ensures that the global policy learns how to use previously learned skills by issuing instructions to the base policy. In other words, this phase teaches the agent how to connect its instruction policy to its base policy. Once the average reward for all base tasks exceeds a certain threshold, we proceed to the next phase. In the novel skill acquisition phase, we sample tasks from the full task set, G k , for the k-th stage of skill acquisition. It is in this phase that the agent can learn when to rely on the base policy and when to rely on the augmented flat policy for executing novel tasks. In each of these phases, all policies are trained with advantage actor-critic (A2C) (Section 4.1) and distributions in the STG are estimated based on accumulated positive episodes (Section 4.2). We use advantage actor-critic (A2C) for policy optimization with off-policy learning BID22 . Here, we only consider the gradient for global policies (i.e., k > 0) as we assume the terminal policy has been trained as initial condition. Let V k (s t , g) be a value function indicating the expected return given state s t and task g. To reflect the nature of the branch switching in our model, we introduce another value function V sw k (s t , e t , g) to represent the expected return given state s t , task g and current branch selection e t .Thus, given a trajectory Γ = { s t , e t , g t , a t , r t , µ DISPLAYFORM0 where DISPLAYFORM1 , and ω DISPLAYFORM2 are importance sampling weights for the three terms respectively; A(s t , g, e t ), A(s t , g, e t , g t ), and A(s t , g, e t , a t ) are estimates of advantage functions, which have multiple possible definitions. In this paper, we define them by the difference between empirical return and value function estimation: A(s t , g, e t ) = DISPLAYFORM3 , where γ is the discounted coefficient. Finally, the value functions can be updated using the following gradient: DISPLAYFORM4 To increase the episode efficiency, after running an episode, we conduct n mini-batch updates where n is sampled from a Poisson distribution with λ = 4, similar to . Note that one can also apply other common policy optimization methods, e.g., A3C , to our model. We leave this as future work to evaluate the efficiency of different methods when using our model. Optimizing all three sub-policies together leads to unstable learning. To avoid this, we apply a simple alternating update procedure. For each set of M iterations, we keep two of the sub-policies fixed and train only the single policy that remains. When we reach M iterations, we switch the policy that is trained. For all experiments in this paper, we use M = 500. This alternating update procedure is used within both phases of curriculum learning. If at any point in the aforementioned training process the agent receives a positive reward after an episode, we update the stochastic temporal grammar. ρ k and q k of the STG are both initialized to be uniform distributions. Since the STG is a finite state Markov chain over tuples e t , g t , we use maximum likelihood estimation (MLE) to update the distributions BID2 . As the training progresses, the STG starts to guide the exploration. To avoid falling into local minima in the early stages of training, it is important to encourage random exploration in early episodes. Based on our experiments, we find that using -greedy suffices. Figure 3 (left) shows the two room environment in Minecraft that we created using the Malmo platform BID6 . In each episode, an arbitrary number of blocks with different colors (totaling 6 colors in our experiments) are randomly placed in one of the two rooms. The agent is initially placed in the same room with the items. We consider five sets of tasks: i) G (0) = {"Find x"}, walking to the front of a block with color x, ii) G (1) = {"Get x"}, picking up a block with color x, iii) G (2) = {"Put x"}, putting down a block with color x, iv) G (3) = {"Stack x"}, stacking two blocks with color x together, and v) G (4) = {'Put x on y'}, putting a block with color x on top of a block with a different color y. In total, there are 54 tasks. An agent can perform the following actions: "move forward," "move backward," "move left," "move right," "turn left," "turn right," "pick up," "put down."Without loss of generality, we assume the following skill acquisition order: DISPLAYFORM0 , ∀k = 0, 1, 2, 3, 4, which is a natural way to increase skill sets. One may also alter the order, and the main conclusions shall still hold. This results in policies {π k : k = 0, 1, 2, 3, 4} for these four task sets. For the last task set, we hold out 6 tasks out of all 30 tasks (i.e., 3 pairs of colors out of 15 color combinations) for testing and the agent will not be trained on these 6 tasks. We adopt a sparse reward function: when reaching the goal of a task, the agent gets a +1 reward; when generating an instruction g that is not executable in current game (e.g., trying to find an object that does not exist in the environment), we give a −0.5 reward; otherwise, no reward will be given. Whenever a non-zero reward is given, the game terminates. Note that the negative reward is only given during training. We specify the architecture of the modules in our model in Appendix B, where the visual and instruction encoding modules have the same architectures as the ones in BID5 . We train the network with RMSProp BID26 ) with a learning rate of 0.0001. We set the batch size to be 36 and clip the gradient to a unit norm. For all tasks, the discounted coefficient is γ = 0.95. For the 2-phase curriculum learning, we set the average reward threshold to be 0.9 (average rewards are estimated from the most recent 200 episodes of each task).To encourage random exploration, we apply -greedy to the decision sampling for the global policy (i.e., only at the top level k at each stage k > 0), where gradually decreases from 0.1 to 0. To evaluate the learning efficiency, we compare our full model with 1) a flat policy FIG2 ) as in BID5 fine-tuned on the terminal policy π 0 , 2) H-DRLN BID25 and variants of our approach: 3) ours without STG, 4) ours without alternating policy optimization, and 5) ours without V sw k (s, e, g) (replaced by V k (s, g) instead). Note that all the rewards have been converted to the same range, i.e., [0, 1] for the sake of fair comparison. In FIG5 , we use various methods to train policy π 1 for the task set G 1 based on the same base policy π 0 . The large dip in the reward indicates that the curriculum learning switches from phase 1 to phase 2. From FIG5 , we may clearly see that our full model and variants can all converge within 22,000 episodes, whereas the average reward of the flat policy is still below 0.8 given the same amount of episodes. In addition, our full model finishes phase 1 significantly faster than other methods and its curve of average reward maintains notably higher than the remaining ones. To further examine the learning efficiency during phase 2 when new tasks are added into the training process, we first pretrain π 3 using our full model following our definition of phase 1 in the curriculum learning. We then proceed to learning phase 2 using different approaches all based on this pretrained policy. As shown in FIG5 , our full model has the fastest convergence and the highest average reward upon convergence. By comparing FIG5 and FIG5 , we further show that our full model has a bigger advantage when learning more complex tasks. Since we have a large number of previously learned tasks, H-DRLN is clearly not able to learn a descent policy according the results. Note that an H-DRLN can only learn one task at a time, each of its curves in FIG5 is for a single task (i.e., "Get white" and "Stack white" respectively).To demonstrate the effects of our 2-phase curriculum learning and the −0.5 penalty on the training efficiency, we visualize the learning curves of our model trained without the curriculum learning or without the penalty along with the one trained with the full protocol in FIG7 . According to the results, the curriculum learning indeed helps accelerate the convergence, which empirically proves the importance of encouraging a global policy to reuse relevant skills learned by its base policy. It also appears that adding the penalty is an insignificant factor on learning efficiency except that it helps shorten the episode lengths as an episode ends whenever a penalty is given. Finally, we evaluate how the hierarchical design and encoding tasks by human instructions benefit the generalization of learned policies in the following three ways. First, we train π 1 in a simpler setting where in each episode, only one item (i.e, the target item of the given task) is present. We then test the policy π 1 for "Get x" tasks in a room where there will be multiple items serving as distraction and the agent must interact with the correct one. Both the flat policy and the hierarchical policy can achieve near perfect testing success rate in the simple setting. However, in the more complex setting, flat policy can not differentiate the target item from other items that are also placed in the room (the success rate drops to 29%), whereas our hierarchical policy still maintains a high success rate (94%). This finding suggests that the hierarchical policy not only picks up the concept of "find" and "get" skills as the flat policy does, but also inherits the concept of items from the base policy by learning to utter correct instructions to deploy "find" skill in the base policy. Second, we reconfigure the room layout in FIG3 (left) and test the flat policy and our full model in the new rooms shown in FIG3 (right) for various tasks. Both policies are trained in the same environment. There are multiple items in a room for both training and testing cases. The success rates are summarized in TAB1 . Using the flat policy results in a much bigger drop in the testing success rate compared to using out full model. This is mainly because that our global policy will repeatedly call its base policy to execute the same task until the agent finally achieves the goal even though the trained agent is unable to reach the goal by just one shot due to the simplicity of the training environment. Third, we evaluate the learned policy on the 6 unseen tasks for the "Put x on y" tasks as a zeroshort evaluation. The success rate reported in TAB1 suggests that our model is able to learn the decomposition of human instructions and generate correct hierarchical plans to perform unseen tasks. We visualize typical hierarchical plans of several tasks generated by global policies learned by our full model in Appendix C ( FIG10 and FORMULA6 1 . It can been seen from the examples that our global policies adjust the composed plans in different scenarios. For instance, in the second plan on the first row, π 1 did not deploy base policy π 0 as the agent was already in front of the target item at the beginning of the episode, whereas in the plan on the second row, π 1 deployed π 0 for the "Find x" base task twice consecutively, as it did not finish the base task in the first call. In this work, we have proposed a hierarchal policy modulated by a stochastic temporal grammar as a novel framework for efficient multi-task reinforcement learning through multiple training stages. Each task in our settings is described by a human instruction. The resulting global policy is able to reuse previously learned skills for new tasks by generating corresponding human instructions to inform base policies to execute relevant base tasks. We evaluate this framework in Minecraft games and have shown that our full model i) has a significantly higher learning efficiency than a flat policy does, ii) generalizes well in unseen environments, and iii) is capable of composing hierarchical plans in an interpretable manner. Currently, we rely on weak supervision from humans to define what skills to be learned in each training stage. In the future, we plan to automatically discover the optimal training procedures to increase the task set. A PSEUDO CODE OF OUR ALGORITHMS Algorithm 1 RUN(k, g) Input: Policy level k, task g ∈ G k Output: Episode trajectory Γ at the top level policy 1: t ← 0 2: Γ = ∅ 3: Get initial state s0 4: repeat 5:if k == 1 then 6:Sample at ∼ π k (·|st, g) and execute at 7:Get current state st+1 8:rt ← R(st+1, g) 9:Add st, at, rt, π k (·|st, g), g to Γ 10: else 11:Sample et and g t as in Section 3.3 for using STG as guidance 12:Sample at ∼ π aug k (·|st, g) 13:if et = 0 then 14:// Execute base policy π k−1 by giving instruction g t 15:RUN(k − 1, g t ) 16: else 17:Execute at 18: end if 19:Get current state st+1 20:rt ← R(st+1, g) 21:Add st, et, g t , at, rt, π DISPLAYFORM0 end if 23: if in curriculum learning phase 1 then 9: DISPLAYFORM1 Sample a task g from base task set G k−1 10: else 11:Sample a task g from global task set G k 12:end if 13://Run an episode 14: DISPLAYFORM2 if the maximum reward in Γ is +1 then 17: DISPLAYFORM3 Re-estimate the distributions of the STG based on updated D+ by MLE 19:end if 20:Sample n ∼ Possion(λ) 21:for j ∈ {1, · · · , n} do 22:Sample a mini-batch S from D 23:Update Θ based on (9) and the τ -th term in (8) 24:i ← i + 1 25:if i%M = 0 then 26:τ ← τ %3 + 1 27: end if 28: end for 29: until i ≥ N The architecture designs of all modules in our model shown in FIG2 are as follows:Visual Encoder extracts feature maps from an input RGB frame with the size of 84 × 84 through three convolutional layers: i) the first layer has 32 filters with kernel size of 8 × 8 and stride of 4; ii) the second layer has 64 filters with kernel size of 4 × 4 and stride of 2; iii) the last layer includes 64 filters with kernel size of 3 × 3 and stride of 1. The feature maps are flatten into a 3136-dim vector. We reduce the dimension of this vector to 256 by a fully connected (FC) layer resulting a 256-dim visual feature as the final output of this module. Instruction Encoder first embeds each word into a 128-dim vector and combines them into a single vector by bag-of-words (BOW). Thus the output of this module is a 128-dim vector. For more complex instructions such as "Put x on y", we replace BOW by a GRU with 128 hidden units. Fusion layer simply concatenates the encoded visual and language representations together and outputs 384-dim fused representation. We then feed this 384-dim vector into an LSTM with 256 hidden units. The hidden layer output of the LSTM is served as the input of all policy modules and value function modules. Switch Policy module has a FC layer with output dimension of 2 and a softmax activation to get π sw k (e|s, g). Instruction Policy module has two separate FC layers, both of which are activated by softmax to output the distribution of skill, p skill k (u skill |s, g), and the distribution of item, p item k (u item |s, g), respectively. Augmented Policy module outputs π aug (a|s, g) also through a FC layer and softmax activation. The two Value Function modules, V (s, g) and V sw (s, e, g), all have a scalar output through a FC layer. Note that all tasks must start from the top-level policy. The branches are ordered from left to right in time indicating consecutive steps carried out by a policy. We also show the egocentric view and the item in hands at critical moments for a real episode example.
A novel hierarchical policy network which can reuse previously learned skills alongside and as subcomponents of new skills by discovering the underlying relations between skills. This paper aims to learn hierarchical policies by using a recursive policy structure regulated by a stochastic temporal grammar This paper proposes an approach to learning hierarchical policies in a lifelong learning context by stacking policies and then using an explicit "switch" policy.