Continual Skill and Task Learning via Dialogue
Weiwei Gu Suresh Kondepudi Lixiao Huang Nakul Gopalan

Bi-capacity Choquet Integral for Sensor Fusion 
 with Label Uncertainty



This material is based upon work supported by the National Science
Foundation under Grant IIS-2153171-CRII: III: Explainable Multi-Source Data Integration with Uncertainty.

    Hersh Vakharia
University of Michigan 

Ann Arbor, MI 

hershv@umich.edu
Xiaoxiao Du
University of Michigan 

Ann Arbor, MI 

xiaodu@umich.edu

    Accepted XXX. Received YYY; in original form ZZZ
========================================================================================================================================================================================================================================================





§ ABSTRACT 
Continual and interactive robot learning is a challenging problem as the robot is present with human users who expect the robot to learn novel skills to solve novel tasks perpetually with sample efficiency.  
In this work we present a framework for robots to query and learn visuo-motor robot skills and task relevant information via natural language dialog interactions with human users. 
Previous approaches either focus on improving the performance of instruction following agents, or passively learn novel skills or concepts. 

Instead, we used dialog combined with a language-skill grounding embedding to query or confirm skills and/or tasks requested by a user. 
To achieve this goal, we developed and integrated three different components for our agent.
Firstly, we propose a novel visual-motor control policy ACT with Low Rank Adaptation (ACT-LoRA), which enables the existing state-of-the-art Action Chunking Transformer <cit.> model to perform few-shot continual learning. 
Secondly, we develop an alignment model that projects demonstrations across skill embodiments into a shared embedding allowing us to know when to ask questions and/or demonstrations from users. 
    
Finally, we integrated an existing Large Language Model (LLM) to interact with a human user to perform grounded interactive continual skill learning to solve a task. 
Our ACT-LoRA model learns novel fine-tuned skills with a 100% accuracy when trained with only five demonstrations for a novel skill while still maintaining a 74.75% accuracy on pre-trained skills in the RLBench dataset where other models fall significantly short.  

    
    
    
    
    
    We also performed a human-subjects study with 8 subjects to demonstrate the continual learning capabilities of our combined framework. We achieve a success rate of 75% in the task of sandwich making with the real robot learning from participant data 
    
    demonstrating that robots can learn novel skills or task knowledge from dialogue with non-expert users using our approach.



 





§ INTRODUCTION




 define natural interaction as an interaction between a human and a robot that resembles the way of natural communication between human beings such as dialogues, gestures, etc. without requiring the human to have prior expertise in robotics. 
 The capability of learning tasks and acquiring new  skills from natural interactions is desirable for robots as they need to perform unique tasks for different users.
One direction of this interaction channel is well studied as instruction following <cit.>, where the robot performs the tasks requested by the human via natural language. 
Our work focuses on the other side of this communication channel, where the robot starts the conversation with human when it needs their help. This reverse direction of communication plays an important role for robots to learn with non-expert human users as it enables robots to convey their lack of task knowledge to perform tasks in a way that non-expert users can understand. Furthermore, our framework can leverage the feedback from users and learn to perform the task. 





Human-Robot interaction via language is a well studied problem <cit.>. 
Robot agents have been able to interpret language instructions from the human users, and perform visual-motor policies to complete tasks <cit.>. 
These methods rely on the emergent behaviors of large models, and do not continually learn new skills or add to their task or skill knowledge. 

To address this issue, some works have proposed life-long learning for robot agents <cit.>. Some recent works learn neural visuo-motor skills in a continual setting <cit.>. However, these approaches are passive and do not query the user for novel skills that the agent might need to complete given tasks.    





We propose a framework that utilizes dialogue to enable the robot agent to express its need for new skill or task information actively.
When encountering a novel task, our robot agent starts a conversation with the human user to learn to execute the task. 
Throughout the interaction, the robot agent specifies the help that it needs from the human user via natural language, such as a human enacting the skill to find a feasible skill within the existing set of skills to perform the task or requesting multiple robot demonstrations to learn a completely novel skill for this specific task.


Our contributions are as follows:

    
    
  * We compare ACT-LoRA against the baseline ACT model on few-shot continual learning on RLBench dataset. Our model demonstrates its strong adaptability by achieving 100% success rate on the tasks that it finetuned on with only 5 demonstrations. Furthermore, it achieves an average success rate of 74.75% on the tasks that it is pre-trained on, showing that our policy is effective in preventing catastrophic forgetting. 
    
    
    
    
  * We present a model that can determine whether a pair of demonstrations of different embodiments, in our case human enactment of a skill or a robot demonstration, are performing the same task. Our alignment model achieves an overall accuracy of 91.4% on the RH20T dataset on aligning demonstrations from humans and robot.
    
  * Finally, we conduct a novel two-phase human-subjects experiment with eight participants to show that our system is able to learn to reason over and perform novel skills by interacting with non-expert human users for the task of making a sandwich using dialog. We find a success rate of 75% in making sandwiches for participants where the participants taught a skill in a continual fashion to our robot enabling it to make a sandwich. 
    

 












§ PROBLEM FORMULATION



We formulate a task solving problem where both the robot and the human agent can take actions on their turns. There is a joint physical state s of the world shared by both the human and the robot. In each turn, n, either the human or the robot acts, one after the other. Each turn can take longer than one time step, t, and continues until the robot or the human indicates a turn to be over. The actions can be physical actions represented by a_h, and a_r for the human and the robot actions respectively, or speech acts l_h and l_r for the human and the robot speech respectively for the human-robot grounded dialog. The problem has an initial state s_0 and a task θ specified by the human using a speech act l_h^0. Each of these actions updates the joint physical state s of the world, and internal dialog state s^d of the robot.  The dialog state is hidden from the human user, but the human receives speech observations for the same. Over multiple turns and actions taken by the human and the robot these physical and robot states update over time. The objective of this turn taking problem is to complete the task θ. We measure the task completion rates for this interaction problem. Moreover, in our specific instance of the problem the human also teaches behaviors to the robot, we also measure the success of the individual learned behaviors within the task in simulation.





§ METHODS







The goal of our framework is a robot agent that 1) actively generalizes its known skills to novel tasks when it is applicable; 2) queries the user for unknown skills; and 3) learns new skills with only a few instances.
When encountered a task θ, the robot agent first searches for a learned skill using semantic representation, which comes from the language embedding of the linguistic description of the skills and tasks. This is a challenging question as the robot needs to know what it does not know. This work is performed by our queryable skill library.
If the agent fails to find any usable skill for the task based on the semantic information, it attempts to search for a learned skill using skill representations, which come from human demonstrations and robot trajectories. We developed a novel sample efficient continual skill learning approach ACT-LoRA for this task.
The robot agent can directly execute the task τ whenever it finds a learned skill that aligns with τ in either the semantic space or the skill space, and learns a novel skill to execute the task otherwise.
During this whole process, the robot agent needs to interact with the human user based on information from our queryable skill library for which we use an LLM.










 §.§ How to Know What the Robot Does not Know a.k.a. a Queryable Skill Library

The skill library consists of four parts - a text encoder E_text; a human demonstration encoder E_human; a robot trajectory encoder E_robot; and a set of learned skills 𝒮 = {S_1,…, S_k}. 
Each learned skill S_i is a tuple of a linguistic description and a robot trajectory, denoted as S_i = (l_i, τ_i). 
The linguistic representation r^l_i and skill representation r^s_i of skill S_i can be obtained by encoding l_i and τ_i with the corresponding encoder, denoted as r^l_i = E_text(l_i), and r^s_i = E_robot(τ_i) respectively.

Finding a usable skill from the skill library.
The skill library is provided two inputs to find an appropriate skill to execute the task θ, the linguistic description l_θ and a human demonstration d_θ for the task.
We obtain the linguistic or semantic representation and skill representation for the task by encoding the linguistic description and the human demonstration with the corresponding encoders.
We then compute two sets of similarity scores between the task θ and any known skill S_i for both the linguistic representation and the skill representation. The state machine within the interaction module of the agent decides the skill to use to execute the task θ based on these scores. 


We use a pre-trained CLIP as a text encoder E_text.
For E_robot and E_human, we first extract features from each frame using a Resnet-18 <cit.>, and then encode the sequence using a transformer encoder <cit.>.

The robot trajectory encoder E_robot and the human demonstration encoder E_human are trained to encode the human demonstrations and robot trajectories into the same latent space. The two encoders are jointly trained with tuples (d, τ, y), where d denotes human demonstration videos, τ denotes robot trajectories, and y is the label of whether the human demonstration and the robot trajectory is in the same task. We use a cosine similarity loss to learn this embedding with a hyperparamter ϵ to act as a margin to declare a human demonstration to be the same skill as a skill the robot knows. More details about learning this embedding space are in the Appendix <ref>.












 §.§ Interaction Module using a Large Language Model (LLM)

The dialog state s^d in our pipeline is maintained with an internal state machine. The state machine uses an LLM, ChatGPT 4 <cit.>, as the natural language generator to produce speech acts for the robot agent.
This state machine with the LLM has two major functionalities.
Firstly, it interacts with the human user to asks for demonstrations or explanations based on the checks from our queryable skill library.
Secondly, the interaction module also interprets the user's language feedback to update the dialog state s^d.



The interaction module is given the autonomy to continue the dialogue with the user until that it acquires the designated information for the agent. The module can also explain the dialog state s^d with language to the user explaining the robot's confusion.









 §.§ ACT-LoRA as Visual-motor Policy


Combining Low-Rank Adaptor with Action Chunking.Adapter-based methods <cit.> have exhibited promising capabilities of light-weight and data-efficient fine-tuning of neural networks across various domains such as NLP <cit.>, and computer vision <cit.>. <cit.> extend Low-Rank Adaptor(LoRA) into robotics with TAIL, enabling a simulated robot to continually adapt to novel tasks without forgetting the old ones. Unfortunately in our experiments TAIL <cit.> fails to provide high precision control on the robot leading to a lot of failures in even short skills. 

On the other hand, Action Chunking Transformer(ACT) <cit.> is capable of performing fine-grained tasks with high precision, but cannot be directly used for continual learning due to catastrophic forgetting. Therefore, we introduce LoRA adaptor to the ACT model, obtaining both the precision from action chunking and the capability of continual learning from the LoRA adaptor. 
We want to point out that we are using TAIL <cit.> as the baseline in this work as it is the closest continual learning agent to our approach. 

We also want to point out that TAIL still requires our queryable skill library to function as a baseline. 

Continual Imitation Learning. Our policy needs to continually learn new skills from demonstrations throughout the agent's lifespan. The robot agent is initially equipped with K skills {S_1, …, S_K}. Whenever the robot agent encounters a task that requires a novel skill 𝒮_n, n>K, it needs to adapt its existing policy π to the novel skill without forgetting any of the existing skills S∈{S_1, …, S_n-1}. Provided a number of demonstration trajectories for each skill, the continual learning policy of the robot agent can then be optimized with a behavior cloning loss, which in this case we use L_1 loss for action chunks following <cit.>. On top of the policy of the vanilla ACT model π_ϕ, the LoRA adaptor introduce a small set of additional low-rank parameters ϕ_i for each skill S_i. During the pre-training phase, the additional parameters ϕ_1,…, ϕ_K for skills S_1, …, S_K are jointly trained with the model's parameter ϕ. When we are finetuning with a skill  S_n, n>K, we freeze the model's original parameters ϕ, and only allow gradient updates to the parameters from the task-specific adaptor ϕ_n. Such finetuning strategy prevents the policy from catastrophic forgetting the skills that it already possessed when adapting to novel skills.











 §.§ Human-Subjects Experiment




  §.§.§ Robotics Domain for Sandwich Making
 Our human subjects' experiment was on a sandwich making robot domain where the robot does not know all the skills required to make a butter, cheese and lettuce sandwich. Specifically, the robot does not know how to slice cheese.
The robotic setup includes a Franka FR3 Robot and three Realsense D435 cameras. We 3D printed tools needed to complete the tasks with planners capable of picking the tools of knife and spatula on demand. We use a 6D Spacemouse to collect data from human participants. Figure <ref> demonstrates our sandwich making domain. We used the sandwich-making task for two reasons. 
Firstly, the sandwich-making task includes a lot of contact-rich and dynamic sub-tasks, such as applying butter and slicing cheese. 
Secondly, sandwich-making is a multi-step process, allowing the robot agent and the participants to have multiple rounds of conversations. 
Fake food was used as our ingredients for environmental reasons.








  §.§.§ Study Design and Measures

Participants interacted with the robot in two phases.
During the first phase - the interaction phase, participants interact with the robot and teach the robot novel skills and task knowledge as they interact with it, this includes dialogue, human demonstration, and robot demonstration. In the second phase - the evaluation phase, participants request the robot to perform the same tasks and evaluate the performance of the robot agent. We needed a two-phase study because we wanted to collect data for skill learning in the first phase and then run a learned policy on the agent in the second phase. The participants came in for another session at least one day apart, allowing 5 hours of time to train novel skills using user demonstrations. This makes our study two separate 3 × 1 between-subjects experiment to measure our framework's ability to learn novel skills and task knowledge by interacting with non-expert human users. We do not compare any tasks between the two phases as they were performed on different days and their subjective metrics might be different depending on the subjects' memory of the experience.


The objective metrics we used for the human-subjects experiment are as follows.
We measured the overall success rate (SR) of completing the entire sandwich and the success rate for completing each independent sub-task. We make a distinction in the evaluation phase for skills that were taught by the participant vs pre-existing skills in our skill library. This demonstrates that we can add new skill without loss of performance to our existing skills. 


In the post-study survey, we administered the Godspeed Likability sub-scale <cit.>, System Usability Scale (SUS)<cit.>, and the NASA TLX <cit.>. 

















  §.§.§ Procedure




The procedure of the study is as follows.
Participants first filled out the consent form and a pre-study survey. 
Then, we handed out a general introduction of the experiment and administered the two phases sequentially. Before each phase, a demonstration video and the instructions for the corresponding phase were provided to the participant. The anonymized instruction manual and the link to these videos are provided in the Appendix and the associated webpage[https://sites.google.com/view/corl-24-dialog]. 

The interaction phase:- Here the participant requested the robot agent to make a type of sandwich. During the process, the robot agent asked the participant for task knowledge, human demonstrations, or robot demonstrations through dialogues. The participant answered task knowledge-relevant questions directly with their language responses and provided human demonstrations or robot demonstrations on request. We recorded all replies from the participants in audio and converted them to text using audio-to-text tools. At the end of the interaction phase, the participant was asked to fill out a survey to evaluate the experience with the robot. The three conditions that the participant observed here were: 1) Doing - the participant performed the skill that the robot does not know; 2) Learning - The participant provides a demonstration to the robot so it can learn the skill for the evaluation phase; 3) Dumb - The dialog state machine acts randomly and is not guaranteed to complete the sandwich. The participants observe all these conditions in random order.


The evaluation phase:- The participant comes back and asks the robot agent to make the exact same sandwich as the one requested in the interaction phase. The participant watches the robot finish the task while helping in certain conditions, finally filling a survey to evaluate the robot's performance. The three conditions observed in this phase are - 1) Doing - the participant performed the skill that the robot does not know; 2) ACT-LoRA - the robot makes the whole sandwich based on the participants data from the interaction phase with our ACT-LoRA model; 3) TAIL - the robot makes the sandwich with participant data but with the TAIL <cit.> model.




  §.§.§ Hypotheses


In the human-subjects study, we aim to verify the following hypotheses:

Hypothesis 1 - Our framework allows the robot to learn skills continually with human data while performing better than the existing framework of TAIL. 

Hypothesis 2 - Our framework allows the robot to complete the task autonomously when compared to the existing baseline of TAIL.

Hypothesis 3 - Our framework is preferred by the users when compared against a baseline of the robot asking for help and expecting the human to complete the unknown skill.


	




§ EXPERIMENTAL RESULTS

0.45
0.45!

Model        Pre-trained Skills(SR)       Few-shot Skills(SR) 
 
ACT-LoRA    74.75     100.0          

ACT    1.5     100.0          

TAIL    0.25      5.0 



Experimental results on RLBench simulator. Pre-trained skills(SR) measures the policies' average success rate on the 8 skills that policies are pre-trained on. Few-shot skills(SR) measures the policies' average success rate on the 6 new skills that they are finetuned on over 50 rollouts. Detailed results of each skill are in Appendix <ref>.





In this section, we present three sets of experimental results.
Firstly, we present the results of our policy on few-shot continual imitation learning in the simulated RLBench environment <cit.>.
These experiment results show that our behavior cloning model is able to continually to learn novel skills with only few demonstrations and avoid catastrophic forgetting.
Then, we evaluate our demonstration alignment model on a subset of the RH20T dataset <cit.>, and demonstrate that we are able to project demonstrations from different embodiments into the same latent space.
Lastly, we present the results for the human subject study, which demonstrate that our framework can learn symbols and skills to execute task from interacting with true human users.



 §.§ Experiments on Continual Imitation Learning



We evaluate our policy on few-shot continual imitation learning using the RLBench environment <cit.>. 
A total of 14 skills are chosen from the pre-defined skills of the environment, 8 for pre-training and 6 for continual training.
We use 1000 demonstrations for each of the pre-training skills for training during the pre-training phase, and 5 demonstrations for each of the continual training skills in the continual training phase.
The SoTA visual policy model ACT <cit.> and SoTA continual policy learning model TAIL <cit.> were chosen as the baselines for comparison against our model.
We evaluate all models for 50 times on each of the 14 skills to measure the success rate. Our model learns novel skills with 100% accuracy while maintaing its pre-trained performance at 74.75% demonstrating its suitability for continual learning. We observed TAIL <cit.> to fail in tasks which require precision, and ACT fail to remember older skills. We evaluated these models with just one random seed due to computational challenges.






 §.§ Experiments with our alignment model


Our alignment model is evaluated on a subset of the RH20T dataset <cit.>, which includes robot trajectories for diverse range of tasks and their corresponding human demonstration videos. Our alignment model achieves 91.4% in overall accuracy in distinguishing whether a pair of demonstrations are performing the same task. Detailed results are in Appendix <ref>. This high performing alignment model allows our robot to ask demonstration queries in our user study.









 §.§ Human Subject Study Results


We conducted an IRB approved study with 8 participants and 6 pilot subjects. 

Only one of the subjects was female (12.5% of the user study). The age demographic of our users is 25.25 ± 3.059. The subjects spent 120 minutes in the interaction phase and then another 75 minutes for the evaluation phase. They were compensated with a $35 Amazon gift card.  

We present our objective success rate results in Tables <ref>,  <ref>. 
Our Hypothesis 1 is supported by table 2 as our ACT-LoRA when compared to TAIL <cit.> learns the novel skills (100% vs 5%) and existing skills (74.75% vs 0.25%) at a higher success rate. This allows better acquisition of novel skills few shot in a real robotics domain. We expected TAIL to perform better but we noticed it had issues with long horizon precise skills. Note that the doing model does not learn any new skills. 

Our Hypothesis 2 is supported by Table 2 demonstrating that our LoRA Act model completes the entire sandwich at a higher rate than TAIL <cit.> which is the learning based approach. Notice that the Doing model has a similar completion rate as even though the human can help with some tasks the robot might still fail in skills it has learned previously such as grasping bread.

We reject Hypothesis 3 as we did not notice any preferences in Workload, System Usability or Likebility between our learning agent using ACT-LoRA and the Doing agent that always asks for help from users. While we thought that users would prefer an autonomous agent, subjects actually preferred helping the agent as humans can make a sandwich with trivial ease where as the robot performs the task at a much slower rate.    













§ RELATED WORK



Skill Discovery and Continual Learning. 

The area of visuo-motor continual learning is getting a lot of attention recently <cit.>.
<cit.> discover new skills from segments of demonstrations by unsupervised incremental clustering.
<cit.> learn the skill representation by aligning skills from different embodiments, and can re-compose the learned skills to complete a novel combination.
<cit.> introduce task-specific adapters using low-rank adaptation techniques <cit.>, preventing the agent from forgetting the learned skills when learning the new skills.
However, these frameworks assume the presence of the demonstrations for the new tasks, and only discover skills in a passive fashion. 
Our proposed framework actively reasons and requests the human users for the demonstrations of the unseen skills while performing the ones it knows.
This reasoning is done in two stages: first the human enacts the behavior, once the robot has seen this behavior it decides if it can perform the enacted behavior or not. After this reasoning the robot can choose to source demonstrations from the human using a joystick.

This is a more natural setup for a language enabled continual learning agent in the real world. 

Furthermore, our agent requires less than ten demonstrations from the user to discover the new task without forgetting any of the learned skills which is an improvement over existing passive continual learning methods <cit.>. 


Human-Robot Dialogue. Human-Robot dialog is a mature problem <cit.>. 
Traditional methods use statistical algorithms with a pre-defined grammar, such as semantic parsing <cit.>, to connect the semantics of the dialogue to the environment's perceptual inputs. 

On the other hand, recent advancements in natural language processing (NLP) have led to Large Language Models (LLMs) that process natural language in free form.
Grounded with perceptive inputs from the environment, these LLMs have been used in robotics research generate executable plans <cit.>.
Furthermore, <cit.> and <cit.> use LLMs to ask for human feedback for the robot agents demonstrating the importance of dialog.

However, these approaches leverage planning with LLMs where as we are attempting to learn continuous visuo-motor skills on the robot by asking for help.



Active Learning. Our work is related to active learning, where a learning agent actively improves its skills by asking a human for demonstrations <cit.>.
Defining an appropriate metric that triggers the request for assistance or information gathering becomes the key research problem in this domain.
<cit.> measure the semantic similarity between a newly introduced concept and the known concepts to ask for classifier labels.

<cit.> train a confidence classifier conditioned on the current state of the agent, and request expert demonstrations when the confidence score does not meet a pre-defined threshold.
<cit.> use the uncertainty of Gaussian Processes(GPs) as the metric to trigger the request for assistance.
These existing methods reason over the semantic information in a task such as the goal condition or features of classifiers that identify the goal condition.


We use a cosine distance metric to measure similarity for both the semantic information from language and the behavior information of a skill. 










§ LIMITATIONS



We present an approach to teach skills to robots using techniques from active learning and continual learning while using language as a modality to query and reason over the skills known to the agent. We acknowledge that we need to conduct a wider user study with more subjects over a larger number of cooking tasks using our approach. 
The turn-taking in our framework is tightly controlled, and not dynamic.
Our ACT-LoRA approach while being sample efficient has been observed to have issues with heterogeneous demonstrations. 

We also want to compare such continual learning approaches with pre-trained policy approaches such as RT <cit.> to scale up the policy learning approach while maintaining sample efficiency allowing for novice users to personalize skills for their robots.   





§ CONCLUSION


In conclusion, we present a novel framework for robot agents to learn task relevant knowledge and skills from interactions with human users. To the best of our knowledge this is the first work to demonstrate skill learning while querying a user with dialog to express doubt.  
By maintaining metrics in semantic and skill similarity, our agent can actively interact with human users and adapt its known skills to novel tasks. 

Furthermore, our framework is able to learn a completely new visual-motor skill (at 100%) with only a few robot demonstrations, without affecting the performance of any existing skills (at 74.75%)fulfilling continual learning requirements in robotics. 

Finally, we conducted a human-subjects experiment to demonstrate our framework's ability to complete tasks such as sandwich making from interactions with participants at a 75% task success rate.









If a paper is accepted, the final camera-ready version will (and probably should) include acknowledgments. All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.




  



§ APPENDIX



§ EXPERIMENT DETAILS



0.55
0.55!

Model        Precision       Recall     F_1     Accuracy 
 
Resnet + Transformer     88.0± 2.2     95.9 ± 2.0     91.8 ± 2.0     91.4 ± 2.11  



Experimental results of our alignment model on aligning human videos with robot trajectories on subset of the RH20T dataset <cit.>.







 §.§ Detailed results of the alignment model on RH20T

We present the detailed results of the alignment model in Table <ref>.
We conduct five-split evaluation on the dataset, and report the mean score and standard deviation of each metric. Each model is trained on 80% of the trajectories and evaluated on the other 20%. In total, we use 1240 robot trajectories and 1193 human demonstrations across 98 tasks of the RH20T dataset configuration 5.
As shown in Table <ref>, our model achieves 91.8% on the F_1 metric, and 91.4% on the overall accuracy metric. This strong performance of the alignment model enables the robot agent to actively adapt learned skills to perform novel task, or to understand that it needs to learn a novel skill from seeing a single human demonstration. 



 §.§ Detailed results of the continual learning policy on each task of the RLBench




We present the per-task success rate of the policies in the RLBench simulator. Table <ref> shows the performance of the three policies on each pre-trained task after fine-tuning, and Table <ref> demonstrates the performance of the policies on the tasks that they are finetuned on. All the three models are trained to predict joint positions for the same number of gradient steps.
During the pre-train phase, each model is trained with 1000 robot demonstrations from each pre-train task for 1000 epochs. In the fine-tune phase, each model is trained with 5 robot demonstrations from each fine-tune task for 20000 epochs. 
Notice that due to the limitation of the visual-motor policies, we use a static location for all the finetune tasks during both training and evaluations. However, for all the pre-train tasks, we use randomized locations during both training and evaluation.
As presented in the tables, TAIL achieves a near 0% success rate on majority of the tasks except for close fridge. This is because that close fridge is a relatively easier task in the environment, and the agent has a non-trivial chance to accidentally hit the fridge door and close it even if it is doing random behaviors. On the other hand, the baseline ACT model achieves a strong 100% success rate on the tasks that it is fine-tuned on, demonstrating its strong capability of learning fine-grained control. However, it also achieves a extremely low success rate on all the pre-train tasks after fine-tuning. This shows that ACT suffers from catastrophic forgetting and can no longer perform the pre-train tasks after fine-tuning. In comparison, ACT-LoRA achieves a 100% success rate on the fine-tune tasks, while still being able to perform on all the pre-train tasks with an overall success rate of 74.5%. This experiment result demonstrates that ACT-LoRA inherited the capability of fine-grained controls from ACT, and the ability to prevent catastrophic forgetting from the additional Low-Rank Adaptors, and hence is suitable for the use case where fine-grained control and continual learning are needed.


Limitation. ACT-LoRA does not generalize well to situations when the objects are present in completely novel positions. Similarly learning tasks with heterogeneous demonstrations is difficult for ACT-LoRA as there is not encoding for the possibility of heterogeneous demonstrations. These limitations exist with our baselines as well. Diffusion based policies can solve this problem but do not work well with fewer than five demonstrations in our experience.



 §.§ Detailed results of the human-subjects study






























  §.§.§ Study details

We describe the details of the human-subjects study. 
Our human-subjects study is approved by the Institutional Review Board(IRB) of the university.


We tested the study with 5 pilots before conducting the experiments on the participants. We fixed the issues of unclear instructions, short execution times for the learned skills and ambiguous phrases when the LLM was asking questions. We had to finetune the prompts of the LLMs a lot so the robot asked questions pertinant to the task of sandwich making.  







For the actual study a total of 10 participants were recruited through campus advertisements.
We rejected the 2 of these users for the following reasons respectively. One user was over-excited to interact with the talking robot agent, and requested the robot to perform tasks that are not in our instructions forcing us to stop the study. These tasks were impossible to complete using the configurations of the study. The other user did not fully understand the instructions, and crashed the robot into the table forcing us to stop the study. 










The study is composed of two separate phases, the interaction phase that takes 150 minutes and the evaluation phase that takes 60 minutes, with a voluntary participation. The participants, including the pilots, are compensated with $35 Amazon gift card for their time.

We designed the two-phase study for two major reasons. Firstly, the learning agent requires five hours to train for the novel skill. Secondly, we want to demonstrate a thorough comparison for the workload between our learning agent and the doing agent in the two phases.
The learning agent requires the users to remotely control the robot arm to perform the task in the interaction phase, and is fully automated in the evaluation phase, whereas the doing agent behaves the same in both phases by requesting the users to directly perform the task that it does not know.
We hypothesize that the users experience higher workload for our learning agent than the doing agent in the interaction phase, and a lower workload for the learning agent than the doing agent in the evaluation phase because we consider that for remotely controlling the robot arm to complete the task requires higher workload than directly completing the task themselves for the users, and the fully automated robot agent requests the least workload.
We reject our hypothesis and accept the null hypothesis of –  there is no difference in the users's perception of workload, likeability, anthropomorphism, and perceived intelligence between our method and a baseline where the robot just asks for help.
There are two major reasons for this result. 
Firstly, the human users take much shorter time to complete the task than the robot agent. From the users' perspective, despite being fully automated, the agent is not saving their time by doing the task for them because they have to sit and watch the agent doing the task. This issue can be addressed by assigning distractor tasks for the human users <cit.>, which could not be achieved due to the limited time.
Another reason is that a robot that asks for help seems more intelligent and human like compared to an autonomous sandwich making robot. Future studies will have to weigh these design decisions more carefully.
We do perform better than the baseline of TAIL here but TAIL generally fails at completing the task itself which makes the result unsurprising. 
A more elaborate study with more choices and more participants can reduce the variance in the results.



  §.§.§ Detailed procedure

We describe the detailed procedure for the study as follows.

Interaction Phase. Participants first filled out the consent form and a pre-study survey. Then, we handed out a general introduction of the experiment. The participants were then asked to read the instructions for the interaction phase, and watch a demonstration video. The demonstration video introduces how the robot agent requests for different types of help differently, and how to answer different requests from the robot agent. We use a completely different domain(Opening a washing-machine) as example in the demonstration video. The instruction introduces domain relevant information, such as the configuration of the robot's workspace, the sandwich to make, and the steps to make the sandwich. The anonymized instructions and the video can be found on the associated webpage. Then, the participants interacted with the three agents, the dumb agent, the doing agent, and the learning agent, in a random order. The dumb agent never interacts with the users except for getting the initial instruction set from the user. The doing agent always asks the human users for help when it encounters any task that it is uncertain with. The learning agent interacts with the human users by asking task-relevant questions, asking for human demonstration, and asking for robot demonstrations. The details of how the learning agent behaves when encountered an unknown task can be found in Appendix <ref>. 
After interacting with each system, the participants were asked to fill-out a post-survey, including questions from NASA-TLX <cit.>, SUS <cit.>, and 4 sub-scales from the GodSpeed Questionnaire Series <cit.>(Likability, Animacy, Natural, Perceived Intelligence). After the participants finished the interaction phase, we fine-tuned the ACT-LoRA policy and the TAIL policy with the robot demonstrations collected from the users.

Evaluation Phase. Participants came back to the lab. We handed the same instructions to the participants for them to ask the robot to make the same sandwich. The participants interacted with three robot agents, the doing agent, the learning agent with TAIL, and the learning agent with ACT-LoRA. All the three agents remember the instructions to make the sandwich provided by the participants from the interaction phase. The doing agent did not learn from the robot demonstrations from the interaction phase, and asked for help from the human users on the same task. Both learning agents learned the novel skill from the demonstration in the interaction phase, and did not interact with the human users except for the initial interactions. After watching each agent, the participants were asked to fill out the same post-survey for the system. After watching all the three systems, the participants were asked to rank the three systems on 7 different description(helpful, useful, efficient, competent, uncooperative, inefficient, incompetent).



  §.§.§ Detailed study results

The objective results of the study are presented in Table <ref> and Table <ref>, and the subjective results of the study are presented in Table <ref> and Table <ref>.
Although no hypothesis can be accepted in the comparison of any metric between the learning agent with ACT-LoRA and the doing agent with ACT-LoRA in both the interaction phase and the evaluation phase, the basic hypothesis that the learning agent with ACT-LoRA has a higher success rate than the dumb agent holds, because that the dumb agent will behave randomly on the skill that it doesn't know, and is fated to fail. 


For a similar reason, the basic hypothesis that the learning agent with ACT-LoRA has a higher success rate than the learning agent with TAIL also holds. Furthermore, in the evaluation phase, the learning agent with ACT-LoRA is considered better than the learning agent with TAIL by users in the following metrics:

Comparative. Conditions for normality were not met for the data points to run a t-test. Hence, we conducted a Wilcoxon Signed-Rank test to compare the learning agent with ACT-LoRA with the learning agent with TAIL in the comparative metric. Results from Wilcoxon Signed-Rank test suggest that the ACT-LoRA learning agent is preferred by user in the direct comparison with the TAIL learning agent with significance(p<0.01, Z=2.47).

SUS. Results from Shapiro Wilk test suggest that our data in the SUS metric satisfies the condition for a parametric test(W=0.93, p=0.24). Results from the paired t-test suggest that the LoRA-ACT learning agent is considered better than the TAIL learning agent by users in the SUS metric with significance(p<0.01, t=4.15).

Natural. Conditions for normality were not met for the data points to run a t-test. Hence, we conducted a Wilcoxon Signed-Rank test to compare the learning agent with ACT-LoRA with the learning agent with TAIL in the natural metric. Results from Wilcoxon Signed-Rank test suggest that the ACT-LoRA learning agent is preferred by user in the direct comparison with the TAIL learning agent with significance(p<0.05, Z=2.25).

Likability. Results from Shapiro Wilk test suggest that our data in the likability metric satisfies the condition for a parametric test(W=0.93, p=0.23). Results from the paired t-test suggest that the LoRA-ACT learning agent is considered better than the TAIL learning agent by users in the likability metric with significance(p<0.05, t=2.19).

Animacy. Results from Shapiro Wilk test suggest that our data in the animacy metric satisfies the condition for a parametric test(W=0.93, p=0.27). Results from the paired t-test suggest that the LoRA-ACT learning agent is considered better than the TAIL learning agent by users in the animacy metric with significance(p<0.05, t=2.87).

Perceived Intelligence. Results from Shapiro Wilk test suggest that our data in the perceived intelligence metric satisfies the condition for a parametric test(W=0.95, p=0.41). Results from the paired t-test suggest that the LoRA-ACT learning agent is considered better than the TAIL learning agent by users in the perceived intelligence metric with significance(p<0.01, t=3.39).



  §.§.§ Limitation of the study

There are two major limitations on the human-subjects study.
Firstly we need a better distractor based study which needs to be designed more carefully. Our current result just demonstrates the success of making sandwhichs but that does not mean people prefer such a robot. 
Secondly, due to the limited time, the study was conducted on a single sandwich with a small number of participants. We would like to conduct a more elaborate study with more domains and participants in the future.







§ IMPLEMENTATION DETAILS







 §.§ Implementation details of the dialogue state machine


We describe the details of the implementation of the dialogue state machine. Algorithm <ref> is the pseudo code of the dialogue state machine. The robot agent first initializes the conversation with the human user, and repeatively asks questions until it obtains a clear list of instructions from this initial conversation. Then, the agent attempts to execute the list of actions sequentially until all the instructions are finished.

During execution of each task, if the agent finds that the task can be executed with one of the known skills, the agent directly executes the task with the corresponding policy. 
We use a very high threshold for the cosine similarity metric to simulate an exact match to prevent the robot agent to ask trivial questions in the user study. 
If the robot agent fails to directly find an executable skill for the task, it first searches for a usable skill in the semantic space. 
If it finds  a skill that has a higher similarity score than the threshold in the semantic space, it proposes to the human user to use this skill to execute the task, and proceeds after obtaining the agreement from the human user. 
Otherwise, if the agent fails to find a usable skill, or the human user rejects the agent's proposed skill, the agent asks the human user for a human demonstration, and attempts to find a usable skill in the skill space based on the human demonstration. 
The skill search in the skill space is similar to that of the semantic space. 
If the agent finds a skill that has a higher similarity score than the threshold in the skill space, it proposes the skill to the human user. 
If the human user agrees with such skill proposal, the agent learns that a known skill can be adapted to the new task and executes the task. 
Otherwise, if the agent fails to find an aligned skill or its proposal is rejected by the human user, it realizes that it doesn't possess the skill to execute the task, and will ask for several robot demonstrations to train a completely new skill for the task.

The LLM serves as the interface between the robot agent and the human user. Whenever the robot agent is in a state that it needs inputs from the human user, it prompts the LLM with the current state of the agent and the information needed from the human user. The LLM then initiates a dialogue with the human user, and continues the dialogue until it retrieves the information needed by the robot agent. Such share autonomy between the state machine has more reliability than fully relying on the LLM, and can fully exploit the linguistic capability of the LLM.



 §.§ Implementation details of the alignment model

We describe the details of the implementation of the alignment model. Following the notation in the main paper, we use E_robot, and E_human to denote the robot trajectory encoder and human demonstration encoder respectively. We also use ϵ_t and ϵ_v to denote the different thresholds for training and validation. To reduce the computational cost, we downsample all the human demonstrations and robot trajectories to 100 timesteps uniformly, and use image inputs from a single camera for both the human demonstrations and robot trajectories.
We use a 6-layer transformer encoder with 8 heads for both the human demonstration encoder and the robot trajectory encoder. Both encoders use a resnet-18 feature extractor to extract features from the raw image inputs. The robot trajectory encoder also takes in proprioceptive inputs from each time-step.
During training, we minimize the cosine embedding loss between the human demonstration and robot trajectory with the training threshold ψ_t, denoted as following:

    L(d,τ, y) = 
        1 - cos(E_human(d), E_robot(τ))    if y=1,
    max(0, cos(E_human(d), E_robot(τ)) - ϵ_t)    if y=-1.
 
During inference, two trajectories are said to be the same skill if their cosine similarity is above the threshold ϵ_v. For the experiment of RH20T, we use ϵ_t = 0.5 and ϵ_v = 0.7, and train the alignment model for 10000 gradient step with a batch size of 16. 
For the domain of the human subject study, we use ϵ_t = 0.5 and ϵ_v = 0.95. Furthermore, we freeze the pre-trained resnet-18 visual encoder, and train the model for 3000 steps in prevention of over-fitting due to the lack of data points in the real world domain.




 §.§ Implementation details for ACT-LoRA
 
We describe the details of our implementation of the ACT-LoRA policy. Following <cit.>, we train with a CVAE architecture and discard the additional encoder during inference. For both the CVAE encoder and the state encoder, we use a 4-layer transformer encoder with 8 heads. We extract features from raw image inputs from multiple cameras using resnet-18. These visual features are fed to the transformer encoder along with the proprioceptive inputs. For the decoder side, we use 6-layer transformer decoder with trainable embeddings. We also use a chunk size of 100 as it gives the best performance empirically <cit.>. The same configuration is also used for the baseline ACT model.
As for the configuration of the low-rank adaptors, we follow TAIL <cit.> and use a rank size of 8. For both the simulation experiments and the human subject study, each skill is associated with a set of unique adaptor weights. 



 §.§ Implementation details for TAIL
 
As there is no publicly available source code for TAIL <cit.>, we tried our best attempt to re-implement TAIL for a fair comparison. To reduce the computation cost for the original TAIL model, we use a transformer encoder in replacement to the GPT-2 temporal decoder to speed up the training process. Furthermore, due to the limited time, the LoRA weights are only introduced to the transformer encoder, but not to any pre-trained feature extractors, including the CLIP text encoder and CLIP image encoder. Apart from these changes, we choose hyperparameters as close as possible to the original TAIL paper <cit.>.
The TAIL model takes in linguistic task descriptions, image observations, and proprioceptive inputs over history timesteps.
We first extract the feature of the raw image inputs and the linguistic task descriptions using the pretrained CLIP image and text encoder. Then, we use a FiLM layer to inject the linguistic features into the image features and the proprioceptive inputs. These inputs are treated as the input tokens of the transformer encoder. Then, we use an MLP layer to project the encoded token into parameters for a Gaussian Mixture Model(GMM). During training, the model is optimized by minimizing the negative log-likelihood loss of the ground truth actions. During inference, we sample from the distribution of the GMM predicted by the model.



 §.§ Robot Setup for User Study and Data Collection

The robotic setup includes a Franka Robot and 3 Intel Realsense D435 cameras. The workspace includes a table with items curated for the system. We designed 3D-printed tools tailored to support our task requirements as an attachment for the Franka Robot. We set up our cameras to provide a frontal view, a top-down view, and a wrist-mounted camera for a view from the robot's perspective. This configuration allows us to capture dense and diverse features for training our policy. Our data collection pipeline includes a 6D Spacemouse from 3DConnexion, which dictates the motion of the robot end effector. This facilitates the collection of dense data. Although limited by the data collection rate, this setup allows users to control the robot in the task space with relative ease because of the intuitive nature of the Spacemouse. Throughout the system's operation, picking and placing robot tools is done by pre-specified waypoints because grasping a tool is not our focus.



