E2CL: Exploration-based Error Correction Learning for Embodied Agents
Hanlin Wang Chak Tou Leong Jian Wang Wenjie Li

No Man is an Island: Towards Fully Automatic Programming by Code Search, Code Generation and Program Repair
    Zhenyu Chen
    September 5, 2024
===========================================================================================================

§ ABSTRACT 

Language models are exhibiting increasing capability in knowledge utilization and reasoning. However, when applied as agents in embodied environments, they often suffer from misalignment between their intrinsic knowledge and environmental knowledge, leading to infeasible actions. Traditional environment alignment methods, such as supervised learning on expert trajectories and reinforcement learning, face limitations in covering environmental knowledge and achieving efficient convergence, respectively. Inspired by human learning, we propose Exploration-based Error Correction Learning (E^2CL), a novel framework that leverages exploration-induced errors and environmental feedback to enhance environment alignment for LM-based agents. E^2CL incorporates teacher-guided and teacher-free exploration to gather environmental feedback and correct erroneous actions. The agent learns to provide feedback and self-correct, thereby enhancing its adaptability to target environments. Evaluations in the Virtualhome environment demonstrate that E^2CL-trained agents outperform those trained by baseline methods and exhibit superior self-correction capabilities.






§ INTRODUCTION


Language Models (LMs) are becoming increasingly capable of knowledge utilization and reasoning across various knowledge-intensive tasks <cit.>. This success motivates researchers to apply LMs to build LM-based agents in embodied environments, which similarly requires the use of reasoning and planning upon environmental knowledge <cit.>. 
In this case, LM-based agents are asked to plan appropriate actions based on the given environmental information and the history of actions already taken.
However, the knowledge acquired by these LM-based agents comes from general-purpose corpora during pre-training, and as a result the intrinsic knowledge of these models often misalign with environmental knowledge.
Such environmental knowledge involves physical constraints that LMs have not yet explored.
For example, if the embodied agent holds two objects, it is prohibited to grab one more other object.
This misalignment causes LM-based agents to frequently infer actions that cannot be executed in the environment, hindering their application in real-world environments.



To address this issue, two primary types of environment alignment methods have been explored. 
The first type involves having LM-based agents undergo supervised learning on expert trajectories, which are human-labeled sequences of observations and actions <cit.>. 
Nevertheless, these trajectories often fail to fully cover the knowledge within the environment, such as scenarios where certain actions cannot be executed. 
The second type is based on reinforcement learning, which allows agents to freely explore the environment, collect trajectories that comprehensively cover the environment's knowledge, and obtain rewards based on these trajectories' success or failure <cit.>. 
However, the rewards are sparsely obtained because the performance evaluation of the agent is based on a complete trajectory. This makes the learning process difficult to converge.

Human learning is not comprehensive or efficient if it relies solely on imitating experts' behavior or merely knowing whether an action is correct. Instead, by collecting and understanding feedback from the environment via exploration and learning to correct errors based on the feedback, humans can learn comprehensively and efficiently.
Inspired by this, we propose a novel exploration framework for LM-based agents to align with environments, which is called Exploration-based Error Correction Learning (E^2CL).
As depicted in <Ref>, our framework incorporates exploration-induced errors and environmental feedback, leading to a comprehensive alignment with target environments.

During the process of E^2CL, we ask a pretrained agent to perform predefined tasks and explore the environment to collect experiences in both efficient and comprehensive manners. This is achieved by two different proposed schemes, namely teacher-guided exploration and teacher-free exploration.
The former prompts the agent to perform one-step exploration given sliced expert trajectories, whereas the latter allows the agent to continue exploring until it infers a stop.
In these two exploration phases, we collect the feedback given by the environment when the agent makes errors, as well as the correct actions corresponding to these error actions.
Having these exploration trajectories with additional correction, we train the agent to provide feedback on their trajectories and correct their error actions based on the feedback.
To apply learned self-correction ability, we further propose Speculative Inference, which performs corrections if the initial planned actions are inferred to be errors by the agent's feedback. 

We evaluate the agent trained by E^2CL in Virtualhome (), a household embodied environment. E^2CL-trained agent surpasses the agents trained by other baseline methods in all agentic metrics, demonstrating its effectiveness. 
Furthermore, our analysis reveals that the small models constructed using our method outperform larger models of the same series that have only undergone behavior cloning. In addition, in evaluations based on feedback-driven re-planning, our models demonstrate self-correction capabilities that are comparable to LLMs.

In summary, our main contributions are as follows. (1) We introduce the Exploration-based Error Correction Learning (E^2CL) framework, enabling LM-based agents to align with environments through effective feedback-driven exploration and correction. (2) We propose two novel exploration schemes, teacher-guided and teacher-free exploration, that facilitate the collection of correction and feedback via agent-environment interaction. (3) We introduce a novel action inference algorithm, namely speculative inference, which can prevent executable errors from occurring. (4) We demonstrate the superior performance of E^2CL-trained agents in the Virtualhome environment, surpassing baseline methods and showcasing the potential of our approach for real-world deployment. 



§ METHOD


In this section, we introduce our framework, E^2CL. This framework equips LM-based agents with self-feedback and self-correction capabilities, which enhances their ability to tackle tasks in new environments.



 §.§ Task Formulation




The LM-based embodied agent is asked to complete a set of tasks via interacting with a virtual environment.
The interaction between the agent and the environment can be formalized as a partially observable Markov decision process (POMDP) (𝒬, 𝒮, 𝒜, 𝒪, 𝒯, ℛ) with instruction space 𝒬, state space 𝒮, action space 𝒜, observation space 𝒪, transition function 𝒯: 𝒮×𝒜→𝒮, and reward function ℛ:𝒮×𝒜→[0,1].
In our LM-based agent scenario, 𝒬, 𝒜, 𝒪 are subsets of language space.

The interaction process between the agent and the environment is described as follows. Given a planning instruction q_p∈𝒬 that prompts the agent to plan for a task, the agent with parameter θ generates the first action a_1∼π_θ(·|q_p) ∈𝒜 according to its policy π_θ. Each action at step t induces a transformation in the latent state space s_t ∈𝒮.

And the agent would face a new observation o_t∈𝒪. Then the agent would incorporate task instruction q_p and interaction trajectories j_t = (a_1,o_1,…,a_t,o_t) to generate the next action a_t+1∼π_θ(·|q_p,j_t). The interaction loop repeats until the agent assumes the task is finished or the number of steps exceeds the maximum steps.





 §.§ Exploration-based Error Correction Learning



Our E^2CL framework consists of three phases of learning and exploration within the environment: the pre-tuning phase, the exploration phase, and the training phase. In the pre-tuning phase, we equip the agent with basic planning ability before exploration. Then, in the exploration phase, the agent collects exploration experience in the environment via two complementary schemes, as shown in <Ref>. Following this, in the training phase, the agent is trained to align with the environmental knowledge from the collected experience and to develop the ability to provide feedback and correct its own errors.




  
Pre-tuning Phase
To serve as the foundation for environmental exploration, we aim to empower LM-based embodied agents with a basic planning capability. Given a dataset 𝒥={(q^i_p,j^i_n_i)}^|𝒥| with |𝒥| task instructions and expert trajectories where each trajectory has n_i steps, we first construct a planning dataset D_p by slicing each trajectory into sub-trajectories of varying lengths from 1 to n_i. Formally, the planning dataset D_p is defined as: 

D_p = ⋃_i=1^|𝒥|⋃_t=1^n_i{(q^i_p, j^i_t) | j^i_t ⊆ j^i_n_i (q^i_p,j^i_n_i) ∈𝒥}.
Notably, we sample a subset of D_p, denoted as D_p', for pre-tuning to avoid overfitting to expert trajectories and maintain exploration diversity.
Then, we fine-tune the LM-based agent by minimizing neg-likelihood loss:



    ℒ(θ)    = 𝔼_∼𝒟_p'[ -logπ_θ(a_t | (q_p, j_t-1)) ].


Notably, a_t consists of multiple tokens. Therefore, when calculating the loss, it effectively becomes an auto-regressive loss over a sequence of tokens, following previous practices. This approach is consistently applied in the latter stages of training as well.



  
Exploration Phase


Intuitively, to gather diverse experiences that fully cover environmental knowledge, we can simply let the agent freely execute its predicted plans and collect the trajectories. 
However, when utilizing these trajectories, we need to correct the errors made by the agent.
Since these trajectories are newly generated, we do not have the correct action data corresponding to the errors. Although one can use a more powerful LLM to correct these errors automatically, the quality of the generated data is inevitably lower compared to expert data.
To balance data diversity and quality, we propose a limited exploration scheme guided by expert trajectories, referred to as teacher-guided exploration (TGE). 
Correspondingly, we call the aforementioned free exploration scheme teacher-free exploration (TFE).
These two schemes complement each other and enhance the diversity and quality of the collected experiences.


Specifically, for each expert's sub-trajectory (q_p, j_t)∈ D_p, the agent conducts TGE by executing the action â_t ∼π_θ(·|q_p, j_t-1).
The environment then provides feedback f_t indicating the executability of this action.
Since the agent only performs one step of exploration under the guidance of the expert, we can naturally use a_t as the ground truth action for that step.
After traversing all the expert trajectories, we obtain the feedback dataset D_f^TGE consisting of samples in the form of (q_f, j_t-1, â_t, f_t), and the correction dataset D_c^TGE consisting of samples in the form of (q_c, j_t-1, â_t, f_t, a_t) where â_t≠ a_t, q_f is the instruction that prompts the model to generate feedback, and q_c is the instruction that prompts the model to correct errors. Please refer to <Ref> for templates of these samples.

During TFE, the agent iterates through each task instruction q_p∼𝒬 and accrodingly obtain trajectories j_t=(â_1,ô_1,…,â_t,ô_t). Similar to TGE, whenever the agent predicts a non-executable â_t, the environment provides feedback f_t indicating why this action is non-executable. To obtain the executable action at this step without manual intervention, we leverage an LLM with powerful reasoning ability (e.g., GPT-4o) to automatically correct the action, yielding a_t. Considering the LLM may not always provide perfect corrections due to a lack of environment alignment, we further filter its predictions to ensure the corrections are executable. As a result, we obtain the feedback dataset D_f^TFE and D_c^TFE, each in the same form as the samples in D_f^TGE and D_c^TGE, respectively.




  
Training Phase

After the above two phases, we obtain the planning dataset D_p, the feedback dataset D_f=D_f^TGE⋃ D_f^TFE, and the correction dataset D_c=D_c^TGE⋃ D_c^TFE.

Next, we train the agent to align with the environmental knowledge gathered from these datasets and to develop the ability to provide feedback and correct its own errors.
This is achieved by fine-tuning the agent to minimize the following losses:




    ℒ_p(θ) = 𝔼_∼𝒟_p[ -logπ_θ(a_t | q_p, j_t-1) ], 
       ℒ_f(θ) = 𝔼_∼𝒟_f[ -logπ_θ(f_t | q_f, j_t-1, â_t) ], 
       ℒ_c(θ) = 𝔼_∼𝒟_c[ -logπ_θ(a_t | q_c, j_t-1, â_t, f_t) ], 
       ℒ_total(θ) = ℒ_p(θ) + ℒ_f(θ) + ℒ_c(θ).


We refer the reader to the pseudo-code of the overall E^2CL process in Appendix <ref>.



 §.§ Speculative Inference


To utilize the learned abilities in the training phase, we propose speculative inference, a process of predicting the error occurring ahead and correcting itself. This process reduces execution errors and generates correct action based on self-generated feedback.

To be more precise, when given each test task instruction q_p, the agent initially predicts a action â_t ∼π_θ(·|q_p, j_t-1). However, this action â_t will not be executed immediately. The agent will `reflect' itself and generate an environment feedback f̂_t ∼π_θ(·|q_f, j_t-1, â_t). If the agent believes the initial action â_t is executable, then this action will be executed. Otherwise, the embodied agent will correct this action â and predict a new action â_c ∼π_θ (·|q_c, j_t-1, â_t, f̂_t). Once the corrected action â_c passes its own check, this action will be executed at this step and concatenated into trajectory ĵ_t-1. The above process is iterated until the agent assumes the task is finished or the total steps exceed the maximum threshold. 
The process of speculative inference for each test task is shown in  <Ref>.







§ EXPERIMENTS




 §.§ Experimental Settings




  
VirtualHome Environment & Tasks

We evaluate our method in the VirtualHome Environment <cit.>, a simulation platform for performing typical household activities. 

During the experiment, we are able to access data, such as states of objects, observation of agents, and environmental feedback, from the simulator. 

We refer to Appendix <ref> for more details of the environment.

We use the predefined tasks from ActivityPrograms Knowledge Base <cit.> for the experiment. It contains 292 unique high-level household tasks, with 1374 unique action plans and 6201 unique environmental settings in total extracted from VirtualHome.

After filtering low-quality tasks, we conduct experiments on a total of 285 tasks. They are randomly divided into a training set of 235 tasks and a test set of 50 tasks. 
We select 50 tasks from the training set as seen tasks, while the 50 tasks in the test set as unseen tasks.
We evaluate the method on both seen tasks and unseen tasks. 







  
Baselines We compare our method with both prompting-based methods and other tuning-based baseline methods. Similar to our approach, tuning-based methods achieve alignment between the embodied agent and the environment via model fine-tuning.
(1) Language-planner <cit.> aims to inject environment knowledge into prompt and prompted Large Language Models to output action. We utilize GPT-4o as the foundational model for our baseline method.
(2) We perform Behavior Cloning (BC) on expert planning data <cit.>, which is the same method used in the pre-tuning phase of our methods and other baselines.
(3) LWM <cit.> employs a robot agent to interact with the environment and collect a large amount of environmental knowledge data to fine-tune the model.
(4) Plasma <cit.> leverages ChatGPT to generate multi-task planning-related data for model training.
(5) Lema <cit.> enhances the agent's reasoning capabilities by providing error-correction data pairs during model fine-tuning.
(6) NAT <cit.> implements a negative-aware training approach, enabling LM-based agents to effectively learn from both positive and negative examples.




  
Evaluation Metrics

Following previous studies <cit.>, we evaluate our action plans across three metrics: executability (Exec.), affordance rate (AR), and longest common sequence (LCS).
Executability measures whether an action plan can be correctly parsed and satisfies the common-sense constraints of the environment. Specifically, the parsed action must contain only allowable action, and the objects must be in the environment. Moreover, the action must satisfy the pre-conditions (e.g., the embodied agent cannot send email before walking to the computer) and post-conditions (e.g., the state of TV changes from closed to open after the agent opens it). Similarly to executablility, affordance rate measures the average percentage of all plan steps that are executable, in cases where the entire plan is not executable. However, executability and affordance rate only can reflect whether the agent could compliant with environment physical constraints, but they cannot reflect whether the plan is correct. LCS calculates the length of the longest common subsequence between generated plans and the ground truth plans, normalized by the maximum length of the two. 



 §.§ Results



As shown in  <Ref>, our method outperforms both prompting-based and tuning-based baseline methods across multiple metrics, showing the superiority of our method. 

Notably, the prompting-based method significantly lags behind all tuning-based methods in different metrics.
Despite this contradicts with the experience that LLMs exhibit exceptional general reasoning capabilities, we observe that the actions generated by prompt-based methods, while seemingly reasonable, fail to comply with the physical constraints of the environment often.
Regarding tuning-based baseline methods, our method demonstrates significant improvements over BC in both seen and unseen tasks. 

Moreover, LWM and Plasma, which are also fed by expert planning data and can be seen as augmented versions of BC, only show a marginal increase in performance.
Compared to these BC-based methods, the method utilizing failure data, i.e., Lema and NAT, demonstrates better performance. Taking a step further, we evolve this idea by training the agent to develop self-feedback and self-correction capabilities through its failure experiences. The results show that our method increases executability-related metrics by up to 15% and LCS by up to 10% compared with Lema and NAT. This demonstrates that these two capabilities effectively enable the agent to align with the environment for task-solving.



 §.§ Ablation Study on Training Data





In this section, we explore the impact of the collected training data, i.e., feedback data D_f and correction data D_c, on overall performance by ablating them in the training. During the inference phase, we employ speculative inference for all settings to ensure consistency.
As shown in <Ref>, we observe that both D_f and D_c are each beneficial for the agent, but lag behind the combination of them. 
We hypothesize that the improvement observed when training with D_c is primarily due to the enhanced self-correction capability of the agent. However, the limited ability to generate high-quality action feedback hampers the effectiveness of self-correction during speculative inference, as demonstrated in Section <ref>.
Compared to the agent training without both D_f and D_c, training with D_p and D_f improves the performance by training to predict environmental feedback, which explicitly aligns with environmental knowledge. However, the weak self-correction capability of the agent constrains the agent from generating executable and correct action in speculative inference, which is demonstrated in  <Ref>.
In our method, we integrate both types of data, enabling our agent to generate higher-quality action feedback and exhibit stronger self-correction abilities. This results in a substantial performance boost compared to other ablation settings.




 §.§ Analysis on Different Size of the Model




To investigate the impact of model size on performance, we train models of different sizes on seen tasks using both BC and our method, and evaluated them on unseen tasks.
Our results are in line with common experience, where larger models perform relatively better across all aspects, indicating that model scale significantly impacts performance.
Moreover, it can also be observed that our method outperforms BC in both Affordance rate and LCS across models with different parameter sizes, which demonstrates that our method consistently provides superior performance regardless of model size.
Notably, when using our method, smaller models achieve performance surpassing larger models using BC across all metrics.
This finding suggests that our method is able to release the potential of small language models and lays the foundation for building agents that work on edge devices in the future.






 §.§ Evaluation on Self-Correction Ability



We further evaluate the self-correction capability of our constructed agent. 
We conduct two different experiment settings to validate the performance of the agent. For seen tasks, we randomly select 100 samples from correction data. For unseen tasks, we collect 100 correction data samples in a similar process to TGE.
For comparison, we also evaluate the prompting-based agent and the agent trained by BC. Since these two agents have both undergone general instruction tuning, we instruct them to conduct self-correction off the shelf.

As shown in  <Ref>, our method generates correct corrected actions far more frequently than BC and prompting-based methods in both seen tasks and unseen tasks, which demonstrates our agent's strong self-correction capability.
The powerful self-correction capability reflects our agent can truly align with the environment and generate correct corrective actions that do not violate physical constraints. 
Furthermore, we can observe from <Ref> that our agent generates correct actions at a high proportion in both seen and unseen tasks. This ensures a reliable self-correction process in speculative inference.







 §.§ Analysis on Speculative Inference





To analyze the contribution of speculative inference to overall performance, as well as to explore the quality and effectiveness of self-generated feedback, we conduct an analysis on speculative inference.

Firstly, as shown in  <Ref>, we conduct three kinds of experiment settings and test their performance on unseen tasks. Employing speculative inference significantly improves the agent's executability and affordance rate. This shows that speculative inference effectively reduces errors during execution, which demonstrates the effectiveness of the design. Moreover, LCS has not changed regardless of using speculative inference. This indicates that speculative inference contributes to the performance gain mainly by generating more executable actions, instead of recovering the expert trajectories in the training data.








Next, we provide our agent with self-generated feedback as well as three other types of feedback, and test its performance on unseen tasks. 
As shown in  <Ref>, given random feedback to the agent, the agent performs worst in both affordance rate and LCS, which underscores the importance of high-quality feedback.
When fed with self-generated feedback, the agent performs better than that of using random feedback and boolean executability signals, while slightly worse than that of using ground truth. This suggests that our method enables the agents to generate feedback with good quality.
Overall, we can observe that feedback with better qualities yields a better performance, which demonstrates that the speculative inference process faithfully relies on high-quality feedback.





 §.§ Error Analysis



We also perform an error analysis to identify the aspects where the agent constructed using our method outperforms BC. There are a total of eight types of errors, which can be further classified into grounding errors (object availability) and execution-related errors (others). The detailed demonstration can be found in the Appendix <ref>. 
As shown in Figure <ref>, we observe that all error types decreased by more than 24%, with Over occupied error showing the highest reduction rate of 94.4%. This highlights the effectiveness of our method in reducing various types of errors, highlighting its comprehensiveness.

For the two most frequent types of execution-related errors, unflipped boolean state and agent proximity, our method achieves a reduction in error count by over 37% compared to BC, thereby demonstrating its effectiveness.
Although our method primarily aims to avoid execution errors related to physical constraint and does not specifically target grounding errors such as object availability, the fact that it still reduces this type of error demonstrates the generalizability of our method.






§ RELATED WORK




  
LM-based Agent 


Nowadays, due to the increasingly powerful generalization capabilities of language models, they are often regarded as the policy function of agents to plan their behavior <cit.>. However, one issue is that there may be a misalignment between the knowledge in the environment and the internal knowledge of the model. Consequently, a significant amount of work aims to ground the language model to the environment <cit.>. Some studies harness the immense capabilities of large language models and employ intricate prompts or integrate specifically designed modules <cit.>. However, LLM-based agents would cost heavily and are not suitable for offline scenarios. Some line of work deploys language model as decision-making agents to align with embodied environments via reinforcement learning <cit.>. This type of approach tends to have low learning efficiency in embodied environments with large action spaces. In addition, similar to our approach, other research efforts have proposed frameworks where the agent first explores the environment and subsequently utilizes the exploration experience for learning <cit.>. These approaches often overly focus on the agent and lack comprehensive environmental feedback modeling, making it difficult to avoid execution errors.



  
Learning from Failure


After exploration, the agent would encounter failure in the past experience, which is assumed as negative samples. The topic of learning from negative samples has increasingly gained attention as an alternative approach to learning solely from positive samples. Traditionally, some studies aim to decrease the probability of negative samples while increasing the probability of positive samples in order to achieve better performance <cit.>. Additionally, some works construct correcting dataset and tuning language models on these data <cit.>. Besides, there are other efforts aimed at leveraging the comprehension abilities of language models to widen the gap between positive and negative samples <cit.>. In our work, we similarly leverage the inherent understanding capabilities of language models and enhance the embodied agent's learning from environmental feedback on exploration errors, as well as its ability to self-correct.






§ CONCLUSION


In this work, we aim to align the embodied agent with the environment to enhance its task-solving performance. 
Firstly, we present E^2CL, a novel framework that leverages exploration-induced errors and environmental feedback to enhance environment alignment for LM-based agents during teacher-guided and teacher-free exploration. 
Furthermore, we introduce speculative inference, a process in which the agent utilizes learned abilities for self-feedback and self-correction to reduce execution errors. Extensive experiments show that our method outperforms behavior cloning and other baseline methods.



§ LIMITATIONS


The baseline model for the robot agent constructed using our method is a text-based model, meaning the agent's observations are input in textual form. However, there is a gap between textual descriptions of real-world visual images and the actual visual information, which cannot fully encapsulate all real-world details. This discrepancy affects the robot agent's ability to ground itself in the environment. In future work, we aim to incorporate visual information directly into the input to better align with real-world scenarios. Additionally, although VirtualHome <cit.> is a relatively complex environment, we have not conducted experimental validation in other embodied environments or the real world. In the future, we will perform more experiments for validation.



§ ETHICAL CONSIDERATIONS


This work aims to construct a robot agent within Virtual Environment. 
The virtual environment setup and related data strictly follow the specifications of VirtualHome <cit.>. 
We refer to VirtualHome v2.3.0[<https://github.com/xavierpuigf/virtualhome/tree/master>] to conduct out our experiments (MIT license[<https://github.com/xavierpuigf/virtualhome/blob/master/LICENSE>]).
The models, i.e. flan-t5-small, flan-t5-base and flan-t5-large <cit.>, we use for fine-tuning are all open-source, and we will strictly follow the protocols for the academic use of these language models (Apache License 2.0[<https://huggingface.co/google/flan-t5-base>]). 
In addition, we partially use AI assistants, such as Copilot and ChatGPT, to help with our coding and writing.











§ APPENDIX





§ DATA TEMPLATE



We show the data template of planning data (D_p), feedback data (D_f) and correction data (D_c). Each piece of data adheres to the specified format.





§ ILLUSTRATION OF ERROR TYPE



During the interaction between the agent and the environment, we collect error feedback from the environment and classify it into eight categories as followings.

Unflipped Boolean State error occurs when an action meant to change the state of an object with a Boolean attribute (such as open/closed or on/off) does not achieve the intended effect, like attempting to open an already open door. Missing Object error arises when the agent is not holding the necessary object to complete an action, preventing the task's execution. Enclosed Object error involves the target object being contained within a closed structure, with the action failing to free the object for use. Invalid Action error occurs when the agent attempts to perform an action on a target object that is not afforded to it, such as trying to pull a ceiling. Over-occupied Agent error happens when the agent's hands are occupied or already interacting with objects, leaving it unable to interact with the target object in the current step. Agent Proximity errors arise when the agent is not close enough to the target object to perform the action. Object availability errors occur when the agent attempts to interact with an object that does not exist in the environment. The remaining errors are categorized under Others.





§ LENGTH ANALYSIS



Following our common sense, tasks with a greater number of steps are generally considered more challenging for the agent. To evaluate the performance of the agent on tasks of varying difficulty, we collected and analyzed the executability of tasks with different lengths of generated steps between our method and BC. As shown in  <ref>, in terms of execution rates for different lengths of generated steps, our method outperforms BC, particularly in tasks with longer steps. This indicates the widespread efficacy of our method.








§ MORE DETAILS OF VIRTUALHOME ENVIRONMENT


VirtualHome provides diverse and customizable household environments that support a wide array of possible interactions in the form of atomic action steps. There are three kinds of action template based on the action type, which are "[Action]", "[Action] <Object> <id>" and "[Action] <Object> <id> <Object> <id>".  Each [Action] refers to one of 42 atomic actions supported in Virtualhome. Full list of atomic actions are shown in  <Ref>.
In each scene, there are around 350 objects which the embodied agent can interact with and each object refers to the specific <id>. Each object has properties(e.g. drinkable, eatable) corresponding to its action affordances. Some objects also have a semantic state like heated, washed, or used. 

In ActivityPrograms Knowledge Base <cit.>, there are 292 unique high-level household tasks, with 1374 unique action plans and 6201 unique environments in total extracted from VirtualHome, and task and action plan samples manually annotated by Amazon Mechanical Turk workers. Each data sample consists of a high-level task, the description of the task and the complete action programs, which can be directly executed in the VirtualHome environment. A piece of data sample is shown in  <Ref>. 










§ FURTHER EXPERIMENT DETAILS

In our work, we primarily fine-tuned three models of different sizes: flan-t5-small with 77 million parameters, flan-t5-base with 248 million parameters, and flan-t5-large with 783 million parameters <cit.>. 

All experiments were conducted on eight NVIDIA RTX A6000 GPUs. During the pre-tuning phase, we selected 1000 samples from the expert planning data D_p and trained for one epoch. During the training process, we set the following hyperparameters: a batch size of 30, training for three epochs, and selecting the best-performing checkpoints from these epochs. The learning rate was set to 1e-4. During the inference process, all generation parameters were kept consistent with the default generation parameters of the flan-t5 series models. All experiments are expected to reproduce in one day.

 
[t!]
𝒟_p: Expert planning data, π_θ: initial robot agent policy, T_1: number of epochs in pre-tuning phase, VS: Virtualhome Simulator, M: Number of tasks, n_j: the step length of task j, M: GPT-4o, T_2: number of epochs in training phase.
Final policy π_θ

 
Randomly select few planning training data 𝒟_few⊆𝒟_p 
i=1 T_1
    Optimize θ on BC objective:     ℒ(θ) = 𝔼_(q_p,j_t) ∼𝒟_p[ -logπ_θ(a_t | (q_p, j_t-1)) ]  
 
 
j=1 M
    k=1 n_j
        Predicting the action â_k ∼π_θ(q_p,j_k-1)  
        â_k executed in VS and obtain new observation o_k, environmental execution feedback f_k 
        (â_k ≠ a_k) & â_k  is non-executable
        
             Correction data sample: (q_c, j_k-1,â_k, f_k, a_k) added to D_c  
             Feedback data sample: (q_f, j_k-1, â_k, f_k) added to D_f  
            a_k executed in VS and obtain new observation o_k  
        
         â_k == a_k
         Feedback data sample: (q_f, j_k-1, a_k, True) added to D_f
    

 
j=1 M
    the agent assumes the task is not finished
         Predicting the action â_k ∼π_θ(q_p,j_k-1)  
         â_k executed in VS and obtain new observation o_k, environmental execution feedback f_k   
         â_k is non-executable
         
            Gain corrected action a_c ∼ M(q_c, j_k-1, â_k, f_k)  
            Correction data sample: (q_c, j_k-1, â_k, f_k, a_c) added to D_c
         
         Feedback data sample: (q_f, j_k-1, â_k, f_k) added to D_f
    

 
i=1 T_2
    Optimize θ on autoregressive objective loss: ℒ_SFT (π_θ) = 𝔼_∼𝒟_p[ -logπ_θ(a_t | q_p, j_t-1) ] + 𝔼_∼𝒟_f[ -logπ_θ(f_t | q_f, j_t-1, â_t) ] + 𝔼_∼𝒟_c[ -logπ_θ(a_t | q_c, j_t-1, â_t, f_t) ] 

π_θ
Exploration-based Error Correction Learning





§ PSEUDOCODE



This section presents the pseudocode of E^2CL in Algorithm 2. A detailed discussion of the method is given in  <Ref>.


