Game On: Towards Language Models as RL Experimenters
Jingwei Zhang Thomas Lampe Abbas Abdolmaleki Jost Tobias Springenberg Martin Riedmiller

Accelerating multipartite entanglement generation in non-Hermitian superconducting qubits
    H. H. Jen
    September 9, 2024
=========================================================================================

*Equal contribution.



§ ABSTRACT 
We propose an agent architecture that automates parts of the common reinforcement learning experiment workflow, to enable automated mastery of control domains for embodied agents.
To do so, it leverages a VLM to perform some of the capabilities normally required of a human , including the monitoring and analysis of experiment progress, the proposition of new tasks based on past successes and failures of the agent, decomposing tasks into a sequence of subtasks (skills), and retrieval of the skill to execute – enabling our system to build automated curricula for learning.
We believe this is one of the first proposals for a system that leverages a VLM throughout the full experiment cycle of reinforcement learning.
We provide a first prototype of this system, and examine the feasibility of current models and techniques for the desired level of automation.
For this, we use a standard Gemini model, without additional fine-tuning,
to provide a curriculum of skills to a language-conditioned Actor-Critic algorithm,
in order to steer data collection so as to aid learning new skills.
Data collected in this way is shown to be useful for learning and iteratively improving control policies in a robotics domain.
Additional examination of the ability of the system ability to build a growing library of skills, and to judge the progress of the training of those skills, also shows promising results, suggesting that the proposed architecture provides a potential recipe for fully automated mastery of tasks and domains for embodied agents.









§ INTRODUCTION




Recent progress on leveraging large (vision) language models (VLMs/LLMs) for reinforcement learning and robotics has demonstrated their usefulness for learning robot policies <cit.> as well as for high-level reasoning <cit.>,

and they have also aided research into automated generation of reward functions for policy learning <cit.>. 

In doing so, LLMs have reduced the amount of domain-specific knowledge that an RL researcher would normally need to provide. 
Yet there are still many steps within the experiment workflow of training policies via reinforcement learning (RL) that currently require human intervention;

such as deciding when an experiment has concluded or building a curriculum of tasks <cit.> to facilitate the learning of a target task. 
While some work exists in the literature that attempts to automate some of these steps (e.g. automated training and evaluation of standard machine learning tasks <cit.> or automated curriculum building <cit.>

within the community of automated machine learning), these more automated systems usually consider the individual steps in isolation, using models specifically trained to automate a single step.

In this work, we set out to give a first sketch how a large vision language model (VLM) could be used to automate most of the missing capabilities that would be required to automate a reinforcement learning (RL) experiment.
We propose a system architecture that
uses a VLM for automating most parts of the reinforcement learning experiment loop (with the current exception of not providing the reward signal), and trains a growing set of motor skills to increasing mastery of desired domains.
This architecture integrates several of the capabilities traditionally required from a human experimenter:

    
  * The proposition of new tasks to perform/learn, given a set of already-known tasks.
    
  * The decomposition of higher-level tasks into sequences of low-level skills; paired with retrieving the actual skill the robot possesses.
    
  * Judging whether training of a set of skills has concluded, and a new round of data collection should be started for subsequent reinforcement learning.






To implement our agent, we focus on examining the suitability of currently available VLMs and prompting techniques for the intended level of automation; rather than attempting to expand their capabilities.
We therefore limit the scope of this implementation to only some of the components of the proposed system.
Notably, we do not automate the stopping of the experiment and the gradual additions of skills to the system yet, and instead present a post-hoc evaluation for both to mimic what their effect would have been.
In addition, due to the unavailability of a robust model that can automatically generate reward functions for arbitrary task, we do not automate the addition of arbitrary new skills and limit our evaluation to a domain with known rewards.

In our prototype, all of the reasoning capabilities are driven by a single, general purpose, and publicly available VLM [We use Gemini 1.5- Pro <cit.> in all experiments.], and are achieved zero-shot via prompting techniques.
Data generated under this high-level VLM's supervision is then used offline to improve a separate 'low-level' policy, which is trained specifically to output actions to control a robot, and is task-conditioned on language instructions from the high-level system.
To showcase the usefulness of our approach, we train a policy to perform multiple manipulation tasks on a simulated robot.
We show that the VLM-guided exploration produces richer data diversity, which in turn improves performance during successive iterations of policy self-improvement via fine-tuning.
In addition, we show that the same VLM can provide experiment supervision by judging the point at which an experiment should be considered to have converged.
Lastly, we illustrate that if we provide the VLM with growing sets of skills from different stages of the learning process – as judged by the VLM itself – it can produce reasonable decompositions for each stage and guide learning of progressively more complicated skills.





§ RELATED WORK






 §.§ LLM-based Virtual Agents



Following the significant improvements in performance and capabilities of LLMs,
the field of LLM-based agent has seen a surge of recent interest.
Using an LLM as a general-purpose controller, recent work has attempted to replace 
components or capabilities that used to require different pieces of software, models or human researchers by using outputs generated by prompting large language models.

Among these, there are, for example, works that propose general strategies to obtain enhanced inference from agentic LLMs by the use of chain-of-thought reasoning <cit.>,
self-consistency <cit.>,
self-reflexion <cit.>,
ReACT chains and tool use <cit.>.

More relevant to our work are the increasing number of LLM-empowered agents proposed to automate science and engineering tasks.
For example, LLM-based software engineer agents are now being designed to assist software development, leveraging the greatly improved coding capabilities of these models. This includes work that utilizes language models to enhance various aspects of software development such as assistive AI pair-programming for interactive notebooks or algorithmic reasoning within competitive programming <cit.>. Some recent work goes even further, e.g.
the SWE-Agent <cit.> explores performing end-to-end software engineering with LLM-based agents where a custom agent-computer interface is built to facilitate the agent to navigate repositories, edit code and execute programs.



On the side of automating scientific research,
LLM-based agents have been proposed to perform the work of researchers:
this includes generating novel research directions <cit.>,
reading through relevant literature to gather information <cit.>,
automate discovery of scientific knowledge <cit.>, 
come up with hypothesis and revise it based on experimental evidence <cit.>.
In the specific field of automating machine learning research,
there are studies that use LLMs to help hyper-parameter tuning of machine learning models <cit.>,
as well as work that gives the LLM-agent capabilities to interact with computer files and execute code; thus conducting machine learning experimentation in a more integrated fashion <cit.>.

In this work, using a vision language model to both monitor the progress of a machine learning experiment and to examine the resulting performance to influence later experiments is one of the aspects that we focus on.
Different to existing work, 
instead of automating experimentation in purely virtual domains,
we perform experiments with an embodied, robotic, agent in this work for automating RL research and automating domain mastery.  





 §.§ LLM/VLM-based Embodied Agents


Above we have discussed works that utilize LLMs to accomplish pure virtual tasks.
There are, however, also LLM-empowered methods that are designed to assist embodied agents.

For example, in the Minecraft domain, there is work on using  large-scale foundation VLMs to learn behavior policies within the video pre-training (VPT) line of work <cit.>.
More closely related to our work is the open-ended Voyager agent <cit.>.
In particular, in Voyager, GPT-4 <cit.> acts as the backbone,
proposing tasks for it to accomplish and writing code to help it achieve goals.
It maintains a skill library which keeps track of LLM-generated and self-verified executable code interfacing with the Minecraft environment through JavaScript API; while in our case the stored skills are learned parameterized low-level control policies rather than code.
They employ an auto-curriculum proposed by LLMs to enable the agent to perform open-ended exploration; we adopt a similar mechanism, but use it to facilitate automatic domain mastery via RL, although we limit our prototype application to one robotic domain with a restricted set of skills that we can easily evaluate. 






In the robotics domain,
CaP <cit.> is closely related to Voyager in the code-writing aspect.
It leverages LLMs to write policy code using perception and control APIs to control robots.
While its follow-up work PromptBook <cit.> provides further improvements and guidance in prompting LLMs to write code for low-level manipulation control primitives,
the high-level reasoning capability of LLMs is not highlighted in these works. 
Utilizing LLM-based reasoning for robotics tasks was pioneered by SayCan <cit.> which uses language models to decompose a given high-level task into sub-tasks,
in which the decoding for decomposition is constrained by the availability of the robot skills and weighed by the affordance of skills under a current scene.
In their work, instructions or high-level tasks are provided by human operators rather than suggested by the LLM itself, which restricts the usability of their method for more general and open-ended purposes such as exploration or automatic mastery of domains, as done in this work. Our work is complementary in that we do not restrict the suggestion and decomposition performed by the high-level LLM; but do restrict ourselves to a fixed set of executable low-level skills for which rewards can be computed.
Likewise, <cit.> also uses LLMs to decompose tasks into sub-goals, and uses those as instructions for a language-conditioned policy.
Similar to SayCan, the tasks in their work are explicitly provided by a user.
Furthermore, it requires a separate VLM to obtain text descriptions from visual observations, as well as fine-tuning of an LLM to their specific domain.
This is in contrast to our work where a native multimodal Gemini <cit.> model is used without the need for finetuning.
On the task-proposing front, there are several works that focus on simulated domains: <cit.> propose to leverage LLMs for both proposing tasks and generating simulation code for the proposed tasks, while <cit.> further sketch a system that also includes components like code generation for reward.
Since we are interested in applying our proposed system directly in the real world, these methods would need further adaptation, e.g. by adding automatic reward modeling methods that do not require access to the simulator state.

A perhaps most closely related LLM-assisted agent in the control/robotics domain to our work is AutoRT <cit.>.
They adopt an LLM-based approach to orchestrate a fleet of robots to collect diverse data,
where tasks are proposed on-the-fly by LLMs.
While the open-ended task proposition is very similar to the Voyager paradigm and our work,
there are several notable differences to our approach.
First, AutoRT is purely aimed at data collection without any active learning of policies 'in the loop' wheras we set out to specifically automate the process of automatically steering the learning process of a low level RL learner.
Secondly, since their proposed tasks are at skill-level already,
there is no decomposition component in their agent architecture and particularly no sequencing of skills.


The authors also only measure the diversity of the collected data, but do not validate the quality of the data empirically by conducting any type of policy training with it.
In contrast, we do use the data collected by our proposed system to perform self-improvement and bootstrap a more capable policy.

In order to eventually execute and evaluate arbitrary tasks, plans and subgoals proposed by an automated , one key capability would be to get a reward function from the language caption of the task. As discussed above the current work does not yet include such a step.
While we do not investigate this aspect in this work (we restrict our self-improvement experiments on training skills which the reward function is known), we do note that there is a growing body of works that utilize LLMs to write reward code for desired behavior <cit.> and VLMs as general reward models or success detectors <cit.>, which are candidates for integration into our system once reliable enough for use in reinforcement learning domains.

















§ SYSTEM ARCHITECTURE



We study the setting of an embodied agent with access to a workspace containing objects that it can interact with.
We propose a VLM-based agent architecture that can enable automatic mastery of the environment. By mastery we here mean that we expect the agent to be capable of accomplishing any meaningful task – for which we can measure success by a given set of reward functions – with any object in the environment by the end of the learning process,
and by automatic we mean that no human researcher is required to come up with a decomposition of tasks or a curriculum for learning the tasks in a specific order during the learning process and that no researcher is needed to monitor the training progress.
Our proposed agent architecture to fulfill this goal consists of  the following modules:


    
  * The curriculum module, which performs high-level reasoning to guide the learning process with auto-curriculum.
More specifically,
it is in charge of task proposition, task decomposition, skill retrieval, 

and keeps a record of past successful and failed episodes.
    
  * The embodiment module, which maintains a skill library consisting of the skills available to the embodiment. It will execute skills assigned by the curriculum in the environment, save episode data and report back success or failure. Finally it will trigger a low-level 'Actor-Critic' RL algorithm for learning (or improving) skills from the collected data.
    
  * The analysis module, which monitors the training progress of skills, reports learning status and adds converged ones to the skill library of the corresponding embodiment.







 §.§ The Curriculum Module


This module generates an auto-curriculum to guide automatic mastery of domains.
Each of its components (task proposition, task decomposition, skill retrieval) is realized by prompting the Gemini model.

In the following, we describe each component's prompts conceptually.
For concrete prompt designs used in our prototype implementation, see Appendix <ref>.



  
Task proposition.
We first prompt the VLM to propose a new high-level task in free-form text; for the agent to accomplish given a current image observation and past success and failures.
The VLM is prompted to output tasks that are novel and diverse while not being too far from the current capability of the agent.
The proposition prompt is heavily inspired by the prompt used in Voyager <cit.>, 
consisting of a description of the domain, the request for a proposal and matching reasoning leading to it, as well as a set of constraints and requirements.
This fixed prompt and a number of exemplars are followed by a current image of the domain, and a growing list of successfully and unsuccessfully completed proposals from the same experiment. We refer to the appendix for a description of the instruction and how exemplars images and success detection are included in the prompt.










  
Task decomposition.
Given a free-text high-level task proposition, the list of available skills and a current image observation, we then prompt the VLM to decompose the task into a list of sub-goals/sub-tasks, again described as fee-form text without any restriction.
The decomposition prompt contains a general instruction and several exemplars, to which we concatenate the free-form description of the proposed task from the previous task, the encoding of a current image of the domain, and the skills currently available (note that these skills are naturally limited to those we can evaluate a reward for in our current implementation, and thus are fixed text strings rather than free-form text).
We note that although the fixed-text skills are provided to the decomposition prompt, the decomposition is not instructed to output steps using those fixed-text skills only, this is intended such that it should come up with steps that are necessary,
which may or may not be available (and thus we could use the steps as a proposal for learning a new skill that should be added to our library); the effect of the availability of the returned free-text skills will be discussed further in the retrieval section.









  
Skill retrieval.
Given each decomposed free-text sub-task and the list of available fixed-text skills, we can finally retrieve the most semantically similar skill – from the available skills – to accomplish the desired sub-task.
Note that several previous works perform retrieval using the embeddings of the language instructions,
whereas we formulate it as a direct question answering (QA) task in text, which we find to be more robust.
The retrieval prompt consists of a general instruction that states the retrieval request and the constraint of not rephrasing the retrieved skill name but picking only among the available skills, followed by an exemplar, then appended with the current list of available fixed-text skills and the free-text decomposition step to map. The result of this step then is a plan of a sequence of skills from the library that can be executed by the agent.






In addition, if the retrieval of a step in the decomposition sequence fails,
this can serve as a signal that a new skill is required in the policy.
Such cases can then be included in the next round of policy learning by the embodiment module.
We note, however, that due to the lack of a system to generate rewards for arbitrary skills, in this work we manually perform the choice of skills to add; which also means that in the current implementation, the decomposition plan will be discarded if the retrieval of any of its steps failed.



 §.§ The Embodiment Module


After obtaining a decomposed list of subgoals, the curriculum will communicate the task to the corresponding embodiment to execute.
After executing a sequence of skills, it judges the success of the sequence, i.e. whether the goal that led to the decomposition, has been achieved, and reports the result back to the Curriculum module.
While determining the success of a sequence relies on pre-defined reward functions in our prototype system (see section <ref> for details),
it could, in theory, also draw upon LLM-based reward functions.

The module also collects all executed episodes in a dataset.
Once a stopping criterion has been reached – classically pre-defined as a certain number of episodes, but potentially also triggered by the Analysis module in section <ref> – a new policy learning iteration is launched with this dataset to fine-tune the previous policy; any offline policy learning algorithm could be used in this step and we refer to the next section for our specific exemplary choice.
For increased data efficiency, all of the episodes are re-labeled with the rewards of all of the skills currently known to the agent, including those newly added by the curriculum module described in section <ref>.



 §.§ The Analysis Module


Finally, this module examines the learning progress of skills by few-shot prompting.
The prompt prefix is formatted as:






Each exemplar is given in the format of:





where the reward curve plot for each exemplar is plugged into 
,  
and the exemplar reasoning and answer are placed into
 and
.





For all skills for a certain embodiment,
the analysis module will periodically go through the learning curves of each of them.
Those that are judged as converged will be added to the available skills of that embodiment (and by extension, become available to the curriculum module for decomposition) and its training will be terminated.






We note that this imposes some constraints on the reward functions that can be used: they need to be normalized, and the episode duration of evaluation runs needs to be known, in order to allow meaningful scaling of the curves for analysis.






§ SYSTEM REALIZATION



In order to explore the feasibility of the system, we implement its components, and apply them to a simulated robotic manipulation task.



 §.§ Module Interaction



























The periodically retrieves images from the environment, and includes them into the goal proposal prompt.
The goal is then decomposed into steps and skill captions are retrieved.
If any of the steps cannot be mapped to a known skill during retrieval, the plan is discarded, and the process repeated.
If all steps are retrieved, the skill sequence is sent to the , which uses them to condition a text-conitioned learned policy; we use the perceiver-actor-critic (PAC) algorithm <cit.> to learn and represent such policies.

The program flow is controlled by the : after each decomposition, all of the potentially multiple instances perform a fixed number of episode rollouts, with the skill being changed at fixed (pre-defined) intervals.
We acknowledge that this approach only applies for quasi-static domains like the object arrangement tasks considered here.
For more dynamic domains, it is necessary to also condition the model to return a duration for each skill, or to continuously query it as to whether to switch skills at a given point in time.

At the end of each rollout, the reports whether the plan was successfully executed. Success here is defined as observing a reward >0.5 for each executed skill, and >0.95 for the last skill in the sequence;
all skills in the proposed sequence must be completed to qualify as success.
The includes these success reports into its list of successful and unsuccessful plans, for use in subsequent prompts.

We use a chat-based interface between these modules, similar to that used by <cit.>.
This allows easily connecting them in a natural interface, which also facilitates human introspection and intervention during testing.
Modules simply join a Google Meet session, and interact with each other via chat messages, as well as streaming image and video data through it.
Messages can be broadcast, enabling a single high-level VLM to control the skills of multiple low-level policies at the same time, thus increasing compute efficiency in the face of otherwise expensive queries to the VLM.
The setup is illustrated in Figure <ref>.



The is used outside of the experiment loop in this prototype.
Rather than actually stopping the experiment, we run it after the experiment has concluded, so that we can evaluate whether the termination point chosen by it was indeed the point of convergence.



 §.§ Policy Training


For the low-level control policy, we employ a Perceiver-Actor-Critic (PAC) model <cit.>.
Such a model has been shown to be trainable via offline reinforcement learning, can be text conditioned, and is able to utilize non-expert (exploration) data that our agent will generate.
This in turn allows us to additionally relabel all data with multiple reward functions, and thus reuse a small amount of data to train all desired skills.

In PAC, skills can be represented by either conditioning the policy on language, on goal images, or a mixture thereof. Here, we purely opt for language, as this allows us to directly communicate the high-level system's skill proposals to the low-level policy.



 §.§ Prompting


The high-level system is represented by a standard Gemini 1.5 Pro model <cit.>.
To design the prompts for the Gemini model, we use the publicly available
OneTwo Python library <cit.>.
OneTwo is a model-agnostic layer that abstracts aspects such as injecting components into a VLM prompt template, and extracting pre-defined fields from the model's response.


Each component's prompt contains a small number of exemplars which were hand-designed and include image data from previous experiments.
This includes 2 each for for proposal and decomposition, 1 for retrieval, and 6 for analysis.
It is also worth noting that none of the proposal exemplars contain a scene with three objects, unlike in the domain we apply it to, in order to not bias the responses.
All exemplars used are provided in Appendix <ref>





§ EXPERIMENTAL RESULTS





 §.§ Benchmark


To evaluate the benefits of our approach, we consider a robotic block stacking task, previously described in <cit.>.
In this task, three colored objects in a basket need to be arranged into a desired configuration by a 7-DoF Franka Panda robot fitted with a Robotiq 2F-85 parallel gripper.
The domain is implemented in the MuJoCo simulator <cit.>.
This task was chosen since it provides combinatorial complexity, which lends itself to building up more complex skills, yet is also narrow enough to allow automatic evaluation and manual reward design.

More specifically, our expectation with this domain is for the Gemini-aided auto-curriculum to be able to lead the agent to automatically discover and learn the object configurations such as tower and pyramid, which were previously manually designed by human researchers.



 §.§ Auto-curriculum-based Exploration


To examine the ability of the system to perform task proposal and decomposition, we first train a PAC model to perform a number of simple base skills for the to utilize.
Note that the framework also allows for the agent to learn from scratch, but here as a proof of concept, we start with a base set of skills to allow for faster learning iterations.
We use a pre-existing dataset of approximately 1M episodes collected from a single-task RL experiment, where an MPO agent <cit.> was trained to perform the different permutations of stacking a single object on top of another.
The data is re-labeled with reward functions corresponding to a set of basic skills,
including opening and closing the gripper, reaching an object, lifting an object, holding one object over another, and stacking them. For a full list see Appendix <ref>.
We then train a PAC model with 140M parameters for 1.5M steps, after which performance has stabilized for all skills.

We then use this fixed policy to perform Gemini-driven data collection, following the approach described in Section <ref>.
As this data is intended for further self-improvement training, we roughly follow the CHEF approach <cit.> of performing a hyperparameter search to aid diversity.
However, we do not vary the parameters of the slow PAC training, but instead explore different settings for the .
Firstly, we vary the sampling temperature of the VLM, using both 0.0 and 0.3.
Secondly, we perform collection runs with different sets of skills made available to the agent: either all of the skills including the composite stack A on B, or only simpler ones up to hold A above B.

In each run, the controls 10 simulator instances in order to parallelize data collection.
Each skill proposal and decomposition sequence is also executed 5 times per robot to reduce querying load on the VLM.
Decomposed plans are executed open-loop, in the sense that each skill in the sequence is maintained for a fixed duration of 20 seconds before switching to the next one.
In this manner, we collect a set of 25k robot episodes in total.



  §.§.§ Data Diversity





First, we compare the dataset used for pre-training the PAC policy (which we refer to as set) with the new dataset collected by our method (set), using a distance metric similar to <cit.>.
We do this separately for camera images and proprioception data (i.e. joint angles).
For camera images, we use a CoCa image embedding <cit.>; for proprioception, we use the non-embedded observations, and normalize them first along each dimension and then overall for unit norm.
Then we measure the relative L2 distance of these representations to each other, as well as the distance of each to their respective cluster in a k-means clustering with 5 clusters (where the clusters were learned on the set).
Table <ref> highlights how data in the collected set appears to be more spread out.
The diversity in vision and proprioception data can be taken to be directly beneficial for self-improvement.



Separately, we also compare the diversity in the embeddings of the language instruction of the skills executed throughout the episodes.
We pass these through the embedding available via an older, text-only Gemini model, and contrast the diversity of the set with the combined one used for fine-tuning.
We observe an L2 distance of 0.287 and cluster distance of 0.097 for the set, vs. 0.555 L2 and 0.732 cluster for the combined set.
This matches our expectations, given that the original diversity was low (with only 6 permutation of the form "stack A on B"), the set not only executes more diverse skills, but also multiple per episode.




















When inspecting the data visually,

it also becomes apparent that it generates more complex object arrangements.
For instance, we find multiple attempts to build a tower, as well as pyramid-like structures – the latter of which result e.g. from failed tower building attempts, as the model does not propose pyramid-building itself.

It is also worth noting that the proposals generated by the model are in fact fairly focused, and mostly cover plans such as stack A on B and stack X on Y, as well as put A next to B, for only 27 unique proposals. However, the decomposition module expands these into 102 unique skill sequences.
For instance, decomposing the task of building a red-blue-green tower at different times results in two plans with the same outcome but different skill sequences:

reach blue, lift blue, above green>, open gripper, 
 reach red, lift red, above blue, open gripper


reach blue, lift blue, reach green, above green, open gripper, 
 reach red, lift red, reach blue, above blue, open gripper>


Additional examples, both successful and failed, are provided in Appendix

<ref>.



  §.§.§ Self-Improvement




In addition to quantifying the quality of the collected data, we also use it to perform a round of self-improvement of the pretrained PAC policy.
For this, we introduce three new skill into the set learned by the model that were not available in the skill library given to the for data collection: arranging the three objects into a pyramid shape, arranging them in an inverted pyramid, and arranging them to form a tower of three objects.
We manually chose these for being the same as previously used as benchmark by <cit.>;
while the curriculum module frequently suggests tower building, it does not suggest the pyramid tasks during exploration.
The rewards for these tasks are defined in Appendix <ref>.

Data is relabeled with these new rewards in addition to the existing ones.

We then fine-tune the PAC model with two datasets: once using only the original dataset, and once using the combined and sets.
In the latter case, given that the set is much smaller than the set, we up-sample it so that both datasets contribute 50% of each training batch.
We also up-sample the newly added pyramid-building skill, to in turn account for 50% of the data from each dataset.


Figure <ref> compares the performance of these datasets on a selection of skills.
As is evident from these results, not only does the added data allow the model to learn the "pyramid" skills, but it also leads to better performance on the base skills.
It is worth noting that none of the policy learns to perform tower building; this is due to the low success rate of the pretrained PAC policy when sequencing multiple skills in order to attempt stacking (since this leads to visiting states that are not represented in the original data).
Failed tower building does often lead to "accidental" creation of pyramids however, which explains the better performance on those tasks.
We therefore point out that it seems sensible to separate the proposal of new tasks to learn from the proposal of tasks used during data collection.



Finally, using the skills resulting from learning on the combined datasets we perform one additional iteration: we subsequently collect 15k more episodes with the newly trained pyramid building skills added to the skill library; and thus available to the curriculum module.
We again up-sample data so as to weight all three data sources equally.
When using this data to fine-tune the PAC policy once more, performance for these skills increases substantially, as also seen in Figure <ref>.

















 §.§ VLM-based Performance Analysis



During the initial PAC policy training, we trained the model for approximately 1.5M learner steps.
After running that long, we observe a degeneration of performance, particularly for "simpler" skills, which can be attributed to overfitting.
Normally, a human RL would employ early stopping to avoid such effects, and stop the experiment once the learning curves for all skills appear to have converged.

Here, we use Gemini to judge the convergence state of the experiment post-hoc after the training has concluded and run for an extended number of steps, in order to determine the point at which the model would have proposed early stopping.

All learning curves are scaled to a maximum reward of 400, which is known since rewards are clipped to [0; 1] and evaluation episodes do not exceed 400 steps.
The analysis model is not otherwise informed regarding the expected total reward of each skill.

Figure <ref> illustrates a selection of these judgments.
While these judgments are not fully stable, and false ones do occur, the VLM judges an increasing number of skills as converged as training progresses.
Evident errors occur mostly when judging early plateaus in the learning progress (e.g. place blue on green at 300k steps or stack red on green at 400k steps) – a limitation that would similarly affect a human practitioner if not aware of the expected final reward.
Other unstable classifications involve irregular curves such as those of lift red.
Overall, the ratings reflect both the increasing performance of the skills over time, as well as their relative difficulty, as illustrated in Figure <ref>, where easier skills can be seen to be judged as converged from early on, while harder ones only get judged as such later on average.









 §.§ Progressively Adding Skills





A second purpose of the analysis module lies in determining which skills are trained sufficiently to be used for decomposition.
In this work, we first trained the PAC policy to convergence, before starting curriculum-driven data collection.
But generally, these two processes can be performed concurrently.
In order to illustrate the curriculum module's ability to work with a growing set of skills, we therefore examine some of the plans generated when using those skills judged as converged in section <ref> at certain points in time.

We examine the proposals and decompositions at four points of the experiment: with those skills judged successful after 200k, 500k and 800k learner updates in the first PAC training experiment, as well as the entire set of skills added for self-improvement.

An overview of the model responses is provided in Table <ref>.
For more detailed outputs of the model, including the reasoning provided by the model for each response, see Appendix <ref>.

We see that after both 200k and 500k steps, the proposition yields the same simple goal.
But while after 200k steps the system has to use the most basic skills for decomposition, it employs the more reliable reach green and grasp anything skills at 500k steps.
At 800k steps, when all skills are available, it generates more complex propositions, and directly uses the higher-level stacking skills.
And with the fine-tuning skills included, the model attempts to arrange the objects into a line, which resolves into building a tower – i.e. a vertical line.





§ DISCUSSION AND FUTURE WORK



We have outlined an agent architecture for reinforcement learning that uses a VLM to perform various capabilities normally expected of a human in Section <ref>.
These capabilities would allow automating the training process of the agent beyond current capabilities, and let an embodied agent autonomously acquire an ever-growing set of skills with ever-increasing mastery.

We implemented and evaluated a first prototype of such a system in Section <ref>, including the functionalities of proposing new tasks for exploration, decomposing them into skill sequences, and analyzing the progress of the learning experiment.
For this first proof-of-concept system, we simplified several of the components and their interaction.
This was done both to limit the scope of this study, but also in order to focus on determining whether state-of-the-art methods and models are able to perform the required capabilities – particularly when used zero-shot, without costly fine-tuning of the VLM.

The prototype system showed the ability to automatically collect diverse data, which was successfully used to perform self-improvement of the control policy, and to learn new skills not learnable with a more narrow starting set (section <ref>).
The curriculum also displayed the ability to adapt the complexity of its task propositions and plans to the complexity of the available skills.

Going forward, we intend to reduce the simplifications made for the prototype system and strive for full automation, with several natural next steps outlined in the following.

In our prototype implementation, the analysis of learning progress was performed post-hoc, to illustrate the quality of analysis at different stages.
While the quality of judgments of experiment convergence was not fully reliable and suffered from the same uncertainty at plateaus that a human does, it did show potential to make correct judgments when aggregated over time.
In addition, the successful application of the system to self-improvement of the policy means that even if prematurely terminating the training of skills at plateaus, training of these skills can continue in subsequent self-improvement rounds.
In the future, we would therefore seek to integrate it directly into the automation and allow it to stop the experiment.

In the future, we also plan to include LLM-based reward functions into the architecture, once techniques are mature enough.
While currently it is too easy for RL agents to exploit false reward detections, recent advances such as Eureka <cit.> promise zero- or few-shot LLM-based rewards without the need to train specialized models.
This provides a natural next 's capability to integrate: to have the system automatically add its proposals as new skills.
It will also allow adding the one functionality of the we left out thus far: to automatically add proposed actions as new skills – which requires the PAC training to be able to label datasets with rewards matching those proposals.

Related to this, we currently simply discard decomposed sequences if any of the steps cannot be mapped to a skill known by the policy.
But encountering an unknown step provides a strong signal that there is a skill missing, and a natural next step is to include it in the next PAC training cycle as a base skill.
However, doing so would again require a universal reward module, which is not presently available.

During evaluation of the PAC policies, we observed that skills are often not sequenced correctly, e.g. after completing skill stack red on blue, the policy may not perform stack green on red successfully, even though it can perform it in isolation.
This may be either attributed to incomplete separation between skills, or as insufficient data coverage; generally, the base dataset would never have observed the terminal state of one skill as the initial state of another.
This can lead to otherwise sensible plans generated by the decomposition module to not achieve task success, which in turn causes the stack X on Y and Z on X skills to never achieve non-zero performance after the PAC finetuning.
We hypothesize that this may be rectified through repeated self-improvement, as the skill sequencing would generate more diverse data, and/or by reducing the weight of the narrowly biased set.
However, such extended data collection was not feasible in this work.

During episode unrolls, we executed the skill sequence open-loop, maintaining each skill for a fixed amount of time.
This was justified by the fact that the domain is largely static, but we do note that for more dynamic domains, it would be necessary to have the decomposition provide a duration for each skill, or determine the switching point dynamically.
The latter is infeasible for Gemini-sized models due to limited inference speed, but is expected to become possible in the future, as smaller yet equally capable models become available.











We thank Ksenia Konyushkova for suggestions on prompting, Pierre Sermanet for laying the foundations of the technical infrastructure, and Jonas Adler for general discussions of the research field.




  









§ PROMPT DESIGN



Below are the concrete prompts used in our prototype system.
These contain the static parts of the prompts and the format of exemplars.
For actual exemplars used, see Appendix <ref>.



 §.§ Task proposition

 We note that this prompt is heavily inspired by the curriculum prompt in Voyager <cit.>.





Each exemplar is given in the format of:





The evaluation content at run time will then be given in the format of:





And the model will respond in the following format:







 §.§ Task decomposition






Each exemplar is given in the format of:





The evaluation content at run time will then be given in the format of:





And the model will respond in the following format:







 §.§ Skill retrieval






Each exemplar is given in the format of:





The evaluation content at run time will then be given in the format of:





And the model will respond in the following format:










§ PROMPT EXEMPLARS



This section provides the concrete exemplars that are in the prompt templates described above when querying the model.












 §.§ Proposition


1
    [open gripper, grasp the red object]

    []

    I see one object: red. All possible spatial structures that can be built with it: move the red object to a desired 3D point. Previously, the robot has successfully grasped the red object, so to reach a different spatial structure, you can try move it to a different planar position or lift it up.

    lift the red object up


2
    [open gripper, grasp the red object, lift the red object up]

    [stack the green object on top of the red object]

    I see two objects: red, green. All possible spatial structures that can be built with them: a line where the two objects are placed next to each other; two dots where the two objects are apart from each other; a two-level tower with one object on top of another. a two-level slanted tower with one object on top of another but not aligned at the center. Previously, the robot has successfully manipulated the red object but not the green one, and it also has not built any structure with both two objects, so trying to build a two-level tower by stacking the red object on top of the green object should be both interesting and feasible to try now.

    stack the red object on top of the green object




 §.§ Task decomposition


1
    put the red object next to the blue object

    [open gripper, grasp the red object, lift the red object up, stack the red object on top of the blue object]

    All three objects are on the bottom of the basket. The robot is able to stack the red object on top of the blue object but does not know how to put the red one next to the blue one. However, the task might be accomplished by first stacking the red on blue, then lift up the red object and open the gripper, in which case the red object should drop not too far from the blue object.

    [stack the red object on top of the blue object, lift the red object up, open gripper]


2
    build a three-level tower

    [open gripper, grasp the blue object, stack the blue object on top of the green object, grasp the red object, lift the red object up, stack the red object on top of the blue object]

    The blue object is stacked on top of the green one. To build a three-level tower, I just need to stack the red object on top of the blue object, which I happen to know how to.

    [stack the red object on top of the blue object]





 §.§ Retrieval


1
    stack red on blue

    [open gripper, grasp the red object, lift the red object up, put the red object on top of the blue one]

    The skill in query asks for a object configuration where the red object is on top of the blue one. Except for the 4th one, all other skills in the library concerns with the red object only, while the 4th one not only concerns the red and the blue object but also matches the desired configuration.

   put the red object on top of the blue one




 §.§ Analysis


1
    The learning curve is still going up, the learning has not converged yet.

    NO

2
    The learning curve has reached a plateau and the performance has stablized, the training has converged.

    YES

3
    The learning curve has reached a plateau and the performance has stablized, the training has converged.

    YES

4
    The learning curve has has peaked an is degenerating again, the training had converged already.

    YES

5
    The learning curve had degenerated after an earlier peak but is going up again, the training has not converged yet.

    NO

6
    There is no full learning curve yet, the training has not converged yet.

    NO






§ RESPONSE EXAMPLES



To illustrate the type of responses generated by the model, we present a number of examples from the beginning of a self-improvement run.

In the absence of successful trials, the very first proposal is to perform a tower building task.

EX1
    [grasp anything, open gripper, close gripper, reach red, reach green, reach blue, above red, above green, above blue, lift red, lift green, lift blue, stack red on green, stack red on blue, stack green on red, stack green on blue, stack blue on red, stack blue on green]

    []

    I see three objects: red, green, blue. All possible spatial structures that can be built with them: a line where the three objects are placed next to each other; three dots where the three objects are apart from each other; a two-level tower with one object on top of another, and the third one on the side; a three-level tower with one object on top of another; a three-level slanted tower with one object on top of another but not aligned at the center. Previously, the robot has successfully built two-level towers, so trying to build a three-level tower should be both interesting and feasible to try now.

    build a three-level tower with blue on top of green on top of red


Decomposition of this task illustrates the potential impact of temperature of the model; if we sample multiple decompositions in the same situation (but with different images) with a high temperature of 1.0, the model produces plans of differing complexity.

EX2
    build a three-level tower with blue on top of red on top of green

    [grasp anything, open gripper, close gripper, reach red, reach green, reach blue, above red, above green, above blue, lift red, lift green, lift blue, stack red on green, stack red on blue, stack green on red, stack green on blue, stack blue on red, stack blue on green]

    All three objects are on the bottom of the basket. To build a three-level tower with blue on top of red on top of green, I need to first stack red on green, then stack blue on red.

    [reach green, above green, grasp anything, close gripper, lift green, reach red, above red, stack green on red, open gripper, reach blue, above blue, grasp anything, close gripper, lift blue, reach red, above red, stack blue on red, open gripper]


EX3
    build a three-level tower with blue on top of red on top of green

    [grasp anything, open gripper, close gripper, reach red, reach green, reach blue, above red, above green, above blue, lift red, lift green, lift blue, stack red on green, stack red on blue, stack green on red, stack green on blue, stack blue on red, stack blue on green]

    All three objects are on the bottom of the basket. To build a three-level tower with blue on top of red on top of green, I need to first stack red on green, then stack blue on red.

    [stack red on green, stack blue on red]


The same task is also proposed later, after collecting several trials, with the reasoning that it failed several times and should thus be practiced. Note that if the same trial is attempted multiple times, it is combined and denoted with a success count () for brevity; in the actual prompt, these are expanded into a list.

EX4
    [grasp anything, open gripper, close gripper, reach red, reach green, reach blue, above red, above green, above blue, lift red, lift green, lift blue, stack red on green, stack red on blue, stack green on red, stack green on blue, stack blue on red, stack blue on green, build a three-level tower with blue on top of green on top of red (x2), build a three-level tower with blue on top of red on top of green (x2)]

    [build a three-level tower with blue on top of green on top of red (x2), build a three-level tower with blue on top of red on top of green (x5)]

    I see three objects: red, green, blue. All possible spatial structures that can be built with them: a line where the three objects are placed next to each other; three dots where the three objects are apart from each other; a two-level tower with one object on top of another, and the third one on the side; a three-level tower with one object on top of another; a three-level slanted tower with one object on top of another but not aligned at the center. The robot has successfully built three-level towers but failed many times, so it needs more practice.

    build a three-level tower with blue on top of red on top of green


A common failure case involves the model incorrectly injecting a description of the scene into the answer field of the response. This leads to a failure in the OneTwo library to parse the response even when it contains a task proposal, which in turn will cause the plan to be rejected.

EX5
    build a two-level slanted tower with blue on top of red

    [grasp anything, open gripper, close gripper, reach red, reach green, reach blue, above red, above green, above blue, lift red, lift green, lift blue, stack red on green, stack red on blue, stack green on red, stack green on blue, stack blue on red, stack blue on green]

    All three objects are on the bottom of the basket. The robot can stack blue on red to build a two-level tower, but it might not be slanted. To make it slanted, the robot can first lift the red object up, then move above blue and drop it.

    All three objects are on the bottom of the basket






§ PAC SKILLS AND REWARD FUNCTIONS



The reward functions used to train the PAC policy generally follow those used in <cit.>.
However, we use lower-level skills: instead of the strongly shaped and staged "stack and leave" reward they use, we only use its atomic components, as we would hope for our system to "discover" the heavily engineered, human-provided composite function in prior work. Thus our skill library consists of the following (where ,  and  are placeholders for all possible permutations of red, blue and green):

    
  : Shaped; 0 if the gripper is closed, 1 if is maximally opened.
    
  : Shaped;  Inverse of open_gripper.
    
  : Binary; 1 if the gripper's grasp sensor is triggered, 0 otherwise.
    
  : Shaped; tangentially decaying distance between the robot's TCP and the center of object .
    
  : Shaped; tangentially decaying distance between the robot's TCP and a point 10cm above the center of .
    
  : Shaped; 0 if the center of  is less than 5cm above the workspace surface, 1 if more than 10cm above, linearly interpolated between those limits.
    
  : Shaped; tangentially decaying distance between the center of   and a point 4cm above .
    
  : Shaped; place_X_Y, but set to 0 if grasp_anything is non-zero.

During the self-improvement experiments, we add three more skills, composed of the above:

    
  : Product of stack_X_Y and stack_Y_Z.
    
  : Product of stack_X_Y and stack_X_Z.
    
  : Product of stack_X_Y and stack_Z_Y.






§ PROGRESSIVE SKILL ADDITION


















Below are the detailed model responses when examining proposal and decomposition at different steps of the pre-training process in section <ref>.
We note that since this is a post-hoc analysis, as the system had all skills available during the data collection experiments we performed.
Thus we can not collect  and  for proposition. Therefore for all the following rounds,
we use all skills available at the corresponding training step in the skill library as the  and use an empty list as the  for proposition.




 §.§ 200k Learner Steps



Skill library:




    [open gripper, close gripper, above red, above green, above blue, reach red]



    I see three objects: red, green, blue. All possible spatial structures that can be built with them: a line where the three objects are placed next to each other; three dots where the three objects are apart from each other; a two-level tower with one object on top of another; a three-level tower with one object on top of another; a two-level slanted tower with one object on top of another but not aligned at the center; a three-level slanted tower with one object on top of another but not aligned at the center. Previously, the robot has successfully reached the red object but has not grasped any object yet, so trying to grasp an object should be both interesting and feasible to try now.

    grasp the green object




    grasp the green object

    [open gripper, close gripper, above red, above green, above blue, reach red]



    The robot does not know how to grasp the green object directly. However, it can move its gripper above the green object and then close the gripper, which should result in grasping the green object.

    [above green, close gripper]




    above green 

    [open gripper, close gripper, above red, above green, above blue, reach red]



    The query concerns with a configuration where something is above the green object. All skills except above green are irrelevant.

    above green



    close gripper 

    [open gripper, close gripper, above red, above green, above blue, reach red]



    The query skill concerns with the gripper only. The second entry in the library is the only one that concerns with the gripper.

    close gripper





 §.§ 500k Learner Steps


Skill library (added on top of 200k steps):





    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red]



    I see three objects: red, green, blue. All possible spatial structures that can be built with them: a line where the three objects are placed next to each other; three dots where the three objects are apart from each other; a two-level tower with one object on top of another, and the third one on the side; a three-level tower with one object on top of another; a three-level slanted tower with one object on top of another but not aligned at the center. Previously, the robot has successfully manipulated the red object but not the green and blue ones, so trying to grasp the green object should be both interesting and feasible to try now.

    grasp the green object




    grasp the green object

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red]



    The robot can reach above the green object, reach the green object, and grasp anything. Therefore, the robot can grasp the green object.

    [The robot can reach above the green object, reach the green object, and grasp anything]



    The robot can reach above the green object

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red]



    The query concerns the robots capability of reaching a location above the green object. All skills in the library except above green are about robot actions, while above green describes a location relative to the green object.

    above green




    reach the green object

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red]



    The query asks for reaching the green object. All skills except reach green are either irrelevant or describe a different object.

    reach green




    and grasp anything

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red]



    The query skill does not induce any specific object configuration but asks for a grasping action. The only skill in the library that involves grasping is grasp anything.

    grasp anything




 §.§ 800k Learner Steps


Skill library (added on top of 500k steps):





    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green]



    I see three objects: red, green, blue. All possible spatial structures that can be built with them: a line where the three objects are placed next to each other; three dots where the three objects are apart from each other; a two-level tower with one object on top of another, and the third object next to the tower; a three-level tower with one object on top of another; a three-level slanted tower with one object on top of another but not aligned at the center. The robot has successfully built all two-level towers, so trying to build a three-level tower should be both interesting and feasible to try now.

    build a three-level tower with blue on top of red on top of green




    build a three-level tower with blue on top of red on top of green

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green]



    All three objects are on the bottom of the basket. To build a three-level tower with blue on top of red on top of green, I need to stack red on green first, then stack blue on red.

    [stack red on green, stack blue on red]




    stack red on green

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green]



    The query asks for a specific object configuration where the red object is on top of the green one. All skills except those related to stack only involve individual object manipulation. The desired configuration is specifically achieved by stack red on green.

    stack red on green




    stack blue on red

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green]



    The query asks for a specific object configuration - blue on top of red. All other skills either manipulate a single object or create different configurations. Only stack blue on red matches the desired outcome.

    stack blue on red




 §.§ After Self-improvement



Skill library (added on top of 800k steps):





    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green, stack green on blue and red on green, stack blue on green and red on blue, stack red on green and blue on red, stack green on red and blue on green, stack blue on red and green on blue, stack red on blue and green on red, build a pyramid with red on top and green and blue at the bottom, build a pyramid with red on top and blue and green at the bottom, build a pyramid with green on top and blue and red at the bottom, build a pyramid with green on top and red and blue at the bottom, build a pyramid with blue on top and red and green at the bottom, build a pyramid with blue on top and green and red at the bottom, build an inverted pyramid with green and blue at the top and red at the bottom, build an inverted pyramid with blue and green at the top and red at the bottom, build an inverted pyramid with blue and red at the top and green at the bottom, build an inverted pyramid with red and blue at the top and green at the bottom, build an inverted pyramid with red and green at the top and blue at the bottom, build an inverted pyramid with green and red at the top and blue at the bottom]



    I see three objects: red, green, blue. All possible spatial structures that can be built with them: a line where the three objects are placed next to each other; three dots where the three objects are apart from each other; a two-level tower with two objects on top of one another; a three-level tower with three objects on top of one another; a two-level slanted tower with two objects on top of one another but not aligned at the center; a three-level slanted tower with three objects on top of one another but not aligned at the center. The robot has already built all possible two-level and three-level towers, including slanted ones. The robot has not yet built a line with all three objects.

    build a line with the red object next to the green object and the blue object next to the green object




    build a line with the red object next to the green object and the blue object next to the green object

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green, stack green on blue and red on green, stack blue on green and red on blue, stack red on green and blue on red, stack green on red and blue on green, stack blue on red and green on blue, stack red on blue and green on red, build a pyramid with red on top and green and blue at the bottom, build a pyramid with red on top and blue and green at the bottom, build a pyramid with green on top and blue and red at the bottom, build a pyramid with green on top and red and blue at the bottom, build a pyramid with blue on top and red and green at the bottom, build a pyramid with blue on top and green and red at the bottom, build an inverted pyramid with green and blue at the top and red at the bottom, build an inverted pyramid with blue and green at the top and red at the bottom, build an inverted pyramid with blue and red at the top and green at the bottom, build an inverted pyramid with red and blue at the top and green at the bottom, build an inverted pyramid with red and green at the top and blue at the bottom, build an inverted pyramid with green and red at the top and blue at the bottom]



    The robot can stack green on blue and red on green in one go, which will result in the desired configuration.

    [stack green on blue and red on green]



    stack green on blue and red on green

    [open gripper, close gripper, above red, above green, above blue, reach red, reach green, reach blue, grasp anything, lift red, hold red over green, hold red over blue, hold green over blue, hold green over red, hold blue over red, hold blue over green, stack red on green, stack red on blue, stack green on blue, stack green on red, stack blue on red, stack blue on green, stack green on blue and red on green, stack blue on green and red on blue, stack red on green and blue on red, stack green on red and blue on green, stack blue on red and green on blue, stack red on blue and green on red, build a pyramid with red on top and green and blue at the bottom, build a pyramid with red on top and blue and green at the bottom, build a pyramid with green on top and blue and red at the bottom, build a pyramid with green on top and red and blue at the bottom, build a pyramid with blue on top and red and green at the bottom, build a pyramid with blue on top and green and red at the bottom, build an inverted pyramid with green and blue at the top and red at the bottom, build an inverted pyramid with blue and green at the top and red at the bottom, build an inverted pyramid with blue and red at the top and green at the bottom, build an inverted pyramid with red and blue at the top and green at the bottom, build an inverted pyramid with red and green at the top and blue at the bottom, build an inverted pyramid with green and red at the top and blue at the bottom]



    The query asks for a three-object configuration with green on blue and red on green. This configuration is exactly what the 22nd skill in the library describes.

    stack green on blue and red on green


