# Long-Horizon Planning for Multi-Agent Robots in Partially Observable
Environments

Siddharth Nayak1 Adelmo Morrison Orozco1 Marina Ten Have1 Vittal Thirumalai1
Jackson Zhang1 Darren Chen1 Aditya Kapoor2 Eric Robinson3 Karthik
Gopalakrishnan4
James Harrison5 Brian Ichter 5 Anuj Mahajan 6 Hamsa Balakrishnan1
1MIT 2TCS 3USAF-MIT AI Accelerator
4Stanford 5Google DeepMind 6Apple
Now at Physical Intelligence.Work done outside Apple.

###### Abstract

The ability of Language Models (LMs) to understand natural language makes them
a powerful tool for parsing human instructions into task plans for autonomous
robots. Unlike traditional planning methods that rely on domain-specific
knowledge and handcrafted rules, LMs generalize from diverse data and adapt to
various tasks with minimal tuning, acting as a compressed knowledge base.
However, LMs in their standard form face challenges with long-horizon tasks,
particularly in partially observable multi-agent settings. We propose an LM-
based Long-Horizon Planner for Multi-Agent Robotics (LLaMAR), a cognitive
architecture for planning that achieves state-of-the-art results in long-
horizon tasks within partially observable environments. LLaMAR employs a plan-
act-correct-verify framework, allowing self-correction from action execution
feedback without relying on oracles or simulators. Additionally, we present
MAP-THOR, a comprehensive test suite encompassing household tasks of varying
complexity within the AI2-THOR environment. Experiments show that LLaMAR
achieves a 30% higher success rate compared to other state-of-the-art LM-based
multi-agent planners.

##  1 Introduction

Creating embodied agents that assist humans in real-life scenarios is a
significant challenge when humans communicate their intentions in natural
language. Certain tasks like moving furniture (Jain et al., 2020b, 2019a),
search-and-rescue (Kumar et al., 2004), environmental monitoring (Dunbabin and
Marques, 2012), etc,. require coordination among multiple agents to solve the
tasks efficiently as compared to single-agent scenarios. This challenge of
understanding these natural language inputs and effectively coordinating these
agents to solve the task is exacerbated in the multi-agent scenario. Recent
works (Ichter et al., 2023; Huang et al., 2023b, c; Singh et al., 2023; Liang
et al., 2022; Lin et al., 2023; Shah et al., 2022; Huang et al., 2023a, 2022)
have shown that Language Models (LMs) 111We denote Large Language Models as
LLMs, Vision Language Models as VLMs can effectively use language instructions
to develop plans for robots. However, most studies focus on single-agent long-
horizon task planning. Naïve extensions of single-agent planning algorithms to
multi-agent settings often fail due to non-stationarity in the environment,
where the policies of other agents—modeled as a part of the environment—are
continuously changing (Tan, 1993; Tampuu et al., 2015). Such failures lead to
suboptimal performance as agents struggle to anticipate and adapt to the
actions of others. We therefore formulate a centralized process in which
decisions are made simultaneously for all all agents based on their (partial)
observations, similar to the centralized multi-agent system framework (CMAS)
proposed in (Chen et al., 2023). Leveraging the ability of pre-trained LMs to
generalize across diverse tasks, we aim to use LMs for long-horizon embodied
multi-agent task planning.

The key insight of our work is that integrating a _plan-act-correct-verify_
framework with LMs enables a robust and adaptive approach to multi-agent task
planning in dynamic, partially observable environments that allows agents to:
(1) plan subtasks required to complete the task, (2) select high-level actions
for every agent that can complete the proposed subtasks, (3) identify and
correct failures after high-level action execution, and (4) self-verify
subtask completion based on high-level action execution. Unlike existing
methods, our approach uses real-time execution feedback, observations, and
agent histories to iteratively refine action planning and execution. This
allows agents to adjust strategies based on reasoned insights on action
execution, effectively addressing failures without relying on perfect
environmental knowledge or oracle feedback. The correction and verification
process in our cognitive architecture (Arora and Kambhampati, 2023) is
grounded in the environment’s reality, which sets it apart from LM self-
verification methods that lack such grounding (Valmeekam et al., 2023a). This
framework enhances agents’ ability to complete complex, long-horizon tasks,
yielding substantial improvement over current state-of-the-art methods.

Similar to our approach, recent works (Kannan et al., 2023; Wang et al., 2024;
Singh et al., 2024; Zhang et al., 2024; Yu et al., 2023; Mandi et al., 2023;
Chen et al., 2023) utilize LMs for multi-agent planning, often adopting a
hierarchical decision-making structure. The LMs are used for high-level
planning to determine subtasks, sometimes in conjunction with planning domain
definition language (PDDL) that together with the LM planner, functions as a
feasibility solver. Specific actions are executed using low-level policies
pre-trained through reinforcement learning, behavior cloning, or heuristic
approaches. While these methods effectively use LMs as high-level planners,
they assume perfect low-level primitive action policies and simulator or
oracle-provided environmental information. By contrast, LLaMAR does not assume
perfect knowledge of the environment, does not rely on oracle feedback, and
does not assume perfect execution of low-level primitive policies. This
approach moves us closer to enabling real-world robots that operate
independently of privileged knowledge.

To avoid ambiguity, we use the following conventions. We refer to the
objectives within the environments as "tasks" or "goals" and "subtasks" to
describe the breakdown of tasks or goals. "High-level actions" are defined as
skills the agent can perform, while "low-level actions" or "primitive actions"
refer to existing policies—either learned or predefined using heuristics—that
execute a sequence of actions to accomplish a high-level action. More details
and examples can be found in Appendix A.

The main contributions of this paper are:

  * •

LLaMAR: An LM-based Long-Horizon Planner for Multi-Agent Robotics, designed
for iterative planning of long-horizon, multi-objective tasks in partially
observable environments, with the following key features:

    * –

It operates without prior knowledge of the environment, allowing agents to
explore and make decisions based on new observations.

    * –

It evaluates outcomes through direct observation of images, rather than
relying on oracles for feedback, enabling independent identification and
correction of action failures.

  * •

MAP-THOR (Multi-Agent Planning in THOR): a benchmark suite of tasks within the
AI2-THOR simulator under partial observability to standardize methodologies
and metrics for evaluating multi-agent planning effectiveness and robustness.

##  2 Related Work

Reinforcement Learning (RL) for Long-Horizon Planning: While RL algorithms
have shown promise in many applications, they still struggle with long-horizon
tasks. Hierarchical reinforcement learning (HRL) has been used to address
these challenges in both single-agent (Barto and Mahadevan, 2003; Nachum et
al., 2018; Pateria et al., 2021; Wohlke et al., 2021) and multi-agent settings
(Tang et al., 2019; Yang et al., 2019; Meyer et al., 2020). However, these
approaches are typically applied to single-task, stationary environments, such
as games, where agents solve for one goal in a fixed environment.
Consequently, these methods do not generalize well across multiple
environments or tasks. Multi-task RL has been explored as a potential
solution, requiring sophisticated task planning to handle diverse objectives
(Stooke et al., 2021; Yang et al., 2020). This often involves decomposing
tasks into manageable subtasks, a process well-suited for hierarchical
frameworks. However, subtasks are known apriori in multi-task RL formulations.
Real-world long-horizon RL necessitates robust task planning, and LMs have
emerged as a promising approach for this purpose.

LMs for Embodied Single-Agent Planning: Recent studies have demonstrated the
effectiveness of LMs in generating and executing plans in embodied single-
agent environments (Yang et al., 2023; Wang et al., 2023; Xi et al., 2023;
Sumers et al., 2023; Sharma et al., 2021; Raman et al., 2022; Gramopadhye and
Szafir, 2023) and creating plans in single-agent embodied robotic environments
(Kolve et al., 2017; Savva et al., 2019; Xia et al., 2018; Li et al., 2023;
Padmakumar et al., 2021; Shridhar et al., 2019; Misra et al., 2018; Zhu et
al., 2017; Brodeur et al., 2017; Xiang et al., 2020; Jain et al., 2019b,
2020a). Works like SayCan (Ichter et al., 2023) and Grounded Decoding (Huang
et al., 2023b) use a combination of value functions and LLM predictions for
long-horizon tasks. ProgPrompt (Singh et al., 2023) and Zero-Shot Language
Planner (Huang et al., 2022) generate static plans executed in the
environment, which may fail in partially observable and dynamic settings. To
mitigate this, LLM-planner (Song et al., 2023) updates plans based on new
observations, similar to our approach.

LMs for Multi-Agent Planning: (Xu et al., 2023) use LLMs in multi-agent games,
while CoNavGPT (Yu et al., 2023) creates global plans for two robots in an
embodied environment. RoCo (Mandi et al., 2023) and CoELA (Zhang et al., 2024)
assign separate LMs to each agent for decentralized action prediction,
allowing natural language communication between agents. However, RoCo and
CoNavGPT require detailed environment information for planning, and CoELA’s
action space is filtered by an oracle. Relying on privileged information from
an oracle is impractical in real-world applications. By contrast, our work
focuses on free-form action generation and handles tasks with more ambiguous
descriptions. Prior work (Chen et al., 2023) compare centralized (CMAS) and
decentralized (DMAS) planning frameworks, showing that centralized planners
perform better, though their experiments are in simple, known environments
with limited number of agents. Two-Step (Singh et al., 2024) decomposes goals
for main and helper agents, using PDDL planners for high-level actions.
SmartLLM (Kannan et al., 2023) uses multiple LLM modules for subtask
decomposition, multi-robot group formation and task allocation but assumes
robots have complete knowledge of the environment, making plans prone to
errors in unknown settings. (Wang et al., 2024) use LLMs with conformal
prediction for safe multi-agent planning, but the action choices are limited
to a small set of objects. Table 1 presents a comparison of the
characteristics of different LM-based approaches to multi-agent planning with
our work.

LMs can interpret high-level instructions and break them down into feasible
subtasks, making them ideal for long-horizon, multi-task scenarios. Our work
leverages LMs to enable long-horizon planning across a variety of tasks and
environments, building on these advances to address the limitations of
traditional RL and HRL methods. By integrating LMs into our planning
framework, we enhance the ability to generalize across diverse tasks and
scenarios, making significant strides toward practical, real-world
applications of RL in dynamic, multi-agent settings.

Method | Dynamic | Local | Failure | Self
---|---|---|---|---
Planning | Information | Correction | Verification
Two-Step (Singh et al., 2024) | ✗ | ✗ | ✗ | ✗
Smart LLM (Kannan et al., 2023) | ✗ | ✗ | ✗ | ✗
Conformal Prediction LLM (Wang et al., 2024) | ✓ | ✗ | ✗ | ✗
CoELA (Zhang et al., 2024) | ✓ | ✓ | ✗ | ✗
LLaMAR (this paper) | ✓ | ✓ | ✓ | ✓
Table 1: The proposed model, LLaMAR: 1) performs dynamic planning, avoiding
the open-loop plan-and-execute paradigm; 2) operates without privileged
simulator information (e.g., access to all objects in the environment); 3) re-
plans when low-level actions fail, not assuming perfect execution; and 4)
self-verifies subtask completion without relying on the simulator.

##  3 Background

Problem Setting: We consider a setting where multiple robots perform a series
of everyday tasks in a home-like environment, such as cleaning a room or
putting groceries in the fridge. These tasks typically require long-horizon
planning, involving around 100 low-level actions to reach the goal. Our
objective is to compute plans for a team of robots to execute high-level
language instructions, $I$. We formalize these tasks as partially observable
Markov decision processes (POMDP) (Puterman, 1994; Kaelbling et al., 1998),
denoted as $\langle
N,\mathcal{I},\mathcal{S},\\{\mathcal{O}_{i}\\},\\{\mathcal{A}_{i}\\},\mathcal{P},\mathcal{G},T\rangle$.
$N$ is the number of agents and $\mathcal{I}$ is the high-level language
instruction set. Here, $s\in\mathcal{S}$ represents the joint state of all
agents, and $o\in\mathcal{O}$ denotes the observation set for all agents.
Particularly, $o_{i}\in\mathcal{O}_{i}$ is the observation set of agent $i$,
that captures incomplete environment state information.
$a\in\mathcal{A}=\mathcal{A}_{1}\times\mathcal{A}_{2}\cdots\mathcal{A}_{N}$
represents the joint action space. The joint action space comprises of
different categories of high-level actions
$\mathcal{A}=\mathcal{A}_{NAV}\cup\mathcal{A}_{INT}\cup\mathcal{A}_{EXP}$,
where $\mathcal{A}_{NAV}$ is the joint navigation action set,
$\mathcal{A}_{INT}$ is the joint interaction actions which allow the agents to
interact with objects, and $\mathcal{A}_{EXP}$ are the joint exploration
actions which allow the agents to explore the environment. Examples of the
high-level actions include PickUp(object) $\in\mathcal{A}_{INT}$ and
NavigateTo(location) $\in\mathcal{A}_{NAV}$. Each high-level action is
associated with a low-level primitive action (pre-trained RL, behavior cloned,
or heuristic-based policy). These actions are executed synchronously by all
agents at every high-level decision step. $\mathcal{P}(s^{\prime}|s,a)$ is the
joint transition probability function that defines the probability of arriving
at $s^{\prime}\in\mathcal{S}$ after taking joint action $a\in\mathcal{A}$ in
$s\in\mathcal{S}$. $\mathcal{G}=\\{g_{1},\cdots,g_{k}\\}$ defines the subtasks
that the agents need to perform to accomplish the language instruction task.
$T$ is the length of the planning horizon.

Environment: To simulate open-ended, long-horizon tasks that resemble everyday
activities, we use the AI2Thor simulator (Kolve et al., 2017), which supports
a diverse set of interactions and photorealistic rendering. Since our approach
does not require any parametric training, it can potentially translate to
other similar embodied environments like VirtualHome (Puig et al., 2018),
Habitat (Savva et al., 2019; Szot et al., 2021; Puig et al., 2023), and
ThreeDWorld (Gan et al., 2021), possibly extending beyond household domains.
We instantiate the problem with $N$ agents cooperating to accomplish a long-
horizon rearrangement task (Batra et al., 2020) in an indoor environment. The
agents do not know the objects present in the environment a priori and are
encouraged to explore the environment to gather more information to complete
the task. We list the complete set of actions in the Appendix B. Unlike
previous solutions in similar settings (Zhang et al., 2024; Kannan et al.,
2023; Wang et al., 2024), we do not rely on an oracle/simulator to verify
subtask completion. In prior works, a predefined conditional satisfiability
check is used as subtask completion feedback.

##  4 Approach

Figure 1: An overview of LLaMAR’s modular cognitive architecture. LLaMAR
leverages LMs within four key modules: Planner, Actor, Corrector, and
Verifier, each with specific roles. The Planner breaks down the high-level
language instruction into feasible subtasks to achieve the environment goal.
The Actor determines the high-level actions each agent should perform. These
actions trigger low-level policies that generate and execute a sequence of
primitive actions in sync across all agents. Based on execution feedback, the
Corrector suggests corrections for high-level actions and the Verifier Module
validates completion of subtasks.

We describe our approach in this section. Figure 1 illustrates LLaMAR’s
architecture comprising four modules: _Planner_ , _Actor_ , _Corrector_ , and
_Verifier_ , each an LM with a distinct role. Prior work (Prasad et al., 2023)
shows that splitting roles across different LMs improves performance in
sequential decision-making. Our initial experiments confirm that LMs tasked
with reasoning about multiple inputs and providing long outputs perform
poorly. We iterate through these four modules at every high-level decision
step. The pseudocode for our approach is in Appendix 2. We define some key
notation below:

  * •

Memory $\mathcal{M}$: A textual description of the joint memory of all agents,
summarizing past observations, high-level actions, plausible reasons for
action failures, and specific subtasks that each agent is attempting to solve.

  * •

Open Subtasks $\mathcal{G}_{O}\subset\mathcal{G}$: Feasible subtasks proposed
by the _Planner_ LM to achieve the environment task that are yet to be
accomplished by the agents.

  * •

Completed Subtasks $\mathcal{G}_{S}\subset\mathcal{G}$: Subtasks completed by
the agents.

  * •

Corrective Actions $a_{c}$: Corrective actions for each agent based on failure
information from the previous step.

At the start of each episode, Memory $\mathcal{M}$, Open Subtasks
$\mathcal{G}_{O}$, Completed Subtasks $\mathcal{G}_{S}$, Actions $a$,
Corrective Actions $a_{c}$, and Failure Information $\mathcal{F}$ are
initialized as empty sets.

Consider an example of a kitchen with groceries, a fridge, and a counter. Two
agents are tasked with "Fetch the groceries and place them in the fridge".
This example will help illustrate the utility of each module. All LMs receive
a language task instruction $\mathcal{I}$, joint observations from all agents,
and information about open and completed subtasks and memory unless stated
otherwise. We next discuss the various components in our architecture in
detail:

Planner Module The _Planner_ LM module suggests feasible subtasks to ensure
the completion of the environment task. This method, similar to SmartLLM’s
(Kannan et al., 2023) zero-shot planning, uses only observations in the
agents’ field of view. The _Planner_ suggests subtasks related to objects seen
in the current observation or memory of all the agents. For the example
considered, it decomposes the task into subtasks like "transport the tomato to
the fridge" and "transport the lettuce to the fridge", which are added to
$\mathcal{G}_{O}$.

Actor Module The _Actor_ LM additionally uses corrective actions suggested by
the _Corrector_ module in the previous time step to predict high-level actions
for the current step. These actions are then executed in the environment to
progress a subset of subtasks in the open subtask set $\mathcal{G}_{O}$ and
accordingly updates the joint memory. For instance, the _Actor_ module might
suggest actions such as
$a=[\texttt{Pickup(Tomato)},\texttt{NavigateTo(Lettuce)}]$, updating memory
with "We saw a tomato on the counter-top, Alice is picking up the tomato, and
Bob is navigating to the lettuce".

Corrector Module The _Corrector_ LM self-corrects high-level actions suggested
by the _Actor_ LM after failures in the previous step’s execution222We use the
simulator just to verify the successful execution of the high-level action..
It suggests corrective high-level actions and provides reasons for failures
and chosen corrections. For example, it might suggest "The action of picking
up the tomato failed because it is too far away. Alice first needs to navigate
closer to the tomato."; $a_{c}=[\tt{NavigateTo(Tomato)},\tt{None}]$.

Verifier Module After executing high-level actions, the _Verifier_ LM assesses
whether these actions have completed any subtasks in the open subtask set.
Successful subtasks are moved to the completed subtask set. Without the
_Verifier_ LM, the method would need to rely on the simulator/oracle for
success or failure information. The _Verifier_ LM along with other information
uses the successfully executed high-level actions proposed by the _Actor_ LM
to predict subtask completion. For example, after transporting the lettuce to
the fridge, the _Verifier_ updates the completed subtasks with "transport
lettuce to the fridge".

Admissible Action parsing with Semantic Translation When LMs generate action
plans, natural language outputs often fail to translate to executable high-
level actions. This happens when the output does not match the predefined
format or refers to unrecognized contextually similar objects. We use a cosine
similarity method from (Huang et al., 2022), fine-tuning a pre-trained
sentence-BERT (Reimers and Gurevych, 2019) to transform the free-form text
into admissible high-level actions. Hyperparameters and additional details of
the sentence transformer fine-tuning are provided in Appendix G.

Exploration Strategy In unexplored environments, agents need to search for
task-relevant objects. If agents cannot find the required objects, the
language model can choose an ‘exploration’ action $a_{exp}$. We use a
semantically-guided heuristic to determine the choice of region to be
explored. The agent rotates to four cardinal directions
$d\in{North,South,East,West}$, capturing image observations $o_{n,d}$. These
images are processed through a pre-trained CLIP image encoder (Radford et al.,
2021) to obtain embeddings $I_{d}$. The list of open subtasks
$\mathcal{G}_{O}$ is processed through the corresponding CLIP text encoder to
get text embeddings $g_{O,i}$. The exploration score $\mathcal{E}_{d}$ in
direction $d$ is defined as
$\mathcal{E}_{d}=\sum_{i=1}^{|\mathcal{G}_{O}|}\frac{g_{O,i}\cdot
I_{d}}{\|g_{O,i}\|\|I_{d}\|}$. The direction with the highest score
$d^{*}=\arg\max_{d}\mathcal{E}_{d}$ is chosen. Summing the scores helps select
the best direction to explore in expectation. The agent rotates towards
$d^{*}$ and moves $J=2$ steps, repeating this process $K=3$ times in one
_explore_ action. This approach ensures that images relevant to identifying
potential subtasks are prioritized. For example, if $\mathcal{G}_{O}$ includes
"locate a computer", it is more likely to find a computer on a table than on a
sofa, resulting in a higher cosine similarity score between the subtask CLIP
text embedding and table CLIP image embedding. Refer to Appendix B.2 for more
details about the exploration heuristic.

##  5 Experiments

MAP-THOR: To evaluate the performance of LLaMAR and benchmark other baseline
methods, we create a benchmark dataset of tasks which we call MAP-THOR (Multi-
Agent Planning tasks in AI2-THOR). While Smart-LLM (Kannan et al., 2023)
introduces a dataset of 36 tasks within AI2-Thor (Kolve et al., 2017)
classified by complexity, their tasks are limited to single floor plans. This
limitation hinders testing the robustness of planners across different room
layouts. Additionally, some tasks in their dataset cannot be performed by
multiple agents, regardless of task division, such as Pick up the pillow, Open
the laptop to turn it on, and Turn off the lamp.

By contrast, MAP-THOR includes tasks solvable by both single and multiple
agents. We classify the tasks into four categories based on the ambiguity of
the language instructions. To test the planner robustness, we provide five
different floor plans for each task. We also include automatic checker modules
to verify subtask completion and evaluate plan quality. Our dataset comprises
45 tasks, each defined for five distinct floor plans, ensuring comprehensive
testing and evaluation.

We conduct experiments with tasks of varying difficulty levels, where an
increase in difficulty of the tasks corresponds to an increased ambiguity in
the language instructions. The complete task list of each category can be
found in the Appendix C.

  * •

Explicit item type, quantity, and target location: Agents are explicitly
instructed to transport specific items to specific target locations. For
example, put bread, lettuce, and a tomato in the fridge clearly defines the
objects (tomato, lettuce, bread) and the target (fridge).

  * •

Explicit item type and target location but implicit item quantity: The object
type is explicitly described, but its quantity is not disclosed. For example,
Put all the apples in the fridge. Agents must explore the environment to
locate all specified items and also predict when to stop.

  * •

Explicit target location but implicit item types and quantity: The target
location is explicitly defined but the item types and their quantities are
concealed. For example, Put all groceries in the fridge.

  * •

Implicit target location, item type and quantity: Item types and their
quantities along with the target location are implicitly defined. For example,
Clear the floor by placing the items at their appropriate positions. The agent
is expected to place items like pens, books, and laptops on the study table,
and litter in the trash can.

Metrics

We evaluate the algorithms using the following metrics to compare their
performances on the tasks:

  * •

Success Rate (SR): The fraction of episodes in which all subtasks are
completed. Success equals 1 if all subtasks are successfully executed in an
episode, otherwise it is 0.

  * •

Transport Rate (TR): The fraction of subtasks completed within an episode,
which provides a finer granularity of task completion.

  * •

Coverage (C): The fraction of successful interactions with target objects. It
is useful to verify if the LMs can infer the objects to interact with, in
scenarios where the tasks have objects that are specified implicitly.

  * •

Balance (B): The ratio between the minimum and maximum number of successful
high-level actions executed by any agent that contributed towards making
progress in a subtask necessary for the completion of the language instruction
task. We only check for a subset of high-level actions that must be executed
for accomplishing critical subtasks that leads to the successful completion of
the language instruction task. If each agent $i$ out of $n$ agents completes
$s_{i}$ successful tasks, the balance is defined as: $B:=\frac{\min\
\\{s_{1},\cdots,s_{n}\\}}{\max\\{s_{1},\cdots,s_{n}\\}+\epsilon}$. This
measures how evenly the work is distributed among agents. A balance of zero
indicates at least one agent performed no successful high-level actions, while
a balance of one indicates all agents performed the same number of successful
high-level actions. Here $\epsilon=1e-4$ is a small number to avoid division
by zero.

  * •

Average steps (L): The number of high-level actions taken by the team to
complete the task, capped at $T=30$ in our experiments. If the task is not
completed within $T$ steps, the episode is deemed a failure.

For all the metrics, we report the means along with the $95\%$ confidence
interval across all the tasks. Since SR is a binomial metric, we report the
Clopper-Pearson Interval as the confidence interval.

Baselines For a fair comparison with our method, we make modifications to the
baselines to make them work in partially observable settings with limited
reliance on the simulator. More details about implementations can be found in
Appendix E.

  * •

Act: We query the LLM with the task and the observations to suggest a high-
level action.

  * •

Chain-of-Thought (Wei et al., 2022): We modify the Act prompt with a chain-of-
thought style addendum to let the LM reason about the possible implications
while selecting a high-level action.

  * •

ReAct (Yao et al., 2023): We use a ReAct-style prompting to let the LMs reason
after suggesting high-level actions and possibly suggest ways to correct for
any failures.

  * •

SmartLLM  (Kannan et al., 2023): We modify the official codebase to only
include information from the local observations of the agents instead of
assuming full observability.

  * •

CoELA (Zhang et al., 2024): We modify the list of available high-level actions
to include all possible valid combinations of actions with interactable
objects in the agent’s local observation. As the scene becomes more cluttered,
this list and the prompt becomes combinatorially longer. In the original
implementation, the list of available actions is filtered based on the
feasibility of the actions as suggested by a conditional checker.

It should be noted that Act, Chain-of-Thought, ReAct and SmartLLM are all CMAS
frameworks where CoELA follows the DMAS framework.

##  6 Results and Discussion

Choice of the underlying LM: To understand the impact of the underlying LM’s
quality on decision-making, we initially experimented with different LMs.
Specifically, we utilize both the language-only and vision-language models of
GPT-4 (OpenAI et al., 2023), IDEFICS-2 (Laurençon et al., 2023, 2024), LLaVA
(Liu et al., 2023b, a), and CoGVLM (Hong et al., 2023a). Among these, GPT-4,
when used solely with text inputs, exhibits the poorest performance. This is
attributed to the agents’ inability to reason about visual observations, which
is particularly detrimental for the _Corrector_ module. Substituting GPT-4V
with other vision-language models results in a decline in performance (refer
Table 2) and hence we use GPT-4V as the underlying VLM while comparing to the
baselines.

Underlying | Success | Transport | Coverage | Balance | Steps
---|---|---|---|---|---
LM | Rate | Rate
GPT-4 | 0.51 | 0.85 | 0.95 | 0.83 | 25.80
(0.36, 0.66) | (0.80, 0.91) | (0.91, 0.98) | (0.78, 0.86) | (23.72, 27.88)
LLaVA | 0.54 | 0.84 | 0.91 | 0.75 | 26.21
(0.41, 0.65) | (0.71, 0.90) | (0.87, 0.98) | (0.64, 0.83) | (21.56, 28.97)
IDEFICS-2 | 0.57 | 0.86 | 0.94 | 0.78 | 25.27
(0.43, 0.67) | (0.74, 0.91) | (0.89, 0.98) | (0.65,0.84) | (20.14, 28.37)
CogVLM | 0.61 | 0.89 | 0.95 | 0.80 | 23.21
(0.47, 0.68) | (0.73, 0.95) | (0.89, 0.99) | (0.73, 0.86) | (20.57, 26.82)
GPT-4V | 0.66 | 0.91 | 0.97 | 0.82 | 21.87
(0.50, 0.76) | (0.81, 0.96) | (0.93,0.99) | (0.75, 0.87) | (18.76, 26.43)
Table 2: Metrics by varying the underlying LLM/VLM in LLaMAR for the 2-agent
scenario.

Baseline Comparisons: Table 3 compares our method, LLaMAR, with other
baselines in a 2-agent scenario using GPT-4V as the underlying VLM. Act and
ReAct show similar performance, with Act struggling due to its lack of
strategic planning or correction and ReAct performing slightly better by
dynamically adjusting actions based on reasoning on immediate feedback. CoT’s
performance declines with longer planning horizons due to its inability to
maintain coherence over extended planning sequences, consistent with findings
in (Stechly et al., 2024), showing its effectiveness only with highly specific
prompts. SmartLLM, operating in a _plan-and-execute_ paradigm, generates
impractical plans with issues like infinite loops and failure to handle low-
level action failures, leading to lower success rates and poor transport
metrics. It also tends to hallucinate objects. CoELA, using a decentralized
multi-agent system (DMAS), performs poorly due to large input prompts and
struggles to select the correct action from numerous choices. Its
decentralized decision-making is less efficient than the centralized multi-
agent system (CMAS) used by LLaMAR. Previous research (Chen et al., 2023)
confirms CMAS frameworks are more effective than DMAS frameworks. Overall, our
method, LLaMAR, benefits from its modular cognitive architecture, which
integrates planning, acting, correcting, and verifying through distinct LLM
roles, resulting in superior performance across various evaluation metrics. By
avoiding reliance on privileged information and incorporating a robust
exploration strategy allowing it to scout for objects not initially visible,
LLaMAR ensures higher success rates and balanced task execution among agents.

Algorithm | Success Rate | Transport Rate | Coverage | Balance | Steps
---|---|---|---|---|---
Act |  0.33 (0.19, 0.49) |  0.67 (0.59, 0.76) |  0.91 (0.86,0.95) |  0.59 (0.52, 0.66) |  24.92 (22.12,27.73)
ReAct |  0.34 (0.20, 0.49) |  0.72 (0.63,0.80) |  0.92 (0.86, 0.97) |  0.67 (0.61, 0.73) |  24.08 (21.27, 26.89)
CoT |  0.14 (0.06, 0.28) |  0.59 (0.51, 0.67) |  0.87 (0.81, 0.92) |  0.62 (0.56,0.69) |  28.4 (26.91, 29.97)
SmartLLM |  0.11 (0.05, 0.23) |  0.23 (0.13, 0.31) |  0.91 (0.80, 0.96) |  0.45 (0.37, 0.52) |  29.87 (26.20, 30)
CoELA |  0.25 (0.10, 0.36) |  0.46 (0.35, 0.56) |  0.76 (0.67, 0.85) |  0.73 (0.67, 0.80) |  28.93 (27.77,30)
LLaMAR | 0.51 (0.36, 0.66) | 0.85 (0.80, 0.91) | 0.95 (0.91, 0.98) | 0.83 (0.78, 0.86) | 25.80 (23.72, 27.88)
(w/o vision)
LLaMAR | 0.62 (0.46, 0.76) | 0.87 (0.80, 0.93) | 0.95 (0.91, 0.98) | 0.82 (0.77, 0.87) | 23.44 (20.88, 26.00)
(w/o exploration)
LLaMAR | 0.66 (0.50, 0.78) | 0.91 (0.81,0.96) | 0.97 (0.93,0.99) | 0.82 (0.75,0.87) | 21.87 (18.76,24.23)
(w/ exploration)
Table 3: Comparison of evaluation metrics against baselines averaged across
all tasks. More details about peculiar behaviors for the baselines can be
found in Appendix E

Roles of different modules in LLaMAR: To demonstrate the effectiveness of the
various modules in our cognitive architecture, we performed ablation studies
by evaluating performance metrics with each module removed individually. The
results are summarized in Table 4. Using only the _Actor_ module corresponds
to the "Act" baseline, which demonstrates its fundamental capabilities in
isolation but shows limited effectiveness without planning and correction due
to relatively lower success and transport rates. Adding the Planner and
Verifier modules improves performance, benefiting from better task planning
and validation, increasing the overall SR and TR, and ensuring more effective
task completion and even work distribution, as indicated by the increase in
balance (B). However, in scenarios where the suggested action fails, the actor
suggests the same action in the next decision step since it is not able to
reason on why the action failed until the end of the planning horizon.
Incorporating the

Modules | Success | Transport | Coverage | Balance | Steps
---|---|---|---|---|---
Used | Rate | Rate |  |  |
Actor | 0.33 | 0.67 | 0.91 | 0.59 | 24.92
(0.19, 0.49) | (0.59, 0.76) | (0.86,0.95) | (0.52, 0.66) | (22.12,27.73)
Planner + | 0.45 | 0.78 | 0.92 | 0.69 | 24.87
Actor + | (0.29, 0.57) | (0.67, 0.84) | (0.84, 0.95) | (0.61, 0.75) | (20.48, 27.95)
Verifier |  |  |  |  |
Planner + | 0.67 | 0.91 | 0.97 | 0.84 | 22.81
Actor + | (0.51, 0.80) | (0.83, 0.96) | (0.94, 0.99) | (0.79, 0.89) | (19.95, 25.76)
Corrector‡ |  |  |  |  |
LLaMAR | 0.66 | 0.91 | 0.97 | 0.82 | 21.87
(0.50, 0.76) | (0.81, 0.96) | (0.93,0.99) | (0.75, 0.87) | (18.76, 26.43)
Table 4: Ablating different modules LLaMAR with GPT-4V as the underlying VLM,
2-agents scenarios.

Corrector module with access to privileged information from an environment
oracle significantly boosts performance, enhancing the SR, TR, and C, further
improving B, and reducing L by approximately two time steps on average,
consistent with the findings in (Arora and Kambhampati, 2023). This highlights
the Corrector module’s importance in adjusting actions based on feedback,
resulting in higher task success and more efficient task completion, albeit
with reliance on oracle knowledge. Finally, the complete LLaMAR system,
without privileged information, achieves SR, TR, C, and B values close to
those of the oracle setup, with better L. This demonstrates the system’s
robustness and effectiveness in a realistic setting. The Corrector module
plays a crucial role in enabling agents to learn from past failures and avoid
repeating actions, preventing task failures due to timeout. Despite lacking
oracle knowledge, LLaMAR performs nearly as well as the oracle-enhanced setup.
These results highlight the importance of each module in our cognitive
architecture. Removing any module diminishes effectiveness, highlighting their
essential roles in achieving state-of-the-art results.

Increasing the number of agents Increasing the number of agents in the
environment shows clear trends in our method’s performance metrics (refer
Table 5).

# agents | Success | Transport | Coverage | Balance | Steps
---|---|---|---|---|---
Rate | Rate
2 | 0.62 | 0.87 | 0.95 | 0.82 | 23.44
(0.46, 0.76) | (0.80, 0.93) | (0.91, 0.98) | (0.77,0.87) | (20.88, 26.00)
3 | 0.70 | 0.91 | 0.98 | 0.66 | 21.30
(0.55, 0.82) | (0.85, 0.95) | (0.95, 0.99) | (0.61, 0.71) | (18.60, 23.99)
4 | 0.68 | 0.90 | 0.99 | 0.62 | 22.83
(0.52, 0.79) | (0.84, 0.94) | (0.95, 0.99) | (0.57, 0.68) | (19.63, 25.69)
5 | 0.62 | 0.90 | 0.99 | 0.54 | 22.91
(0.46, 0.75) | (0.85, 0.94) | (0.97,1.00) | (0.48, 0.59) | (20.26, 25.57)
Table 5: LLaMAR with more agents

With 2 agents, we establish a solid baseline for success rate (SR) and
transport rate (TR). Adding a third agent improves both SR and TR, indicating
enhanced task completion and transportation efficiency. Coverage (C) also
increases, suggesting better exploration and interaction with objects.
However, balance (B), which measures the even distribution of tasks among
agents, decreases with more agents. This drop highlights the challenge of
ensuring equal contributions from all agents in a larger multi-agent system.
While SR remains high, the balance metric drops significantly from 2 to 5
agents, indicating some agents do more work than others. The number of steps
(L) taken to complete tasks generally decreases with more agents, showing
improved efficiency but is less pronounced from 4 to 5 agents, suggesting
diminishing returns on efficiency with each additional agent. In summary,
adding more agents improves task performance and efficiency but introduces
challenges in maintaining balanced contributions. Addressing this imbalance is
crucial for refining multi-agent planning algorithms.

Correcting Failures: In numerous instances, the actions proposed by the
_Actor_ module, such as pick up $<$object$>$, are unsuccessful due to the
agent’s insufficient proximity to the target object. In such situations, the
_Corrector_ module uses visual feedback to learn from these failures and
recommends appropriate corrective actions, such as navigate to $<$object$>$ to
facilitate closer proximity. Figure 2 shows examples where the _Corrector_
module interprets low-level action failures and suggests remedies,
highlighting its importance.

Figure 2: A few examples of the _Corrector_ module mitigate failures in
predicted actions by the _Actor_ module. (a) the Corrector suggests getting
closer to the agent before attempting to pick it up, (b) the Corrector
recommends opening the fridge because the previous action of placing the plate
failed, (c) the Corrector advises rotating right so that it can access the
table to place the tissue box on it when the low-level navigation policy
failed to find a path to the table

##  7 Limitations and Future Work

Higher number queries to the LM: Since each high-level decision step requires
querying 4 different LM-based modules, the cost and the compute times are
higher than other baselines, especially compared to the plan-and-execute
baselines like SmartLLM. An interesting future direction to improve this would
be to fine-tune smaller LMs with trajectories collected in the simulator (eg:
ALFRED (Shridhar et al., 2019)) as done in (Zhang et al., 2024). Another
potential direction worth exploring is using different sizes of LMs for each
module based on their specific utility.

Limited spatial reasoning: Although we use both textual descriptions and
visual features to guide the language model’s actions, it still lacks the
ability to reason about the spatial features of the environment. Spatial
reasoning is crucial in scenarios such as navigating around obstacles to reach
an object, or determining the shortest path to collect multiple items
scattered across different locations. One way to address this limitation is to
inject information about the 3D world into the LM, as done in (Hong et al.,
2023b), which is an interesting direction for future work.

Performance limited by the underlying VLM: Although LMs make correct reasoning
most of the time, they still occasionally make mistakes, including
misunderstanding the environment rules specified in the prompt. For example,
the agent assumes that the cleaning task requires putting soap, drying, and
putting it in the sink when all it needs is the action “CleanObject”, and
can’t figure out the appropriate level of abstraction. The performance of the
algorithm is limited by the instruction following and reasoning capability of
the underlying LM (Kambhampati, 2024; Valmeekam et al., 2023b); this calls for
developing LMs that are fine-tuned to instruction-image pairs relevant to the
environment (as done in (Zhang et al., 2024)).

##  8 Conclusion

We address long-horizon planning in dynamic, partially observable multi-agent
environments with LLaMAR, an LM-based planner using four specialized modules:
_Planner_ , _Actor_ , _Corrector_ , and _Verifier_. This framework iteratively
refines action planning, adapts to failures, and verifies subtask completion
using real-time observations and action feedback, without privileged
information. We also introduce a heuristic-based exploration strategy to guide
agents to semantically relevant regions. Additionally, we present MAP-THOR, a
benchmark dataset for multi-agent tasks in the AI2Thor simulator. Empirical
results show LLaMAR outperforms existing LM-based approaches, achieving a 30%
higher success rate on MAP-THOR.

## Acknowledgements

We would like to thank Keerthana Gopalakrishnan, Sydney Dolan, Jasmine Aloor,
and Victor Qin for helpful discussions and feedback. OpenAI credits for GPT-4
access was provided through OpenAI’s Researcher Access Program. The research
was sponsored by the United States Air Force Research Laboratory and the
Department of the Air Force Artificial Intelligence Accelerator and was
accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views
and conclusions contained in this document are those of the authors and should
not be interpreted as representing the official policies, either expressed or
implied, of the Department of the Air Force or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for Government
purposes notwithstanding any copyright notation herein.

## References

  * Arora and Kambhampati (2023) Daman Arora and Subbarao Kambhampati.  Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models.  _arXiv preprint arXiv:2305.17077_ , 2023.
  * Barto and Mahadevan (2003) Andrew G. Barto and Sridhar Mahadevan.  Recent Advances in Hierarchical Reinforcement Learning.  _Discrete Event Dynamic Systems_ , 13:41–77, 2003.  URL https://api.semanticscholar.org/CorpusID:386824.
  * Batra et al. (2020) Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su.  Rearrangement: A Challenge for Embodied AI.  _CoRR_ , abs/2011.01975, 2020.  URL https://arxiv.org/abs/2011.01975.
  * Brodeur et al. (2017) Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron C. Courville.  HoME: a Household Multimodal Environment.  _CoRR_ , abs/1711.11017, 2017.  URL http://arxiv.org/abs/1711.11017.
  * Chen et al. (2023) Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan.  Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?  _arXiv preprint arXiv:2309.15943_ , 2023.
  * Dunbabin and Marques (2012) Matthew Dunbabin and Lino Marques.  Robots for Environmental Monitoring: Significant Advancements and Applications.  _IEEE Robotics & Automation Magazine_, 19(1):24–39, 2012.  doi: 10.1109/MRA.2011.2181683.
  * Gan et al. (2021) Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel L. K. Yamins, James J DiCarlo, Josh McDermott, Antonio Torralba, and Joshua B. Tenenbaum.  The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI, 2021.
  * Gramopadhye and Szafir (2023) Maitrey Gramopadhye and Daniel Szafir.  Generating Executable Action Plans with Environmentally-Aware Language Models, 2023.
  * Hong et al. (2023a) Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang.  CogAgent: A Visual Language Model for GUI Agents, 2023a.
  * Hong et al. (2023b) Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan.  3D-LLM: Injecting the 3D World into Large Language Models, 2023b.
  * Huang et al. (2023a) Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.  Visual Language Maps for Robot Navigation.  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , London, UK, 2023a.
  * Huang et al. (2022) Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 9118–9147. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/huang22a.html.
  * Huang et al. (2023b) Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, and Brian Ichter.  Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 59636–59661. Curran Associates, Inc., 2023b.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/bb3cfcb0284642a973dd631ec9184f2f-Paper-Conference.pdf.
  * Huang et al. (2023c) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter.  Inner Monologue: Embodied Reasoning through Planning with Language Models.  In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, _Proceedings of The 6th Conference on Robot Learning_ , volume 205 of _Proceedings of Machine Learning Research_ , pages 1769–1782. PMLR, 14–18 Dec 2023c.  URL https://proceedings.mlr.press/v205/huang23c.html.
  * Ichter et al. (2023) Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.  In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, _Proceedings of The 6th Conference on Robot Learning_ , volume 205 of _Proceedings of Machine Learning Research_ , pages 287–318. PMLR, 14–18 Dec 2023.  URL https://proceedings.mlr.press/v205/ichter23a.html.
  * Jain et al. (2019a) Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G. Schwing, and Aniruddha Kembhavi.  Two Body Problem: Collaborative Visual Task Completion.  In _CVPR_ , 2019a.  first two authors contributed equally.
  * Jain et al. (2019b) Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G. Schwing, and Aniruddha Kembhavi.  Two Body Problem: Collaborative Visual Task Completion.  _CoRR_ , abs/1904.05879, 2019b.  URL http://arxiv.org/abs/1904.05879.
  * Jain et al. (2020a) Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing.  A cordial sync: Going beyond marginal policies for multi-agent embodied tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 471–490. Springer, 2020a.
  * Jain et al. (2020b) Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander G. Schwing.  A Cordial Sync: Going Beyond Marginal Policies For Multi-Agent Embodied Tasks.  In _ECCV_ , 2020b.  first two authors contributed equally.
  * Kaelbling et al. (1998) Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra.  Planning and acting in partially observable stochastic domains.  _Artificial Intelligence_ , 101(1):99–134, 1998.  ISSN 0004-3702.  doi: https://doi.org/10.1016/S0004-3702(98)00023-X.  URL https://www.sciencedirect.com/science/article/pii/S000437029800023X.
  * Kambhampati (2024) Subbarao Kambhampati.  Can large language models reason and plan?  _Annals of the New York Academy of Sciences_ , 1534(1):15–18, 2024.
  * Kannan et al. (2023) Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Min.  SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models.  _arXiv preprint arXiv:2309.10062_ , 2023.
  * Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi.  AI2-THOR: An Interactive 3D Environment for Visual AI.  _arXiv_ , 2017.
  * Kumar et al. (2004) Vijay Kumar, D. Rus, and Sanjiv Singh.  Robot and sensor networks for first responders.  _IEEE Pervasive Computing_ , 3(4):24–33, 2004.  doi: 10.1109/MPRV.2004.17.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh.  OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents, 2023.
  * Laurençon et al. (2024) Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.  What matters when building vision-language models?, 2024.
  * Li et al. (2023) Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei.  BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation.  In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, _Proceedings of The 6th Conference on Robot Learning_ , volume 205 of _Proceedings of Machine Learning Research_ , pages 80–93. PMLR, 14–18 Dec 2023.  URL https://proceedings.mlr.press/v205/li23a.html.
  * Liang et al. (2022) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as Policies: Language Model Programs for Embodied Control.  In _arXiv preprint arXiv:2209.07753_ , 2022.
  * Lin et al. (2023) Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2Motion: from natural language instructions to feasible plans.  _Autonomous Robots_ , 47(8):1345–1365, November 2023.  ISSN 1573-7527.  doi: 10.1007/s10514-023-10131-7.  URL http://dx.doi.org/10.1007/s10514-023-10131-7.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved Baselines with Visual Instruction Tuning, 2023a.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual Instruction Tuning.  In _NeurIPS_ , 2023b.
  * Mandi et al. (2023) Zhao Mandi, Shreeya Jain, and Shuran Song.  RoCo: Dialectic multi-robot collaboration with large language models.  _arXiv preprint arXiv:2307.04738_ , 2023.
  * Meyer et al. (2020) Jesse Meyer, Ryan Praeuner, and Luke Vanderbeek.  Hierarchical Multi-Agent Reinforcement Learning.  https://cse.unl.edu/~lksoh/Classes/CSCE475_875_Fall11/seminars/Seminar_JRL.pdf, 2020.
  * Misra et al. (2018) Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi.  Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction.  _CoRR_ , abs/1809.00786, 2018.  URL http://arxiv.org/abs/1809.00786.
  * Nachum et al. (2018) Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine.  Data-Efficient Hierarchical Reinforcement Learning.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.  URL https://proceedings.neurips.cc/paper_files/paper/2018/file/e6384711491713d29bc63fc5eeb5ba4f-Paper.pdf.
  * OpenAI et al. (2023) OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  GPT-4 Technical Report, 2023.
  * Padmakumar et al. (2021) Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur.  TEACh: Task-driven Embodied Agents that Chat, 2021.
  * Pateria et al. (2021) Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek.  Hierarchical Reinforcement Learning: A Comprehensive Survey.  _ACM Comput. Surv._ , 54(5), jun 2021.  ISSN 0360-0300.  doi: 10.1145/3453160.  URL https://doi.org/10.1145/3453160.
  * Prasad et al. (2023) Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot.  ADaPT: As-Needed Decomposition and Planning with Language Models, 2023.
  * Puig et al. (2023) Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Ruslan Partsey, Jimmy Yang, Ruta Desai, Alexander William Clegg, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimír Vondruš, Vincent-Pierre Berges, John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi.  Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots, 2023.
  * Puig et al. (2018) Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  VirtualHome: Simulating Household Activities via Programs, 2018.
  * Puterman (1994) Martin L. Puterman.  _Markov Decision Processes_.  Wiley, 1994.  ISBN 978-0471727828.  URL http://books.google.com/books/about/Markov_decision_processes.html?id=Y-gmAQAAIAAJ.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  _CoRR_ , abs/2103.00020, 2021.  URL https://arxiv.org/abs/2103.00020.
  * Raman et al. (2022) Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex.  Planning with Large Language Models via Corrective Re-prompting.  January 2022.  URL http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127989.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, 2019.
  * Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra.  Habitat: A Platform for Embodied AI Research.  _CoRR_ , abs/1904.01201, 2019.  URL http://arxiv.org/abs/1904.01201.
  * Shah et al. (2022) Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine.  LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action.  In _6th Annual Conference on Robot Learning_ , 2022.  URL https://openreview.net/forum?id=UW5A3SweAH.
  * Sharma et al. (2021) Pratyusha Sharma, Antonio Torralba, and Jacob Andreas.  Skill Induction and Planning with Latent Language.  _CoRR_ , abs/2110.01517, 2021.  URL https://arxiv.org/abs/2110.01517.
  * Shridhar et al. (2019) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox.  ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks.  _CoRR_ , abs/1912.01734, 2019.  URL http://arxiv.org/abs/1912.01734.
  * Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  ProgPrompt: Generating Situated Robot Task Plans using Large Language Models.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11523–11530, 2023.  doi: 10.1109/ICRA48891.2023.10161317.
  * Singh et al. (2024) Ishika Singh, David Traum, and Jesse Thomason.  TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models.  _arXiv preprint arXiv:2403.17246_ , 2024.
  * Song et al. (2023) Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su.  LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models, 2023.
  * Stechly et al. (2024) Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati.  Chain of Thoughtlessness: An Analysis of CoT in Planning.  _arXiv preprint arXiv:2405.04776_ , 2024.
  * Stooke et al. (2021) Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin.  Decoupling representation learning from reinforcement learning.  In _International Conference on Machine Learning_ , pages 9870–9879. PMLR, 2021.
  * Sumers et al. (2023) Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths.  Cognitive Architectures for Language Agents, 2023.
  * Szot et al. (2021) Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimír Vondruš, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra.  Habitat 2.0: Training Home Assistants to Rearrange their Habitat.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Tampuu et al. (2015) Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente.  Multiagent Cooperation and Competition with Deep Reinforcement Learning.  _CoRR_ , abs/1511.08779, 2015.  URL http://arxiv.org/abs/1511.08779.
  * Tan (1993) Ming Tan.  Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents.  In _In Proceedings of the Tenth International Conference on Machine Learning_ , pages 330–337. Morgan Kaufmann, 1993.
  * Tang et al. (2019) Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Zhaopeng Meng, Changjie Fan, and Li Wang.  Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction, 2019.
  * Valmeekam et al. (2023a) Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati.  Can Large Language Models Really Improve by Self-critiquing Their Own Plans?  _arXiv preprint arXiv:2310.08118_ , 2023a.
  * Valmeekam et al. (2023b) Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati.  On the planning abilities of large language models-a critical investigation.  _Advances in Neural Information Processing Systems_ , 36:75993–76005, 2023b.
  * Wang et al. (2024) Jun Wang, Guocheng He, and Yiannis Kantaros.  Safe Task Planning for Language-Instructed Multi-Robot Systems using Conformal Prediction.  _arXiv preprint arXiv:2402.15368_ , 2024.
  * Wang et al. (2023) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen.  A Survey on Large Language Model based Autonomous Agents, 2023.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.  Chain of Thought Prompting Elicits Reasoning in Large Language Models.  _CoRR_ , abs/2201.11903, 2022.  URL https://arxiv.org/abs/2201.11903.
  * Wohlke et al. (2021) Jan Wohlke, Felix Schmitt, and Herke van Hoof.  Hierarchies of Planning and Reinforcement Learning for Robot Navigation.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, May 2021.  doi: 10.1109/icra48506.2021.9561151.  URL http://dx.doi.org/10.1109/ICRA48506.2021.9561151.
  * Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui.  The Rise and Potential of Large Language Model Based Agents: A Survey, 2023.
  * Xia et al. (2018) Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson Env: Real-World Perception for Embodied Agents.  _CoRR_ , abs/1808.10654, 2018.  URL http://arxiv.org/abs/1808.10654.
  * Xiang et al. (2020) Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su.  SAPIEN: A SimulAted Part-based Interactive ENvironment.  _CoRR_ , abs/2003.08515, 2020.  URL https://arxiv.org/abs/2003.08515.
  * Xu et al. (2023) Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng.  MAgIC: Benchmarking Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration.  _arXiv preprint arXiv:2311.08562_ , 2023.
  * Yang et al. (2019) Jiachen Yang, Igor Borovikov, and Hongyuan Zha.  Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery.  _CoRR_ , abs/1912.03558, 2019.  URL http://arxiv.org/abs/1912.03558.
  * Yang et al. (2020) Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang.  Multi-task reinforcement learning with soft modularization.  _Advances in Neural Information Processing Systems_ , 33:4767–4777, 2020.
  * Yang et al. (2023) Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans.  Foundation models for decision making: Problems, methods, and opportunities.  _arXiv preprint arXiv:2303.04129_ , 2023.
  * Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  ReAct: Synergizing Reasoning and Acting in Language Models, 2023.
  * Yu et al. (2023) Bangguo Yu, Hamidreza Kasaei, and Ming Cao.  Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation using Large Language Models.  _ArXiv_ , abs/2310.07937, 2023.  URL https://api.semanticscholar.org/CorpusID:263909555.
  * Zhang et al. (2024) Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan.  Building Cooperative Embodied Agents Modularly with Large Language Models.  _ICLR_ , 2024.
  * Zhu et al. (2017) Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi.  Visual semantic planning using deep successor representations.  In _Proceedings of the IEEE international conference on computer vision_ , pages 483–492, 2017.

##  Appendix A Terminology

We differentiate between the terms subtasks and high-level actions in this
section. In essence, multiple high-level actions are needed to be carried out
in a sequence to complete a subtask. Multiple subtasks need to be satisfied to
complete the high-level language instruction.

  * •

Subtasks: A task is split up into multiple subtasks. For example, if a task is
“Fetch all the groceries and put them in the fridge", then the initial
subtasks could include: “Locate the groceries", “transport the groceries",
“Locate the fridge". These subtasks could get updated with new observations.
For example, while locating the groceries, the agents come across a tomato and
a lettuce. Then the subtasks “transport the tomato to the fridge" and
“transport the lettuce to the fridge" gets updated in the subtasks list. This
basically splits up the high-level instruction $\mathcal{I}$ into multiple
mid-level subtasks

  * •

High-level actions: These are the set of actions required to complete the
subtasks. For example, to complete the “transport the lettuce in the fridge",
we would require: the following set of actions:

    * –

Navigate to lettuce

    * –

Pickup lettuce

    * –

Navigate to the fridge

    * –

Open fridge

    * –

Put lettuce in the fridge

    * –

Close fridge

Note that different agents have to complete different high-level actions that
progress the subtasks efficiently whilst avoiding conflicts.

  * •

Conflicts can arise in the following ways:

    * –

Same actions: Agents performing the same action at the same time. Example:
"Open the fridge".

    * –

Blocking: Agent 1 is blocking Agent 2 and not allowing it to complete its
high-level action. Example: Agent 1 is attempting to execute
“PlaceObject(Tomato)" in front of the fridge to place the tomato in its hand
in the fridge and Agent 2 is attempting to execute “OpenFreezer()" needs to
interact with the fridge. Would require some form of conflict resolution in
the state cell domain. Agent 1 should move away to allow fridge access to
Agent 2. In LLaMAR, the _Corrector_ module helps in figuring out these
conflicts and suggest different corrective high-level actions.

##  Appendix B Environment

The environment is based on the AI2Thor simulator with a multi-agent setup.
All the experiments were performed in the single-room floor plans. When more
than 3 agents are added to some of the floor plans (especially the kitchen
floor plans), the environment gets crowded and does not allow for a lot of
free space to navigate to different objects (the number of reachable paths
reduces).

(a) Kitchen

(b) Bedroom

(c) LivingRoom

(d) Bathroom

Figure 3: Photorealistic rendering of household scenarios in the AI2Thor
simulator enables the usage of multiple autonomous robots to carry out daily
tasks.

###  B.1 Observation Space

The observations for each robot include an image of size resolution
$1000\times 1000\times 3$. The textual observation for each agent in the
prompt is the list of objects visible in this image and the agents’ current
location and rotation. The field of view is $90$ degrees. The agents can
interact with the objects only if it is within its visibility range of $1.5$m.

###  B.2 Action Space

The actions space $\mathcal{A}$ consists of navigation actions
$\mathcal{A}_{NAV}$, interaction actions $\mathcal{A}_{INT}$, exploration
action $\mathcal{A}_{EXP}$.

Navigation actions $\mathcal{A}_{NAV}$ consists of the following actions:

  * •

Move($<$direction$>$): Moves the robot by $0.25$m towards the specified
direction where $<$direction$>$ can be one of (Ahead, Back, Right, Left)

  * •

Rotate($<$direction$>$): Rotates the robot by $90$ degrees towards the
specified direction where, $<$direction$>$ can be one of (Right, Left)

  * •

LookUp($<$angle$>$) rotates the pitch of the robot camera upwards by the
specified angle.

  * •

LookDown$<$angle$>$ rotates the pitch of the robot camera downwards by the
specified angle.

  * •

NavigateTo($<$object_id$>$) makes the robot navigate to the specified object.
The path is found using the $A^{*}-$shortest path algorithm. Note that the
robot is only able to find a path to the specified object in the environment
only if it has encountered that object previously during the episode.
Otherwise, the NavigateTo(.) action will be unsuccessful and the agent will
have to explore.

Interaction actions $\mathcal{A}_{INT}$ consists of the following actions:

  * •

Pickup($<$object_id$>$): Picks up the object

  * •

Put($<$receptacle_id$>$): Puts the object in the robots hand on the receptacle

  * •

Open($<$object_id$>$): Opens the object

  * •

Close($<$object_id$>$): Closes the open object

  * •

Slice($<$object_id$>$): Slices the object

  * •

Clean($<$object_id$>$): Cleans the object

  * •

ToggleOn($<$object_id$>$): Toggles the object on

  * •

ToggleOff($<$object_id$>$): Toggles the object off

Explore action $\mathcal{A}_{EXP}$: The explore action is carried out by the
heuristic mentioned in Algorithm 1. We use the clip-vit-large-patch14-336
model for the CLIP weights which we download from
https://huggingface.co/openai/clip-vit-large-patch14-336.

Figure 4: Choice of direction for the exploration heuristic: The agent (Alice)
rotates towards 4 cardinal directions to get observations. The cosine
similarity between the CLIP embeddings $I_{d}$ for these 4 images are
calculated with the CLIP embeddings for each subtask in the open subtasks set
$\mathcal{G}_{O}$ to get the exploration score $\mathcal{E}_{d}$ for each
direction. The direction with the highest $\mathcal{E}_{d}$ is chosen to
explore and the agent moves $J=2$ steps in that direction.  Algorithm 1
Exploration Heuristic

Input: Agent ID $n$, Environment $env$, number of exploration steps $K$,
number of move steps $J$

1:$g_{O}=\mathrm{CLIP}_{\mathrm{text}}(\mathcal{G}_{O})$

2:while $k<K$ do

3: Exploration Score $\mathcal{E}\in\mathbb{R}^{4}\leftarrow\mathbf{0}$

4: for $d\in\\{North,South,East,West\\}$ do

5: $o_{n,d}=env.step(\mathrm{Rotate(Right,n)})$

6: $I_{d}=\mathrm{CLIP}_{\mathrm{img}}(o_{n,d})$

7: $\mathcal{E}_{d}=\frac{I_{d}\cdot g_{O}}{\|I_{d}\|\|g_{O}\|}$

8: end for

9: $d^{*}=\arg\max_{d}\mathcal{E}$

10: while $j<J$ do

11: $o_{i}=env.step(\mathrm{Move}(d^{*},n))$

12: $j\leftarrow j+1$

13: end while

14: $k\leftarrow k+1$

15:end while

##  Appendix C Task Types

The complete list of task for each task type:

  * •

Explicit object type, explicit quantity and target:

    * –

put bread, lettuce, and a tomato in the fridge

    * –

Put the pots and pans on the stove burners

    * –

Slice the bread and tomato and crack the egg

    * –

Put the butter knife, bowl, and mug in the sink

    * –

Turn off the faucet and light if either is on

    * –

Put the tissue box, keys, and plate in the box

    * –

Put the computer, book, and pen on the couch

    * –

Put the bowl and tissue box on the table

    * –

Put apple in fridge and switch off the light

    * –

Put the watch and Keychain inside the drawer

    * –

Wash the bowl, mug, pot, and pan

    * –

Put the Box on the sofa and the bowl in the box

  * •

Explicit object type and explicit target: Here we explicitly describe the
object type but keep the quantity of the objects ambiguous. E.g. Put all the
apples in the fridge. For this, the agents have to explore the environment to
ensure that they find all of them.

    * –

Open all the drawers

    * –

Open all the cabinets

    * –

Turn on all the stove knobs

    * –

Put all the vases on the table

    * –

Put all the potatoes in the bowl

    * –

Put all pencils and pens in the box

    * –

Move all lamps next to the door

    * –

Turn off all light switches

    * –

Turn on all light switches

  * •

Explicit target, implicit object types: The object types are implicitly
defined whereas the target is explicitly defined. E.g. Put all groceries in
the fridge. This tests whether the model can identify objects of certain
categories.

    * –

Put all groceries in the fridge (should identify the tomato, bread, apple,
potato, and lettuce)

    * –

Put all shakers in the closest drawer (should identify the salt shaker and
pepper shaker)

    * –

Put all tableware on the countertop (should identify the bowl, plate, mug)

    * –

Put all food on the countertop (should identify the tomato, bread, apple,
potato, and lettuce)

    * –

Put all school supplies on the couch (should identify the pencil, computer,
and book)

    * –

Put all kitchenware in the cardboard box (should move the bowl and plate)

    * –

Put all silverware in the sink

    * –

Move everything on the table to the desk (should move the laptop, pencil, pen,
plate, credit card, book, and newspaper)

    * –

Slice the lettuce, trash the mug and switch off the light

    * –

Put all electronics on the couch

    * –

Make a dish by microwaving eggs and tomato

    * –

Put all readable objects on the sofa

    * –

Wash all fruits

  * •

Implicit target and object types: Here both the object type and the target are
explicitly defined. E.g. Clear the floor by placing the items at their
appropriate positions. Here the model is expected to keep items like pens,
book, laptop on the study table, litter in the trash can, etc.

    * –

Clear the floor by placing items at their appropriate positions (depending on
what’s on the floor)

    * –

Clear the table by placing the items in their appropriate positions (depends
on the floorplan, e.g. bread, apple, tomato, knife, bowl, book)

    * –

Clear the countertop by placing items in their appropriate positions (should
move the lettuce, mug, and paper towel roll)

    * –

Clear the desk by placing the items in other appropriate positions (should
move the statue, watch, and remote control)

    * –

Clear the table by placing the items in other appropriate positions (should
move the book, credit card, laptop, plate, newspaper, pen, and pencil)

    * –

Clear the couch by placing the items in other appropriate positions (should
move the pillow)

    * –

Make the living room dark

    * –

Make a mug of coffee and toast the bread

    * –

Trash all groceries

    * –

Slice all sliceable objects

##  Appendix D Pseudocode for LLaMAR

Algorithm 2 LLaMAR

Input: $N$ agents, Task instruction $\mathcal{I}$, Environment $env$
Initialize: Memory $\mathcal{M}\leftarrow\emptyset$; Open Subtasks
$\mathcal{G}_{O}\leftarrow\emptyset$;
Completed Subtasks $\mathcal{G}_{C}\leftarrow\emptyset$; Actions
$a\leftarrow\emptyset$;
Corrective Actions $a_{c}\leftarrow\emptyset$
Actions Executed $d\leftarrow\emptyset$

1:$o=(o_{1},\cdots,o_{N})=env.reset()$

2:while $t<T$ do

3:
$\mathcal{G}_{O}\leftarrow\mathrm{Planner}(\mathcal{I},o,\mathcal{G}_{O},\mathcal{G}_{C},\mathcal{M})$

4:
$a,\mathcal{M}\leftarrow\mathrm{Actor}(\mathcal{I},o,a_{c},\mathcal{G}_{O},\mathcal{G}_{C},\mathcal{M})$

5: $o=(o_{1},\cdots,o_{N}),d=(d_{1},\cdots,d_{N})=env.step(a)$

6:
$a_{c}\leftarrow\mathrm{Corrector}(\mathcal{I},o,a,d,\mathcal{G}_{O},\mathcal{G}_{C},\mathcal{M})$

7:
$\mathcal{G}_{C}\leftarrow\mathrm{Verifier}(\mathcal{I},o,a,d,\mathcal{G}_{O},\mathcal{G}_{C},\mathcal{M})$

8: if $\mathcal{G}_{O}=\emptyset$ then

9: break

10: end if

11: $t\leftarrow t+1$

12:end while

##  Appendix E Baselines

While there are a lot of impressive LLM-based multi-agent planners as
mentioned in Table 1, they vary in the assumptions about access to information
about the environment. We were not able to find the official codebase for the
Safe Multi-Agent Planning with Conformal Prediction (Wang et al., 2024) and
TwoStep (Singh et al., 2024). We describe the prompts used for our model as
well as every baseline. Note that we show the prompt for the 2-agent case, but
it is easily modified to generalize to the $n$-agent case. The italics and
bolding added for emphasis.

###  E.1 LLaMAR

We describe the prompts used for each of the modules used in LLaMAR:

Prompt for Planner Module in LLaMAR You are an excellent planner who is tasked
with helping 2 embodied robots named Alice and Bob carry out a task. Both
robots have a partially observable view of the environment. Hence they have to
explore around in the environment to do the task.
You will get a description of the task robots are supposed to do. You will get
an image of the environment from Alice’s perspective and Bob’s perspective as
the observation input. To help you with detecting objects in the image, you
will also get a list objects each agent is able to see in the environment.
Here the objects are named as “$<$object_name$>$_$<$object_id$>$".
So, along with the image inputs you will get the following information:
### INPUT FORMAT ###
{Task: description of the task the robots are supposed to do,
Alice’s observation: list of objects the Alice is observing,
Bob’s observation: list of objects the Bob is observing,
Robots’ open subtasks: list of subtasks the robots are supposed to carry out
to finish the task. If no plan has been already created, this will be None.
Robots’ completed subtasks: list of subtasks the robots have already
completed. If no subtasks have been completed, this will be None.
Robots’ combined memory: description of robots’ combined memory}
Reason over the robots’ task, image inputs, observations, open subtasks,
completed subtasks and memory, and then output the following:
* Reason: The reason for why new subtasks need to be added.
* Subtasks: A list of open subtasks the robots are supposed to take to complete the task. Remember, as you get new information about the environment, you can modify this list. You can keep the same plan if you think it is still valid. Do not include the subtasks that have already been completed.
The "Plan" should be in a list format where the subtask are listed
sequentially.
For example:
$[$“locate the apple", “transport the apple to the fridge", “transport the
book to the table"$]$
$[$“locate the cup", “go to cup", “clean cup"$]$
When possible do not perform additional steps when one is sufficient (e.g.
CleanObject is sufficient to clean an object, no other actions need to be
done) Your output should be in the form of a python dictionary as shown below.

Example output:
{“reason": "Since the subtask list is empty, the robots need to transport the
apple to the fridge and transport the book to the table.",
“plan": [“transport the apple to the fridge", “transport the book to the
table"]}
Ensure that the subtasks are not generic statements like "do the task". They
should be specific to the task at hand.
Do not assign subtasks to any particular robot. Try not to modify the subtasks
that already exist in the open subtasks list. Rather add new subtasks to the
list.

* NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED
Let’s work this out in a step by step way to be sure we have the right answer.

Prompt for Verifier Module in LLaMAR You are an excellent task verifier who is
tasked with helping 2 embodied robots named Alice and Bob carry out a task.
Both robots have a partially observable view of the environment. Hence they
have to explore around in the environment to do the task.
You will get a description of the task robots are supposed to do. You will get
an image of the environment from Alice’s perspective and Bob’s perspective as
the observation input. To help you with detecting objects in the image, you
will also get a list objects each agent is able to see in the environment.
Here the objects are named as “$<$object_name$>$_$<$object_id$>$".
So, along with the image inputs you will get the following information:

### INPUT FORMAT ###
{Task: description of the task the robots are supposed to do,
Alice’s observation: list of objects the Alice is observing,
Alice’s state: description of Alice’s state,
Alice’s previous action: the action Alice took in the previous step and if it
was successful,
Bob’s observation: list of objects the Bob is observing,
Bob’s state: description of Bob’s state, Bob’s previous action: the action Bob
took in the previous step,
Robots’ open subtasks: list of open subtasks the robots in the previous step.
If no plan has been already created, this will be None.
Robots’ completed subtasks: list of subtasks the robots have already
completed. If no subtasks have been completed, this will be None.
Robots’ combined memory: description of robots’ combined memory}
Reason over the robots’ task, image inputs, observations, previous actions,
open subtasks, completed subtasks and memory, and then output the following:
* Reason: The reason for why you think a particular subtask should be moved from the open subtasks list to the completed subtasks list.
* Completed Subtasks: The list of subtasks that have been completed by the robots. Note that you can add subtasks to this list only if they have been successfully completed and were in the open subtask list. If no subtasks have been completed at the current step, return an empty list.
The “Completed Subtasks" should be in a list format where the completed
subtasks are listed.
For example: $[$“locate the apple", “transport the apple to the fridge",
“transport the book to the table"$]$

Your output should be in the form of a python dictionary as shown below.
Example output:
{
"reason": "Alice placed the apple in the fridge in the previous step and was
successful and Bob picked up the the book from the table. Hence Alice has
completed the subtask of transporting the apple to the fridge, Bob has picked
up the book, but Bob has still not completed the subtask of transporting the
book to the table",
"completed subtasks": $[$"picked up book from the table", "transport the apple
to the fridge"$]$
}
* NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED
When you output the completed subtasks, make sure to not forget to include the
previous ones in addition to the new ones.
Let’s work this out in a step by step way to be sure we have the right answer.

Prompt for the Actor Module in LLaMAR You are an excellent planner and robot
controller who is tasked with helping 2 embodied robots named Alice, and Bob
carry out a task. All 2 robots have a partially observable view of the
environment. Hence they have to explore around in the environment to do the
task.

They can perform the following actions:
$[$“navigate to object $<$object_id$>$", “rotate in $<$rotation$>$ direction",
“pick up object $<$object_id$>$", “put object on $<$receptacle_id$>$", “open
object $<$object_id$>$", “close object $<$object_id$>$", “slice object
$<$object_id$>$", “toggle object $<$object_id$>$ on”, “toggle object
$<$object_id$>$ off”, “clean object $<$object_id$>$", “look up by angle
$<$angle$>$", “look down by angle $<$angle$>$", “move in $<$translation$>$
direction", “stay idle", “Done"$]$

Here “Done" is used when all the robots have completed the main task. Only use
it when you think all the subtasks are complete.
“stay idle" is used when you want the robot to stay idle for a one-time step.
This could be used to wait for the other robot to complete its subtask. Use it
only when you think it is necessary.
Here $<$rotation$>$ can be one of $[$“Right", “Left"$]$.
Here $<$angle$>$ is the angle in degrees and can only be one of
$[30,60,90,120,150,180]$.
Here $<$translation$>$ can be one of $[$“Ahead", “Back", “Left", “Right”$]$.
So, along with the image inputs you will get the following information:
### INPUT FORMAT ###
{Task: description of the task the robots are supposed to do,
Alice’s observation: list of objects the Alice is observing,
Alice’s state: description of Alice’s state,
Alice’s previous action: description of what Alice did in the previous time
step and whether it was successful,
Alice’s previous failures: if Alice’s few previous actions failed,
description of what failed,, Bob’s observation: list of objects the Bob is
observing,
Bob’s state: description of Bob’s state,
Bob’s previous action: description of what Bob did in the previous time step
and whether it was successful,
Bob’s previous failures: if Bob’s few previous actions failed, description of
what failed,
Robots’ open subtasks: list of subtasks supposed to carry out to finish the
task. If no plan has been already created, this will be None.
Robots’ completed subtasks: list of subtasks the robots have already
completed. If no subtasks have been completed, this will be None.
Robots’ subtask: description of the subtasks the robots were trying to
complete in the previous step,
Robots’ combined memory: description of robot’s combined memory}

### OUTPUT FORMAT ###
First of all you are supposed to reason over the image inputs, the robots’
observations, previous actions, previous failures, previous memory, subtasks
and the available actions the robots can perform, and think step by step and
then output the following things:

* Failure reason: If any robot’s previous action failed, use the previous history, your current knowledge of the room (i.e. what things are where), and your understanding of causality to think and rationalize about why the previous action failed. Output the reason for failure and how to fix this in the next timestep. If the previous action was successful, output "None".
Common failure reasons to lookout for include: one agent blocking another so
must move out of the way, agent can’t see an object or its destination and
must explore (such as move, rotate, or look in a different direction) to find
it, agent doing extraneous actions (such as drying objects when cleaning),
etc. If the previous action was successful, output "None".

* Memory: Whatever important information about the scene you think you should remember for the future as a memory. Remember that this memory will be used in future steps to carry out the task. So, you should not include information that is not relevant to the task. You can also include information that is already present in its memory if you think it might be useful in the future.

(CONTINUED) Prompt for the Actor Module in LLaMAR * Reason: The reasoning for
what each robot is supposed to do next

* Subtask: The subtask each robot should currently try to solve, choose this from the list of open subtasks.

* Action: The actions the robots are supposed to take just in the next step such that they make progress towards completing the task. Make sure that these suggested actions make these robots more efficient in completing the task as compared to only one agent solving the task. Your output should just be in the form of a python dictionary as shown below.

Examples of output:
Example 1:
{ "failure reason": "Bob failed to put the mug in the cabinet earlier because
Alice was blocking it when she was putting the knife. To fix this, Alice
should close the cabinet and move away , Charlie should move away to a
different open area than Alice to avoid congestion, and Bob should wait until
the next timestep until Alice can move aside.",
"memory": "Alice finished putting the knife in the cabinet when Alice was at
co-ordinates (1, .5) and was facing north. Bob wanted to put the mug in the
cabinet when Bob was at co-ordinates (1, 0.25) and was facing north.",
"reason": "Alice can close the cabinet door and then later back out in order
help Bob with completing the task. Bob can be idle until the next timestep
when Alice moves aside, by then Bob can navigate to the cabinet.",
"subtask": "Alice is currently closing the cabinet door, Bob is currently
waiting to get to navigate to the cabinet",
"Alice’s action" : "close the Cabinet_1",
"Bob’s action" : "stay idle"
}

Example 2: {
"failure reason": "Bob failed to clean the cup earlier because Bob had not
navigated to it, Bob assumed the cup to be in the sink which was erroneous. To
fix this, Bob should navigate to the cup and in the next step clean cup.",
"memory": "Alice finished navigating to the dish when Alice was at co-
ordinates (-.5, .5) and was facing east. Bob was not able to clean the cup in
the cabinet when Bob was at co-ordinates (1, .25) and was facing north.",
"reason": "Alice can now clean the dish since Alice has navigated to it. Bob
should navigate to the cup in order to be close enough to clean the cup.",
"subtask": "Alice is currently trying to clean the dish, Bob is currently
trying to navigate to the cup",
"Alice’s action" : "clean the dish object",
"Bob’s action" : "navigate to the cup" }
Note that the output should just be a dictionary similar to the example
outputs.

### Important Notes ###
* The robots can hold only one object at a time.
For example: If Alice is holding an apple, she cannot pick up another object
until she puts the apple down.
* Even if the robot can see objects, it might not be able to interact with them if they are too far away. Hence you will need to make the robot navigate closer to the objects they want to interact with.
For example: An action like “pick up $<$object_id$>$" is feasible only if
robot can see the object and is close enough to it. So you will have to
navigate closer to it before you can pick it up.
* In some scenarios, the agents might not see the objects that they want to interact with. In such cases, you will have to make the robot explore the environment to find the object. In such scenarios you can use actions to rotate in place or look up / down or navigate to explore the environment.
* If you open an object, please ensure that you close it before you navigate to a different place.
* Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only when you think it is necessary.
* NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED

###  E.2 Act

We describe the prompt used for the Act baseline:

Prompt for the Act Baseline You are an excellent planner and robot controller
who is tasked with helping 2 embodied robots named Alice and Bob carry out a
task. Both robots have a partially observable view of the environment. Hence
they have to explore around in the environment to do the task.

They can perform the following actions:
$[$“navigate to object $<$object_id$>$", “rotate in $<$rotation$>$ direction",
“pick up object $<$object_id$>$", “put object on $<$receptacle_id$>$", “open
object $<$object_id$>$", “close object $<$object_id$>$", “slice object
$<$object_id$>$", “toggle object $<$object_id$>$ on”, “toggle object
$<$object_id$>$ off”, “clean object $<$object_id$>$", “look up by angle
$<$angle$>$", “look down by angle $<$angle$>$", “move in $<$translation$>$
direction", “stay idle", “Done"$]$

Here “Done" is used when all the robots have completed the main task. Only use
it when you think all the subtasks are complete.
“stay idle" is used when you want the robot to stay idle for a one-time step.
This could be used to wait for the other robot to complete its subtask. Use it
only when you think it is necessary.
Here $<$rotation$>$ can be one of $[$“Right", “Left"$]$.
Here $<$angle$>$ is the angle in degrees and can only be one of
$[30,60,90,120,150,180]$.
Here $<$translation$>$ can be one of $[$“Ahead", “Back", “Left", “Right”$]$.
You need to suggest the action that each robot should take at the current time
step.
### Important Notes ###
* The robots can hold only one object at a time.
For example: If Alice is holding an apple, she cannot pick up another object
until she puts the apple down.
* Even if the robot can see objects, it might not be able to interact with them if they are too far away. Hence you will need to make the robot navigate closer to the objects they want to interact with.
For example: An action like “pick up $<$object_id$>$" is feasible only if
robot can see the object and is close enough to it. So you will have to
navigate closer to it before you can pick it up.
* In some scenarios, the agents might not see the objects that they want to interact with. In such cases, you will have to make the robot explore the environment to find the object. In such scenarios you can use actions to rotate in place or look up / down or navigate to explore the environment.
* If you open an object, please ensure that you close it before you navigate to a different place.
* Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only when you think it is necessary.
### INPUT FORMAT ###
* You will get a description of the task robots are supposed to do.
* You will get an image of the environment at the current time step from Alice’s perspective and Bob’s perspective as the observation input. Here the objects are named as “$<$object_name$>$_$<$object_id$>$".
* You will get a trace of the steps taken by the robots and the actions they took at each time step and whether it was successful or not.
### OUTPUT FORMAT ###
In your output, do not have any extra text or content outside of the python
dictionary as below. Do NOT put any text, spaces, or enter keys (i.e. “/n")
outside of it.

Your output should ONLY be in the form of a python dictionary, without any
reasoning or extra text, as shown below:
{“Alice": “action to be taken by Alice",
“Bob": "action to be taken by Bob}

For example: If you think Alice should pick up an apple and Bob should
navigate to the fridge, you will have to give the output as:
{“Alice": “pick up apple",
“Bob": “navigate to fridge"} * NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN
WHAT HAS BEEN SPECIFIED

###  E.3 ReAct

We describe the prompt used for the ReAct baseline:

Prompt for ReAct Baseline You are an excellent planner who is tasked with
helping 2 embodied robots named Alice and Bob carry out a task. Both robots
have a partially observable view of the environment. Hence they have to
explore around in the environment to do the task.

They can perform the following actions: $[$"navigate to object
$<$object_id$>$", "rotate in $<$rotation$>$ direction", "pick up object
$<$object_id$>$", "put object on $<$receptacle_id$>$", "open object
$<$object_id$>$", "close object $<$object_id$>$", "slice object
$<$object_id$>$", “toggle object $<$object_id$>$ on”, “toggle object
$<$object_id$>$ off”, "clean object $<$object_id$>$", "look up by angle
$<$angle$>$", "look down by angle $<$angle$>$", “move in $<$translation$>$
direction", "stay idle", "Done"$]$
Here "Done" is used when all the robots have completed the main task. Only use
it when you think all the subtasks are complete.
"stay idle" is used when you want the robot to stay idle for a one-time step.
This could be used to wait for the other robot to complete its subtask. Use it
only when you think it is necessary.
Here $<$rotation$>$ can be one of $[$"Right", "Left"$]$.
Here $<$angle$>$ is the angle in degrees and can only be one of
$[30,60,90,120,150,180]$.
Here $<$translation$>$ can be one of $[$“Ahead", “Back", “Left", “Right”$]$.
You need to suggest the action that each robot should take at the current time
step. ### Important Notes ###
* The robots can hold only one object at a time.
For example: If Alice is holding an apple, she cannot pick up another object
until she puts the apple down.
* Even if the robot can see objects, it might not be able to interact with them if they are too far away. Hence you will need to make the robot navigate closer to the objects they want to interact with.
For example: An action like "pick up $<$object_id$>$" is feasible only if
robot can see the object and is close enough to it. So you will have to
navigate closer to it before you can pick it up.
* In some scenarios, the agents might not see the objects that they want to interact with. In such cases, you will have to make the robot explore the environment to find the object. In such scenarios you can use actions to rotate in place or look up / down or navigate to explore the environment.
* If you open an object, please ensure that you close it before you navigate to a different place.
* Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only when you think it is necessary.
### INPUT FORMAT ###
* You will get a description of the task robots are supposed to do.
* You will get an image of the environment at the current time step from Alice’s perspective and Bob’s perspective as the observation input. Here the objects are named as "$<$object_name$>$_$<$object_id$>$".
* You will get a trace of the steps taken by the robots and the actions they took at each time step and whether it was successful or not.
### OUTPUT FORMAT ###
You are supposed to think and suggest the action each robot is supposed to
take at the current time step. Before suggesting an action you need to think,
which requires that you reason over the inputs and logically reflect on the
task, observation and course of actions needed to complete the task. Output
Requirements: At each time step you must ONLY output a PYTHON DICTIONARY of
the following two elements:
*First Element: Key = "Think" $|$ Value:(Type: String): A logical reflection of the best action to be taken given the inputs: task at hand, observations, and trace.
*Second Element: Key = "Action" $|$ Value:(Type: Python Dictionary):
The value should be in the form of a python dictionary as shown below.
{"Alice": "action to be taken by Alice", "Bob": "action to be taken by Bob"}

For example: If you think Alice should pick up an apple and Bob should
navigate to the fridge, you will have to give the output as: {"Alice": "pick
up apple", "Bob": "navigate to fridge"} Here is an example output:
{"Think": "To solve the task, I need to find and put the apple. The apple is
likely to be on the countertop or table. Then find the fridge.", "Action":
{"Alice": "pick up apple", "Bob": "navigate to fridge"} }
* NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED

###  E.4 Chain of Thought

We describe the prompt used for the Chain-of-Thought baseline:

Prompt for Chain of Thought Baseline You are an excellent planner who is
tasked with helping 2 embodied robots named Alice and Bob carry out a task.
Both robots have a partially observable view of the environment. Hence they
have to explore around in the environment to do the task.

They can perform the following actions: $[$“navigate to object
$<$object_id$>$", “rotate in $<$rotation$>$ direction", “pick up object
$<$object_id$>$", “put object on $<$receptacle_id$>$", “open object
$<$object_id$>$", “close object $<$object_id$>$", “slice object
$<$object_id$>$", “toggle object $<$object_id$>$ on”, “toggle object
$<$object_id$>$ off”, “clean object $<$object_id$>$", “look up by angle
$<$angle$>$", “look down by angle $<$angle$>$", “move in $<$translation$>$
direction", “stay idle", “Done"$]$ Here “Done" is used when all the robots
have completed the main task. Only use it when you think all the subtasks are
complete. “stay idle" is used when you want the robot to stay idle for a one-
time step. This could be used to wait for the other robot to complete its
subtask. Use it only when you think it is necessary. Here $<$rotation$>$ can
be one of $[$“Right", “Left"$]$.
Here $<$angle$>$ is the angle in degrees and can only be one of
$[30,60,90,120,150,180]$.
Here $<$translation$>$ can be one of $[$“Ahead", “Back", “Left", “Right”$]$.
You need to suggest the action that each robot should take at the current time
step.
### Important Notes ###
* The robots can hold only one object at a time. For example: If Alice is holding an apple, she cannot pick up another object until she puts the apple down.
* Even if the robot can see objects, it might not be able to interact with them if they are too far away. Hence you will need to make the robot navigate closer to the objects they want to interact with. For example: An action like “pick up $<$object_id$>$" is feasible only if robot can see the object and is close enough to it. So you will have to navigate closer to it before you can pick it up.
* In some scenarios, the agents might not see the objects that they want to interact with. In such cases, you will have to make the robot explore the environment to find the object. In such scenarios you can use actions to rotate in place or look up / down or navigate to explore the environment.
* If you open an object, please ensure that you close it before you navigate to a different place.
* Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only when you think it is necessary.
### INPUT FORMAT ###
* You will get a description of the task robots are supposed to do.
* You will get an image of the environment at the current time step from Alice’s perspective and Bob’s perspective as the observation input. Here the objects are named as "$<$object_name$>$_$<$object_id$>$".
* You will get a trace of the steps taken by the robots and the actions they took at each time step and whether it was successful or not.
### OUTPUT FORMAT ###
You are supposed to FIRST reason through the situation logically and step by
step, then suggest the action each robot is supposed to take at the current
time step.
In your output, do not have any extra text or content outside of the python
dictionary as below.
Your output should ONLY be in the form of a python dictionary as shown below:
{"reason": "Reasoning for action plan….", "Alice": "action to be taken by
Alice", "Bob": "action to be taken by Bob"}
Put all of your reasoning inside of the “reason" key of the dictionary. Do NOT
put any text, spaces, or enter keys (i.e. “/n") outside of it.
For example: If you think Alice should pick up an apple and Bob should
navigate to the fridge, you will have to give the output as:
{"reason": "since the subtask list is empty, the robots need to transport the
apple to the fridge", "Alice": "pick up apple", "Bob": "navigate to fridge"}
Let’s think step by step, but make sure to put all of your reasoning inside of
the “reason" key of the dictionary!
* NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED

###  E.5 SmartLLM

We adapt the prompt from the official codebase of SmartLLM (master branch;
commit #be42930050f7d4d8f2fad027aff14a699c3300aa) as given here:
https://github.com/SMARTlab-Purdue/SMART-LLM/blob/master/scripts/run_llm.py
with a slight modification. Instead of letting the agents access all the
objects in the environment through the simulator metadata, we just give the
list of objects visible from the agents‘ point-of-view.

###  E.6 CoELA

We adapt the prompt from the official codebase of CoELA (master branch: commit
$\\#$3d34de46dc77f9aaabe438cd2b92ea6c5c04973a) as given here:
https://github.com/UMass-Foundation-Model/Co-LLM-
Agents/tree/master/tdw_mat/LLM. We modify some aspects of the prompt as
described: Instead of relying on the simulator/pre-defined conditional logic
for generating the list of available action options, we give a list of all
possible actions based on the observation. This includes the option to send
the communication message, all navigation actions, and all combinations of
valid actions with the interactable objects in the current observation.

##  Appendix F Open Source VLMs

We list the source of the weights we used for the open-source VLMs:

  * •

Idefics 2 (Laurençon et al., 2023, 2024): We use the 8B base model fine-tuned
on a mixture of supervised and instruction datasets (text-only and multimodal
datasets) from HuggingFace. The weights were downloaded from
https://huggingface.co/HuggingFaceM4/idefics2-8b with the commit
#2c031da2dc71f3ac989f9efa9b8ff476df3842c0. We chose Idefics because it is able
to take multiple images as input similar to GPT-4V and reason on them.

  * •

LLaVA (Liu et al., 2023b): We use the 7B model t trained by fine-tuning
LLaMA/Vicuna on GPT-generated multimodal instruction-following data. The
weights were downloaded from https://huggingface.co/llava-hf/llava-1.5-7b-hf
with the commit # 05ae2434cbb430be33edcba0c5203e7023f785b7.

  * •

CogVLM (Hong et al., 2023a): We use the 18B model. The weights were downloaded
from https://huggingface.co/THUDM/cogagent-chat-hf with the commit #
d519da3b191401234f4bd86ce1c287c61bc276a3.

##  Appendix G SentenceBERT fine-tuning

We finetuned a pre-trained BERT model to function as a semantic mapper between
free-form natural language output and the robot’s admissible actions in the
environment. The pre-trained weights were obtained from
https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2. The model was
trained on a dataset consisting of 2800 free-form input, valid action output
pairs. It ran on one (1) Apple M1 core for a wall clock time of 5 minutes.
Table 6 shows the hyper-parameters used for the pre-training of the BERT
model.

Epochs | 10
---|---
Max gradient norm | 1
Learning rate | $2\times 10^{-5}$
Batch size | 64
Encoding dimension | 384
Optimizer | AdamW
Scheduler | Warm-up linear
Warm-up steps | 45
Weight decay | 0.01
Loss scale | 20
Loss type | Multiple negatives ranking loss
Similarity function | Cosine similarity

Table 6: Hyper-parameters for the model fine-tuning including the loss.