# Natural Language as Polices:
Reasoning for Coordinate-Level Embodied Control with LLMs

Yusuke Mikami1, Andrew Melnik2, Jun Miura3, Ville Hautamäki4 1,3Department of
Computer Science and Engineering at Toyohashi University of Technology,
Japan.2Bielefeld University, Germany.1,4School of Computing at University of
Eastern Finland, Finland.1 Correspondence to: Yusuke Mikami
<EMAIL_ADDRESS>Under review.

###### Abstract

We demonstrate experimental results with LLMs that address robotics action
planning problems. Recently, LLMs have been applied in robotics action
planning, particularly using a code generation approach that converts complex
high-level instructions into mid-level policy codes. In contrast, our approach
acquires text descriptions of the task and scene objects, then formulates
action planning through natural language reasoning, and outputs coordinate
level control commands, thus reducing the necessity for intermediate
representation code as policies. Our approach is evaluated on a multi-modal
prompt simulation benchmark, demonstrating that our prompt engineering
experiments with natural language reasoning significantly enhance success
rates compared to its absence. Furthermore, our approach illustrates the
potential for natural language descriptions to transfer robotics skills from
known tasks to previously unseen tasks.

Figure 1: Overview of our approach. We provide one demonstration as an in-
context example, and a planning step employing natural language reasoning
instead of conventional code implementation. We remove the CoT(Chain-of-
Thouht) reasoning component in the in-context example for our ablation study
to check the importance of natural language reasoning. We use low-level
API(pick-and-place or sweep) to control the robot arm. We presents specific
example of natural language reasong in Table.V.

##  I Introduction

Robotics action planning guided by human-level instruction presents a
challenging topic for general robotics applications, as it entails the
decomposition of high-level instructions into low-level executable robotics
commands. Conventional approaches tend to address these problems in a highly
task-specific and static manner and consequently struggle to achieve open-
vocabulary object detection, novel task generalization, and a reduction of the
training process in general.

_Large language models_ (LLMs) [1, 30] have had a significant impact on
various applications, not only text generation tasks but also robotics action
planning where LLMs attempt to interpret human-level instruction or
demonstrations. Specifically, LLM-based robotics action planning has focused
primarily on code generation approaches (CaP [14], Progprompt [33], Chain of
Code [12], SocraticModels [37], and Instruct2Act [6]) which leverage the in-
context learning capability of LLM to produce code implementations by
integrating predefined APIs that interface with the physical world. These
studies solve the embodied control problem from an algorithmic perspective
since they try to make intermediate code implementation from high-level
instructions.

Recently, LLM-based robotic planning has emphasized zero-shot scenarios in
robotic planning (Kwon et al. [11], Socratic Models [37]) which have a huge
advantage since robotics action planning often encounters novel objects,
situations, and tasks. However, we propose that although LLMs can address
general situations without any in-context examples, it is crucial to exploit
their in-context learning capability to address a complicated novel situation
and task by utilizing knowledge of previously encountered similar tasks.

In the in-context learning process of the recent LLM-based code generation
approaches, we suppose there are two primal limitations. First, code
implementation itself lacks high-level contextual meaning to efficiently
describe embodied skills since usually it is symbolized, indirectly connected
to the scene, and abstracted. Second, these approaches are limited by pre-
defined APIs. We hypothesize that the natural language description of the
whole planning process, instead of code, can contribute to removing the
limitations.

To overcome the limitations and advance the current state of robotics toward
more semantic capability, we introduce a _Chain-of-Thought_ (CoT) [35]-based
reasoning framework. In this framework, we initially possess all necessary
information as text and generate the natural language reasoning and action
plans without relying on pre-defined APIs. Our goal is to make everything
explicit with natural language to efficiently describe embodied skills
semantically for LLMs. For prompt-engineering experiments, we used tabletop
manipulation tasks with multimodal prompts (VIMABench [10]).

Our Contributions:

  * •

Reasoning with direct interaction of environment: Our approach enables agents
to interact directly with information from the current environment.

  * •

Coordinate-Level action prediction: The output of our approach consists of
coordinates that can be directly executed by the target robot.

  * •

Teaching robots with natural language: Our approach suggests a way for humans
to teach robots to perform tasks in a manner similar to how they teach other
humans.

  * •

New way to tackle novel situation: Our approach suggests a way to improve the
transferability of robotics skills from a known task to a novel task at the
natural language context level.

##  II Related works

One of the key objectives in the field of robotics is to develop a system
capable of learning new tasks described in natural language, using only a
handful of demonstration examples, and capable of working with an open
vocabulary range of objects, similar to human abilities. Multiple attempts
were made to advance existing architectures towards this long-term goal.
Recent developments in Large Language and Foundation Models [25, 30] allowed
robotic architectures to make substantial progress toward this objective. We
present recent LLM-based approaches in Table.I.

###  II-A Imitation learning and RL for robotics action planning

Imitation learning and reinforcement learning (RL) [2] are common frameworks
for robotics action planning. A neural network takes information from the
environment, and the output comprises executable actions. These learning
frameworks acquire robotics skills in their neural network parameters
implicitly. These conventional approaches encounter limitations including long
training duration [21], overfitting for specific tasks [28], and limited input
flexibility. Dataset-search policy approaches [3, 17, 18, 22] propose zero-
shot adaptation to provided tasks examples, thus improving in flexibility over
imitation learning approach, however still having limitations for novel-task
execution. In contrast, we leverage the reasoning capability of pre-trained
LLMs with an explicit planning process to tackle the limitations.

###  II-B Natural language commands to code scripts with LLMs

Some approaches explored the translation of natural language commands into
executable code scripts [4, 6, 8, 14, 15, 23, 29, 37] where a set of examples
of translation between natural language commands and executable scripts or a
description of pre-defined APIs is provided, such that the model can do
correct translation for new natural language commands. These code-based
approaches attempt to solve robotics action planning from an algorithmic
perspective, rather than a semantic one. Chain-of-thought (CoT) [34] enhances
the reasoning abilities of large language models (LLMs) by decomposing complex
tasks into smaller steps and providing examples of intermediate reasoning
steps through multiple prompts. Some approaches explored the integration of
CoT or reasoning processes into robotics tasks. Statler [36] offers a state
management framework for long-horizon planning tasks. Demo2Code [32] has an
efficient intermediate representation by summarizing demonstrations to produce
final actions. Progprompt [29] generates situated task plans as code
implementation. Inner Monologue [8] focuses on feedback and interaction
processes in the reasoning process. Chain of Code [12] has advantages in both
algorithmic and semantic capability by using LLM as a code interpreter. These
reasoning-focused studies leverage LLMs as reasoners for grounding, however,
still stick to a basic concept where LLM is used as a code generator by
filling arguments of pre-defined API. On the other hand, EmbodiedGPT [24]
attempts to efficiently integrate the imitation learning process and the CoT
reasoning process in its training process, however, it still requires a fine-
tuning process with a customized dataset. In contrast to these conventional
approaches, our approach solve robotics action planning by especially focusing
on a semantic perspective rather than conventional algorithmic without
additional fine-tuning.

###  II-C LLM-based code generation for multi-modal prompts

Jiang et al.[10] introduced VIMA, which can act upon multimodal prompts within
the end-to-end imitation learning approach. Jiachen et al.[13] proposed a pre-
train and fine-tune approach for the VIMA model. Huang et al.[6] introduced
Instruct2Act which utilizes ”Code as policy” [14] to generate executable
actions from multimodal prompts, drawing inspiration from VISPROG [5].
However, this approach solves the VIMABench [10] task in a task-specific way
since it needs code implementation as an in-context example for each task.
Also, this approach has limitations from pre-defined APIs and may not solve
complicated text-image matching problems with only CLIP [25] based perception.
In contrast, our approach tries to achieve a flexible object detection process
by describing objects and reasoning in natural language.

###  II-D Novel Tasks and Generalisation

Novel tasks have been a crucial problem for robotics. BC-Z [9] tackles the
zero-shot problem by having a huge dataset and task embeddings. Huang et al.
[7] introduce action planning in zero-shot situations considering executable
pre-defined actions. Socratic Models [37] leverage the zero-shot capability of
LLMs to translate simple actions into code with in-context examples. Teyun et
al.[11] propose an LLM-based code generation approach without any in-context
examples, relying on reasoning to address zero-shot tasks. Obviously, Zero-
shot is a promising approach, however, LLM can handle only general and simple
situations and prompts in zero-shot situations [26]. Therefore we emphasize
that it is crucial to use in-context examples of previously encountered tasks
if the task is complicated, for instance, VIMABench[10].

###  II-E Open vocabulary object detection

Usage of open vocabulary object detection models [16, 19] is one way of
embracing open vocabulary set of objects into reasoning about a task [20].
Differences in objects of the same type in an environment may be complicated
to express in natural language, thus integration of images of intended objects
and text into multi-modal task specification [31] can provide performance
benefits. We try to achieve open vocabulary by leveraging a pre-trained huge
Vision Language model (for instance, GPT4-Vision) rather than an object
detection model.

TABLE I: Key components of related code generation and reasoning approaches:
EmbodiedGPT[24], Socratic Models[37], Inner Monologue[8], Statler[36],
Demo2Code[32], Chain of Code[12], Progprompt[29], ChatGPT for Robotics[11],
Code as Policies[14], Instruct2Act[6], and Zero-Shot Trajectory[11]. Our
approach has differences, especially in natural language reasoning and
coordinate-level output.

##  III Methods

While some studies focus on the code generation approach with in-context
examples, we explore the decomposition of high-level tasks and the generation
of coordinate-level actions with only natural language reasoning.

###  III-A Problem formulation

Our approach solves robotic planning for tabletop manipulation. The goal is to
modify the state of objects in the environment to match the configuration
described in the instructions. Jiang et al.[10] introduced the VIMABench
framework as an open-source project for evaluating the performance of
multimodal prompts. It comprises 17 tasks across four levels of generation.

####  III-A1 Interface of planning

This section provides inforamtion about the interface of VIMABench[10] for
this work. VIMABench provides two information for planner, which consists of
multimodal prompts, which include both text and images depicting a single
object and a scene with multiple objects. The scene image provided offers both
top and front views, showcasing several objects within the scene.

The output of the planning result must include specific parameters for the
start and end points of the action. These parameters encompass the coordinates
in the $x$ and $y$ dimensions, representing a top view for the action
execution. They are utilized for both sweeping and picking actions within a
two-dimensional space framework. Additionally, the rotation of the end-
effector must be defined for both the start and end points. This parameter
facilitates the rotation action of objects, although it is not relevant for
sweep actions. During sweep actions, the rotation of the end-effector is
automatically managed.

####  III-A2 Actions

The target robot has two available actions: ”Pick and place,” which involves
picking up an object in one location and placing it in another, and ”Sweep,”
which entails moving objects by dragging without lifting them. These actions
are selectable automatically by the benchmark depending on the task,
eliminating the need for explicit action selection.

####  III-A3 Generalization level

VIMABench provides four levels of generalization including placement, novel
combination, novel objects, and novel task. Each level evaluates different
zero-shot capabilities. In this work, we focus on only placement and novel
task generalization. Given that our approach does not necessitate a massive
training dataset, novel combinations, and novel object generalization hold
limited significance for our methodology. For novel generalization, we
manually select an example from a similar task for in-context learning. In
VIMABench, task 10 (follow-motion) and task 13(sweep-without-touching), hold
significance for novel task generalization since these tasks have novel
concepts and words which is not available in other generalizations.
Conversely, tasks 8 and 14 are deemed less crucial from the perspective of
novel tasks, as they share similarities with tasks already present in the
other three generalizations. For instance, task 8(novel-adj-and-noun) has a
similar concept to tasks 6(novel-adj) and 7(novel-noun).

###  III-B Our approach

####  III-B1 Overview

We introduce a robotics action planning framework to solve multimodal prompts
by thinking of everything in natural language with CoT shown in Fig 1. We
first convert all images from the prompt and scene into a text description and
make an action prediction with one in-context example. Our approach stores
robotics skills as natural language explicitly.

####  III-B2 Pipeline

We translate the segmented images into text, adhering to the object
description format. Subsequently, we make action predictions incorporating in-
context examples with a prompt in Fig.2. The language model generates action
predictions, including reasoning steps and actual actions, conforming to the
action output format. Following this, we perform coordinate mapping,
translating output coordinates from front-view to top-view. Finally, we
execute the actions, which can either involve pick-and-place operations or
sweeping tasks.

Figure 2: The full prompt with ellipses indicating omitted sections due to
space limitations.

####  III-B3 Object description format

Our approach converts any images into a unified format for each object. This
format includes descriptions of the shape [27] and texture of the object.
Additionally, it includes a section named ”position,” which signifies that it
contains special information regarding the front view of the object. Within
this section, the coordinates for the center of the object are provided.

####  III-B4 Action output format

The output format comprises a continuous combination of x and y coordinates,
divided into two main components. The first is the inference process, which
delineates the reasoning steps leading to the final action prediction. The
second component is the action plan, which outlines the steps necessary to
accomplish tasks. Within the action plan, multiple plans may exist, each
defined by the following parameters: action_type (either ”pick_and_place” or
”sweep”), target_object (the ID of the object being targeted), rotation
(specifying the degree of rotation required for the target object), from
(numerical values indicating the starting position of interaction), and to
(indicating the ending position of interaction).

The coordinates in the output are from the front view, then it has to be
converted to the top view. We use a general mapping approach without any
training process.

####  III-B5 How to make reasons for each task manually

We generate step-by-step solutions expressed in natural language explanations
for each target task independently as human-to-human teaching happens. Table V
illustrates specific examples. Our focus primarily lies on achieving a high
success rate through natural-sounding reasoning; hence, we do not overly
emphasize the quality of the reasoning process. Typically, this process
involves the following components: defining the target task to facilitate its
decomposition, teaching object matching between the prompt and the scene,
incorporating additional reasoning steps for complex tasks, and providing a
specific action result as the final conclusion.

####  III-B6 LLMs

We employ GPT-3.5 as our primary experimental framework, supplemented by GPT-4
for additional experimentation. Utilizing their respective assistant APIs
facilitates the efficient provision of system prompts and inputs.

####  III-B7 Limitations

We remove task 9 ”twist” and task 8 ”novel_adj_and_noun” from our experiment
due to the limitation of our approach. For instance, our approach cannot
detect an exact rotation of objects for task 9(twist) task.

###  III-C Ablation study

Our approach focuses on the importance of reasoning with natural language.
Therefore, the ablation study is critical. As described in Fig.1, we remove
the reasoning part from our framework. In Table.II, the ”Without Chain of
Thought” (w/o CoT) condition indicates that the input of the Language Model
(LLM) lacks a manual Chain of Thought (CoT) example, yet the output of the LLM
includes CoT reasoning steps. This condition operates as a zero-shot reasoning
scenario.

###  III-D Result

Table.II and III show the success rate as a table for different
generalizations. Each of our models and tasks undergoes at least 30 attempts.
Table.IV shows an additional experiment with GPT4 only for task 10, the
follow-motion task. This additional experiment has 10 attempts only for our
approach.

TABLE II:  Placement generalization: Success rate comparison of our approach with existing approaches through ablation studies. The bold numbers indicate the average value. We compare our approach with VIMA (200M) [10], and Instruct2Act [6]. Our approach utilizes the model GPT-3.5-Turbo-1106 from OpenAI and is described with ablation studies (with CoT and without CoT input). ”w/o CoT” indicates an ablation study as described in Fig.1. The ”one-shot-example-task” column indicates which task is used as an in-context example for our approach. Each of our models and tasks undergoes at least 30 attempts. We remove task 9 (”twist”) due to the limitation of our approach.  Task Num | Task | One-shot example for Ours | VIMA 200M | Instruct2Act | Ours (w/o CoT) | Ours
---|---|---|---|---|---|---
1 | visual_manipulation | visual_manipulation | 100 | 91 | 93 | 100
2 | scene_understanding | scene_understanding | 100 | 81 | 60 | 67
3 | rotate | rotate | 100 | 98 | 93 | 93
4 | rearrange | rearrange | 100 | 79 | 52 | 73
5 | rearrange_then_restore | rearrange_then_restore | 57 | 72 | 25 | 73
6 | novel_adj | novel_adj | 100 | 82 | 13 | 43
7 | novel_noun | novel_noun | 100 | 88 | 8 | 80
11 | follow_order | follow_order | 77 | 72 | 0 | 0
12 | sweep_without_exceeding | sweep_without_exceeding | 93 | 68 | 17 | 47
15 | same_shape | same_shape | 97 | 78 | 10 | 80
16 | manipulate_old_neighbor | manipulate_old_neighbor | 77 | 64 | 8 | 20
17 | pick_in_order_then_restore | pick_in_order_then_restore | 43 | 85 | 10 | 30
|  |  | 87 | 80 | 32 | 59
TABLE III: Novel task generalization: Success rate comparison of our approach
with existing approaches through ablation studies. The bold numbers indicate
the average value. We compare our approach with VIMA (200M) [10], and
Instruct2Act [6]. Our approach utilizes the model GPT-3.5-Turbo-1106 from
OpenAI and is described with ablation studies (with CoT and without CoT
input). ”w/o CoT” indicates an ablation study as described in Fig.1. The ”one-
shot-example-task” column indicates which task is used as an in-context
example for our approach. Each of our models and tasks undergoes at least 30
attempts. We remove task 8 (”novel_adj_and_noun”) due to the limitation of our
approach.

Task Num | Task | One-shot example for Ours | VIMA 200M | Instruct2Act | Ours (w/o CoT) | Ours
---|---|---|---|---|---|---
10 | follow_motion | rearrange_then_restore | 0 | 35 | 0 | 12
13 | sweep_without_touching | sweep_without_exceeding | 0 | 0 | 0 | 3
14 | same_texture | same_shape | 95 | 80 | 3 | 71
|  |  | 32 | 38 | 1 | 29

TABLE IV:  Novel task generalization: Success rate comparison for additional
experiment with GPT4 only for task 10, follow-motion task.

Task Num | Task | One-shot example for Ours | VIMA 200M | Instruct2Act | Ours (GPT3.5) | Ours (GPT4)
---|---|---|---|---|---|---
10 | follow_motion | rearrange_then_restore | 0 | 35 | 12 | 90

##  IV Discussion

We hypothesized the natural language reasoning process of whole robotics
action planning is critical to removing the limitation of existing LLM-based
code generation approach and efficiently describing robotics skills for LLMs.
Overall, our quantitative results demonstrate that the natural language
reasoning process has a critical role in performing a better success rate,
especially for novel-task generalization. On the other hand, our approach does
not perform better than other existing approaches in most tasks.

Importance of Chain of Thought reasoning: According to Table. II and III,
Chain-of-Thought reasoning is crucial for almost all tasks as our hypothesis
mentions. The success rate improves from 32% to 59% in placement
generalization (Table.II) with CoT reasoning. This result demonstrates that a
natural language reasoning step is a critical component in producing low-level
action prediction, especially for novel task generalization (Table.III).

Skill extraction for novel task: Our results suggest that the explicitness of
whole robotics action planning can have the potential to transfer robotics
skills from known to novel tasks with LLMs. To tackle the novel-task
generalization of VIMABench, it is required to extract essential skills from
in-context examples. Our ablation study suggests that the natural language
reasoning process efficiently contributes to novel tasks. Especially for the
follow-motion task, GPT4 performs 90% in extracting skills from the rearrange-
and-restore task to solve the follow-motion task as described in Table.IV. On
the other hand, our approach still struggles to solve task 13 (sweep-without-
touching) which requires understanding a novel concept of collision avoidance.

Numerical values for LLMs: We demonstrate that LLM can effectively handle
numerical values for most tasks. Although both the input and output of our
approach include numerical values, LLM successfully generates action plans
containing coordinates. However, our approach still struggles to solve task
11, a follow-order task that requires how objects are stacked based on their
coordinates. Additionally, We show that LLM has spatial understanding when the
coordinates of each object are injected as an input. Our results of task
12(sweep-without-exceeding), suggest that LLM can handle numerical values of
actions that can not be described with object ID. CaP [14] mentioned that
code-based reasoning outperforms natural language CoT for spatial-geometric
reasoning, however, our study provide novel grounding approach for LLMs not
only for geometric reasoning but also planning itself.

Multi-steps action prediction: Task 5 (Table.II) suggests that the CoT
reasoning process can support long-step action prediction effectively. Tasks 4
and 5 (Table.II) require multi-step actions where the target object’s location
changes in each step. Since our approach does not predict actions in an
autoregressive manner, all actions must be predicted simultaneously.

Open problems: We have identified several challenges in our current approach.
Firstly, there is the limited capability of text description capability of
objects. Extracting nuanced information such as object height, rotation, or
darkness solely using a Vision-Language model, rather than a task-specific
approach, is particularly difficult. Secondly, there’s the matter of feedback.
The current Language-Logic Model (LLM) operates a robot within a closed loop,
lacking the capacity to integrate feedback from its performance. Thirdly, our
approach is limited in its ability to generate complex actions, currently only
capable of producing tabletop actions. Fourthly, our approach focuses on
semantic capability for planning problems, whereas robotic action planning
typically necessitates algorithmic capability as well.

##  V Conclusion

Robotics action planning presents a challenging topic due to issues such as
limited pre-defined skills, and novel task generalizations. We introduce a
novel concept wherein planning occurs solely within a natural language
framework, conferring advantages over conventional LLM-based code-generation
methodologies. Our quantitative results demonstrate that our approach does not
consistently outperform other existing approaches across various tasks,
however, it shows considerable potential. Importantly, our approach
underscores the capability to tackle novel tasks with known skill sets. In
future endeavors, we envision applying our approach to diverse tasks and
situaitons leveraging a flexible reasoning capability of our approach, and
investigating novel task generalization.

## APPENDIX

Table.V shows a natural language reasoning we manually made for in-context
examples.

TABLE V: Specific example of natural language reasoning we manually made for
in-context learning. We do not have any specific format to produce these
reasonings and we try to make natural reasoning in a human way.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Nicolas Bach, Andrew Melnik, Malte Schilling, Timo Korthals, and Helge Ritter.  Learn to move through a combination of policy gradient algorithms: Ddpg, d4pg, and td3.  In Machine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19–23, 2020, Revised Selected Papers, Part II 6, pages 631–644. Springer, 2020.
  * [3] Shivansh Beohar and Andrew Melnik.  Planning with rl and episodic-memory behavioral priors.  arXiv preprint arXiv:2207.01845, 2022.
  * [4] Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, and He Wang.  Robogpt: an intelligent agent of making embodied long-term decisions for daily instruction tasks.  arXiv preprint arXiv:2311.15649, 2023.
  * [5] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023.
  * [6] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li.  Instruct2act: Mapping multi-modality instructions to robotic actions with large language model.  arXiv preprint arXiv:2305.11176, 2023.
  * [7] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022.
  * [8] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.  Inner monologue: Embodied reasoning through planning with language models.  arXiv preprint arXiv:2207.05608, 2022.
  * [9] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.  Bc-z: Zero-shot task generalization with robotic imitation learning.  In Conference on Robot Learning, pages 991–1002. PMLR, 2022.
  * [10] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  Vima: General robot manipulation with multimodal prompts, 2023.
  * [11] Teyun Kwon, Norman Di Palo, and Edward Johns.  Language models as zero-shot trajectory generators, 2023.
  * [12] Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.  Chain of code: Reasoning with a language model-augmented code emulator, 2023.
  * [13] Jiachen Li, Qiaozi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, et al.  Mastering robot manipulation with multimodal prompts through pretraining and multi-task fine-tuning, 2023.
  * [14] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as policies: Language model programs for embodied control.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023.
  * [15] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2motion: from natural language instructions to feasible plans.  Autonomous Robots, 47(8):1345–1365, November 2023.
  * [16] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [17] Federco Malato, Florian Leopold, Andrew Melnik, and Ville Hautamaki.  Zero-shot imitation policy via search in demonstration dataset.  arXiv preprint arXiv:2401.16398, 2024.
  * [18] Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, and Andrew Melnik.  Behavioral cloning via search in video pretraining latent space.  arXiv preprint arXiv:2212.13326, 2022.
  * [19] Neil Houlsby Matthias Minderer, Alexey Gritsenko.  Scaling open-vocabulary object detection.  NeurIPS, 2023.
  * [20] Andrew Melnik, Michael Büttner, Leon Harz, Lyon Brown, Gora Chand Nandi, Arjun PS, Gaurav Kumar Yadav, Rahul Kala, and Robert Haschke.  Uniteam: Open vocabulary mobile manipulation challenge.  arXiv preprint arXiv:2312.08611, 2023.
  * [21] Andrew Melnik, Luca Lach, Matthias Plappert, Timo Korthals, Robert Haschke, and Helge Ritter.  Using tactile sensing to improve the sample efficiency and performance of deep deterministic policy gradients for simulated in-hand manipulation tasks.  Frontiers in Robotics and AI, 8:538773, 2021.
  * [22] Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, Sharada Mohanty, Byron Galbraith, Ke Chen, Yan Song, Tianze Zhou, et al.  Towards solving fuzzy tasks with human feedback: A retrospective of the minerl basalt 2022 competition.  arXiv preprint arXiv:2303.13512, 2023.
  * [23] Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, and et al.  Robocodex: Multimodal code generation for robotic behavior synthesis, 2024.
  * [24] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  Advances in Neural Information Processing Systems, 36, 2024.
  * [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [26] Krishan Rana, Andrew Melnik, and Niko Sünderhauf.  Contrastive language, action, and state pre-training for robot learning, 2023.
  * [27] Markus Rothgaenger, Andrew Melnik, and Helge Ritter.  Shape complexity estimation using vae.  In Intelligent Systems Conference, pages 35–45. Springer, 2023.
  * [28] Malte Schilling and Andrew Melnik.  An approach to hierarchical deep reinforcement learning for a decentralized walking control architecture.  In Biologically Inspired Cognitive Architectures 2018: Proceedings of the Ninth Annual Meeting of the BICA Society, pages 272–282. Springer, 2019.
  * [29] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models, 2022.
  * [30] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [31] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models, 2021.
  * [32] Huaxiaoyue Wang, Gonzalo Gonzalez-Pumariega, Yash Sharma, and Sanjiban Choudhury.  Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of-thought, 2023.
  * [33] Yen-Jen Wang, Bike Zhang, Jianyu Chen, and Koushil Sreenath.  Prompt a robot to walk with large language models, 2023.
  * [34] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * [35] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.
  * [36] Takuma Yoneda, Jiading Fang, Peng Li, Huanyu Zhang, Tianchong Jiang, Shengjie Lin, Ben Picker, David Yunis, Hongyuan Mei, and Matthew R Walter.  Statler: State-maintaining language models for embodied reasoning.  arXiv preprint arXiv:2306.17840, 2023.
  * [37] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  arXiv preprint arXiv:2204.00598, 2022.