# TIC: Translate-Infer-Compile for accurate “text to plan” using
LLMs and logical intermediate representations

Sudhir Agarwal &Anu Sreepathy Intuit AI Research
Mountain View, CA, USA.
{sudhir_agarwal<EMAIL_ADDRESS>
###### Abstract

We study the problem of generating plans for given natural language planning
task requests. On one hand, LLMs excel at natural language processing but do
not perform well on planning. On the other hand, classical planning tools
excel at planning tasks but require input in a structured language such as the
Planning Domain Definition Language (PDDL). We leverage the strengths of both
the techniques by using an LLM for generating the PDDL representation (task
PDDL) of planning task requests followed by using a classical planner for
computing a plan. Unlike previous approaches that use LLMs for generating task
PDDLs directly, our approach comprises of (a) translate: using an LLM only for
generating a logically interpretable intermediate representation of natural
language task descriptions, (b) infer: deriving additional logically dependent
information from the intermediate representation using a logic reasoner
(currently, Answer Set Programming solver), and (c) compile: generating the
target task PDDL from the base and inferred information. We observe that using
an LLM to only output the intermediate representation significantly reduces
LLM errors. Consequently, TIC approach achieves, for at least one LLM, high
accuracy on task PDDL generation for all seven domains of our evaluation
dataset.

##  1 Introduction

Figure 1: An overview of planning based response generation

Customers of large organizations have a variety of questions or requests
(collectively known as queries in the following) pertaining to the
organization’s domain of operation. Providing accurate responses to such user
queries requires a thorough analysis of the user’s context, product features,
domain knowledge, and organizational policies. Certain user queries such as
how-to questions and state changing requests require dynamic composition of
possible actions (i.e., planning). Recently, transformer-based large language
models (LLMs) have shown wide success on many natural language understanding
and translation tasks, also demonstrating some general reasoning and planning
capability on diverse tasks without having to be retrained Ahn et al. (2022);
Huang et al. (2022); Zeng et al. (2022). However, LLMs are known to perform
only shallow reasoning and cannot find complex plans Valmeekam et al. (2023b,
a); OpenAI (2023). On the other hand, classical planners can scale better in
the number of actions they support without compromising on the accuracy
Sohrabi (2019); Helmert (2006b).

We address the problem of responding to planning related queries by combining
the strengths of LLMs and classical planners. Figure 1 illustrates a high
level overview of this process. The work presented in Liu et al. (2023) has
shown that, for classical planning tasks, using an LLM to generate the task
PDDL from a natural language planning task description and then using an
external classical planner to compute a plan yields better performance than
relying solely on an LLM for end to end planning. However, their approach,
which is based on in-context examples, does not achieve acceptable accuracy,
in particular, for slightly complex planning domains. One of the primary
reasons for failure is that the LLM often makes errors generating information
that must abide by the constraints specified in the domain knowledge or the
task description.

[htb]  : Example barman task description and task PDDL You have 1 shaker with
3 levels, 5 shot glasses, 3 dispensers for 3 ingredients. The shaker and shot
glasses are clean, empty, and on the table. Your left and right hands are
empty. The first ingredient of cocktail1 is ingredient2. The second ingredient
of cocktail1 is ingredient1. The first ingredient of cocktail2 is ingredient1.
The second ingredient of cocktail2 is ingredient2. The first ingredient of
cocktail3 is ingredient1. The second ingredient of cocktail3 is ingredient3.
The first ingredient of cocktail4 is ingredient3. The second ingredient of
cocktail4 is ingredient2. Your goal is to make 4 cocktails. shot1 contains
cocktail1. shot2 contains cocktail4. shot3 contains cocktail3. shot4 contains
cocktail2.

    (define (problem prob) (:domain barman)
     (:objects
      shaker1 - shaker
      shot1 shot2 shot3 shot4 shot5 - shot
      ...
    )(:init
     (clean shaker1) (clean shot1) (clean shot2)
     (clean shot3) (clean shot4) (clean shot5)
     ...
     (cocktail_part1 cocktail1 ingredient2)
     ...
     (cocktail_part2 cocktail4 ingredient2)
    )(:goal (and
      (contains shot1 cocktail1)
      (contains shot2 cocktail4)
      (contains shot3 cocktail3)
      (contains shot4 cocktail2))))

Throughout this paper, we use the seven planning domains presented in Liu et
al. (2023). The natural language task descriptions of each domain along with
the domain model PDDL can be found at https://github.com/Cranial-XIX/llm-pddl.
Task PDDLs for these and several other planning domains can be generated using
the collection presented in Seipp et al. (2022).

For example, consider the planning task description in Figure 1 from a well-
known planning domain - barman. It contains cardinality information of shots
(5 shot glasses), but mentions only 4 shot instances explicitly. However, the
correct task PDDL should contain 5 shot instances. An LLM does not always
abide by this relationship. Similarly, the statement in task description that
all shots are clean should lead to generation of PDDL atoms (clean shot$i$)
with $1\leq i\leq 5$. But, an LLM sometimes generates only 4 such atoms (one
for each mentioned shot instance).

Such logical dependency rules in planning domains are deterministic and can be
computed in an error-free manner with appropriate logic reasoners to augment
the information extracted by LLMs. The main idea underlying our approach is to
combine the best of both these technologies (LLMs and logical reasoning) to
achieve more accurate results than possible using either of them alone. More
precisely, we break down the problem of task PDDL generation into the
following three steps (see also Figure 3):

  1. 1.

Translate with an LLM, the natural language planning task description to a
logical language with sufficient expressiveness to model facts, rules and
constraints. We call such a translation an intermediate representation of the
natural language planning task description.

  2. 2.

Infer, with a reasoner for the chosen logical language, the rest of the
required information using the intermediate representation and domain
knowledge rules. We call the union of facts in the intermediate representation
and the inferred facts as the materialized representation.

  3. 3.

Compile the target task PDDL using the materialized representation and domain
PDDL.

Consequently, we refer to our approach as Translate-Infer-Compile (TIC)
approach. Section 3 presents details of the three steps of TIC. Section 4
dives deeper into the translate step and presents two techniques for
generating intermediate representation of natural language planning task
descriptions. Our experimental results (presented in Section 5) show that  1.
TIC supports more complex task descriptions than other intermediate
representation based approaches.  2. TIC achieves high accuracy (100% in some
cases) on all seven planning domains of the evaluation dataset presented in
Liu et al. (2023).  3. TIC is also more general as it achieves high accuracy
(near 100% in some cases) on the same dataset but with variation in language
of the task descriptions.

: TIC pipeline for generating task PDDLs from NL task descriptions

##  2 Preliminaries and Related Work

In this section, we provide a brief overview of relevant background material
and prior work related to responding to natural language planning task
requests.

###  2.1 Classical Planning and PDDL

Given a description of the possible initial states of the world, a description
of the desired goals, and a description of a set of possible actions, the
planning problem involves synthesizing a plan that, when applied to any
initial state, generates a state which contains the desired goals (goal state)
Ghallab et al. (2004). The planning domain definition language (PDDL) serves
as a standardized encoding of classical planning problems Ghallab et al.
(1998); Haslum et al. (2019). The PDDL representation of a planning problem is
typically separated into two files: a domain file and a task file, both of
which become inputs to the planner. The domain PDDL file includes a set of
predicates that define a state space, and a set of possible actions with their
preconditions and effects. The task PDDL file provides a list of objects to
ground the domain, the problem’s initial state and goal conditions.

###  2.2 Planning with LLMs

Some recent works investigate prompting and fine-tuning approaches to enable
LLMs to solve PDDL planning problems. For example, Silver et al. (2022)
presents an approach of prompting pre-trained LLMs with few-shot examples. The
approach presented in Guan et al. (2023) employs LLMs as an interface between
PDDL and sources of corrective feedback, such as PDDL validators and humans.
Zhou et al. (2023) introduces ISR-LLM, a framework that improves LLM-based
planning through an iterative self-refinement process. Plansformer presented
in Pallagani et al. (2022) uses an LLM pretrained on code generation and
further trains it on planning problems and corresponding valid plans.
Frameworks such as LangChain Chase (2022) enable developers to combine LLMs
with external tools or APIs. Such frameworks depend on LLMs for selecting and
composing tools and do not scale well beyond a small set of tools. Other
frameworks such as Gorilla Patil et al. (2023) scale to a large number of APIs
but require fine-tuning of LLMs and have limited planning capability. Although
these works that rely on LLMs’ planning capabilities achieve promising
outcomes, the state of the art classical planning tools such as Pyperplan
Alkhazraji et al. (2020) and Fast Downward Planner Helmert (2006a) remain far
more efficient than LLM based end-to-end plan generation in terms of wall-
clock time and accuracy of plan generation. Due to the limited correctness of
LLMs in generating plans, particularly in the presence of a large number of
possible actions, our approach aims at leveraging an external planner for the
plan generation task.

###  2.3 Planner Input Generation using LLMs

Xie et al. (2023) show that LLMs are more adept at translation than planning,
in particular, by leveraging commonsense knowledge and reasoning to furnish
missing details from under-specified goals (as is often the case in natural
language). With this, LLMs can translate natural language task descriptions to
input formats required by an external planner. Rana et al. (2023) introduces
SayPlan, an approach to LLM-based large-scale task planning for robotics using
3D scene graph (3DSG) representations. SayPlan achieves scalability by
exploiting the hierarchical nature of 3DSGs and using a classical path
planner. The approach in Lyu et al. (2023) presents preliminary results of
integrating LLMs with PDDL using the SayCan dataset Ahn et al. (2022).
However, the SayCan dataset has a limited scope, as it contains only three
predefined actions. The LLM+P approach presented in Liu et al. (2023) uses an
LLM to translate natural language planning task description to task PDDL.
However, as mentioned in Section 1, the drop in accuracy seen in LLM+P
approach is due to an imprecise generation of logically dependent information
in the task PDDL by the LLMs.

###  2.4 Two-Step Generation of Planner Input

To achieve greater consistency and coherence in sentence generation for
textual problems, Nye et al. (2021) proposes decomposing the generation
process into two parts. This entails having an LLM generate candidate
sentences, representing system 1 thinking, and a logical pruning process
implemented via a separate symbolic module, representing system 2 thinking.

The approach presented in Yang et al. (2023) is similar to our approach as it
also uses an LLM for translating a user query to a structured representation
and then uses domain rules (called Knowledge Modules) on the LLM output to
deterministically infer more information. It aims at directly computing the
answer to the query over the provided information. However, the approach has
been shown to support only simple input sentences that translate to ASP facts.
Our approach, on the other hand, can support complex planning scenarios with
complex data-structures and rules. This is achieved by using a set of rules
for translating natural language sentences describing cardinality constraints,
facts, rules, bijective relationships and two-dimensional grids to ASP.
Furthermore, the former approaches have limited applicability for industrial
use cases as it is often not feasible to send proprietary data to an LLM or
load entire databases into a knowledge module. In contrast, we aim at
generating a representation that can be used by an appropriate tool to
efficiently compute the final answer. Specifically, in case of planning
problems, we aim at generating task PDDL that can be sent to a planner to
obtain the final plan.

##  3 Translate Infer Compile Approach

In this section, we present the main steps of the Translate-Infer-Compile
(TIC) approach. The approach is flexible with respect to the logical language
used and can work with any logical language that is expressive enough to
semantically represent the information present in user requests of our
interest. In particular, the logical language should support primitive
datatypes, object types, objects, relationships between objects, cardinality
and logical constraints, and inference rules. In this paper, we use Answer Set
Programming (ASP) Gelfond and Lifschitz (1988); Lifschitz (2008) as the
logical language. We use the example task description in Figure 1 as a running
example to explain the TIC approach.

###  3.1 ASP Schema for Task PDDLs

We first define the ASP schema for modeling the information in task
descriptions (as intermediate representations) as well as that required in
target task PDDLs (as materialized representations). In the following $p/N$
stands for a predicate $p$ with arity (i.e., number of arguments) $N$.

#### object/2

predicate relates objects with their respective types. E.g., a fact
object(cocktail1, cocktail) means that cocktail1 is an object of type
cocktail.

#### cardinality/2

predicate represents the cardinality of the set of objects of a type. E.g.,
cardinality(shot, 5) means that there are five shot glasses. Such cardinality
information is often present in task descriptions whereas the task PDDL does
not include the cardinality information but only the set of objects that
satisfy the cardinality constraint. The cardinality rules below generate new
objects of the given type when necessary to satisfy the cardinality
constraints, where make_id is a predicate external to an ASP reasoner.

    {object(@make_id(X, T), T)} :-
        cardinality(T, N), X = 1..N.
    :- #count {X: object(X, T)} = M,
        cardinality(T, N), M != N.

For example, given cardinality(shot, 5), object(shot1, shot), object(shot2,
shot), object(shot3, shot), and object(shot4, shot), the cardinality rules
would generate the new fact object(shot5, shot).

#### init/1 and goal/1

predicates differentiate initial state facts from goal state facts. All domain
predicates defined in the domain.pddl, the map/3 predicate (described below)
and the grid predicates (described below) may occur as arguments of init and
goal. A task description describes the init and the goal states as a set of
facts that are true in the respective states. For example, an initial state
sentence shot3 is on the table is represented as init(ontable(shot3)), and a
goal state sentence shot3 should be on the table is represented as
goal(ontable(shot3)).

#### map/3

predicate is used as a shorthand for a bijective relationship between two
objects types (of the same cardinality). For example, map(dispenser,
dispenses, ingredient) stands for dispenses(dispenser1, ingredient1), …,
dispenses(dispenserN, ingredientN), where N is the cardinality of type
dispenser as well as of type ingredient. The rules below expand a map
shorthand to corresponding facts of the initial state (analogous rules for the
goal state) where make_fact is a predicate external to an ASP reasoner.

    1 {init(@make_fact(X,P,Y)) : object(X, T1)} 1
        :- init(map(T1,P,T2)), object(Y, T2).
    1 {init(@make_fact(X,P,Y)) : object(Y, T2)} 1
        :- init(map(T1,P,T2)), object(X, T1).

#### Two-dimensional Grid

Some planning domains require capturing the positions of certain objects on a
two-dimensional grid. For example, in the floortile domain, a tile tile_x_y
means that the tile is on row x and column y. We represent such information
with a predicate grid/3 where grid is a placeholder for the appropriate grid
name. For example, floortile_grid(x, y, tile_x_y) represents the position of
floortile tile object tile_x_y.

###  3.2 Intermediate Representation

The core idea of the TIC approach is the introduction of the intermediate
representation. An intermediate representation of a query is a logically
interpretable representation of the query. It serves multiple purposes
(described below) to address the accuracy bottleneck of LLMs when generating
the task PDDL directly.

Unlike the in-context examples of direct PDDL generation approaches, an
intermediate representation does not contain information that is not present
in the task description but required in the task PDDL. In some cases, the
missing information can be inferred from the information in intermediate
representation. For example, if a barman task description states that there
are 5 shots and 3 dispensers as well as mentions shot1, shot2, shot3, and
shot4, then the intermediate representation should not contain inferable
information shot5, dispenser1, dispenser2, and dispenser3. In other cases, the
missing information can be added with the help of rules derived from the
domain knowledge. For example, a barman task description may not contain
information corresponding to the PDDL facts (next l0 l1), (next l1 l2),
(shaker_empty_level shaker1 l0), and (shaker_level shaker1 l0) because the
knowledge that the shaker levels have an order is domain knowledge that is
common for all barman queries.

An intermediate representation bridges the syntactic and semantic
heterogeneity between the task description and target task PDDL. For example,
each of the phrases shot three, Shot 3, and third shot should be represented
as shot3. Sometimes a given domain.pddl does not have semantically meaningful
object types and predicate names. For example, the blocksworld domain does not
define the object type block but requires the blocks to be instances of the
generic object type object. In such cases, an intermediate representation uses
the object types and predicates that are closer to natural language than to
those of the target PDDL.

Some sentences in the task description are more naturally represented
intensionally as a rule rather than extensionally by enumerating all implied
facts. For example, the phrase 3 dispensers for 3 ingredients. is more
directly represented as the shortcut map(dispenser, dispenses, ingredient)
rather than explicitly enumerating all the facts. In another example, the
initial state sentence The shaker and shot glasses are clean is represented
with the following rules rather than with all implied facts.

    init(clean(X)) :- object(X, shaker).
    init(clean(X)) :- object(X, shot).

As a result, when an LLM generates an intermediate representation of a task
description, the LLM output is more often correct. The LLM output is also
shorter and doesn’t grow if the query contains a lot of derived information.
In addition, we require shorter in-context examples which help in keeping the
prompt size small. In Section 4 we present two approaches for translating
natural language planning task descriptions to intermediate representations.

###  3.3 Infer Materialized Representation

An intermediate representation contains only the information present in a task
description. We compute the remaining information required in the task PDDL
deterministically using an ASP solver. A materialized representation is the
union of the facts in the intermediate representation and the inferred facts.
In this section, we present different kinds of ASP rules that may be relevant
for inferring information in a planning domain. Note that not all kind of
rules may be required for every planning domain.

1\. Rules to infer the type of objects that need to be declared in the task
PDDL. For object types that are arguments of a predicate, we can automatically
generate the rules for deriving the object types from the predicate
definitions in domain.pddl. For example, from the predicate definition
(cocktail_part1 ?c - cocktail ?i - ingredient) we can generate the rules

    object(X, cocktail):- cocktail_part1(X, _).
    object(X, ingredient):- cocktail_part1(_, X).

In cases where there is no predicate to infer a certain object type, one has
to add the corresponding rules manually. This case doesn’t occur in the barman
domain. However, in the blocksworld domain, the task PDDL requires the blocks
to be of type object, while they are of type block in the intermediate
representation. Therefore, in the blocksworld domain, we require a rule
object(X, object) :- object(X, block). to assign the type object to each
block.

The following two kinds of rules can be authored semi-automatically based on
domain knowledge that may already be present in the domain.pddl or some
documentation, or with the help of domain experts.

2\. Rules to infer information derivable from other information present in the
query. For example, for the floortile domain, the rule init(clear(T)) :-
object(T, tile), not init(robot_at(_, T)) infers that a tile is clear if there
is no robot at that tile.

3\. Rules to infer information that is not derivable from information present
in the query. For example, in the barman domain, the task PDDL requires the
order of shaker levels known through domain knowledge. The following rules

    init(@make_seq(N-1, l, next, 1)) :-
        cardinality(level, N).
    first_level(X) :- object(X, level),
        not init(next(_, X)).

generate the facts init(next(level0, level1)), init(next(level1, level2)), and
first_level(level0) for N=3, where make_seq is a predicate external to ASP for
generating a sequence. See Appendix C for complete listing of rules for all
seven planning domains considered in this work.

###  3.4 Compile Target Representation

In the last step, we generate the task PDDL. The materialized representation
contains all the information necessary in a task PDDL. So, this step is mostly
a straightforward syntactic transformation of facts from the ASP Logic Program
syntax to the PDDL syntax. In principle, this step can also be performed using
an LLM. But, since the translation process is deterministic and simple, it is
less likely to encounter errors when performed through code. The code based
transformation consists of three main steps:

  1. 1.

For all bindings (obj1, t), …, (objn, t) in the answer of the query object(X,
T), if object type t is defined in the domain.pddl, generate a PDDL string
”obj1 … objn - t”. The concatenation of such strings makes the objects section
of the task PDDL.

  2. 2.

For all bindings p(obj1, …, objn) of X in the answer to the query init(X), if
the predicate p is defined in the domain.pddl, then generate a PDDL string ”(p
obj1 … objn)”. The concatenation of such strings makes up the init section of
the task PDDL.

  3. 3.

Similar to the previous step, generate the PDDL string for all bindings of X
for the query goal(X). The string representing the conjunction of all such
strings makes up the goal section of the task PDDL.

##  4 Translating NL Task Description to Intermediate Representation using LLM

In this section, we present two approaches for generating the intermediate
representation of a given task description. The first approach uses an in-
context example for each domain whereas the second approach uses a generic
prompt that works for all domains.

###  4.1 Using In-Context Domain Examples

LLMs are known for their proficiency in learning in context without the need
for fine-tuning their parameters. In-context learning in LLMs is their ability
to handle unseen tasks using only a few demonstrations represented as input-
label pairs Brown et al. (2020). In the in-context example based approach, we
use an LLM with a domain specific in-context example to generate the
intermediate representation of a given task description. The LLM prompt
consists of an example task description, its intermediate representation, and
the new task task description. We refer to this approach as TIC-IC. (See
Appendix A for the complete LLM prompt). Figure 4.1 shows the LLM generated
intermediate representation of the example task description in Figure 1.

[htb]  : Intermediate representation of the example task description in Figure
1

    cardinality(shaker, 1).
    cardinality(level, 3).
    cardinality(shot, 5).
    cardinality(dispenser, 3).
    cardinality(ingredient, 3).
    init(clean(X)) :- object(X, shaker).
    init(clean(X)) :- object(X, shot).
    ...
    init(handempty(left)).
    init(handempty(right)).
    init(map(dispenser, dispenses, ingredient)).
    init(cocktail_part1(cocktail1,ingredient2)).
    ...
    init(cocktail_part2(cocktail4,ingredient2)).
    goal(contains(shot1, cocktail1)).
    ...
    goal(contains(shot4, cocktail2)).

###  4.2 Using Generic Prompts

The approach based on in-context examples achieves high accuracy but requires
an in-context example for each domain. This can be a bottleneck in use cases
that need to support many, possibly previously unknown, types of user queries.
In this section, we present a domain independent generalized prompt that
addresses this limitation. The approach consists of the three steps:  1.
Cardinality extraction  2. Named objects extraction  3. Rules extraction  .

#### Cardinality Extraction:

This step extracts the explicitly mentioned cardinalities of object types in
the given task description. The LLM prompt for for this step consists of
instructions for  1. splitting the task description into initial state and
goal state, and  2. extracting for each object type $T$ the count of $T$ that
is explicitly mentioned in the initial state description  (see Appendix B for
complete prompt). The set of provided object types includes a name and a short
description for each domain object type.

For the example task description in Figure 1, the LLM generates the following
output:

    {shaker: 1, level: 3, shot: 5, dispenser: 3,
    hand: 2, ingredient: 3}

#### Named objects extraction:

This step extracts the explicitly mentioned named objects for each object type
in the given task description. The LLM prompt for this step consists of
instructions for detecting and formatting object instances (see Appendix B for
complete prompt) . The set of provided object types is the same as that in the
previous step.

For the example task description in Figure 1, the LLM generates the following
output:

    {shaker: [], level: [], shot: [shot1, shot2,
     shot3, shot4], dispenser: [], hand: [left,
     right], cocktail: [cocktail1, cocktail2,
     cocktail3, cocktail4], ingredient:
     [ingredient1, ingredient2, ingredient3]}

#### Rules extraction:

In this step the LLM generates a Logic Program representation of the given
task description. The LLM prompt consists of steps to detect the initial and
goal states, translate each state description per state translation rules, and
output the result as syntactically correct ASP Logic Program. The LLM is
instructed to use both the provided set of extracted objects as well as the
set of domain predicates. The latter consists of the predicate’s name, arity,
argument types, and a brief description. (see Appendix B for example object
types and predicates). The following describe the state translation rules
where <state> is a placeholder for init or goal.

  1. 1.

a property of all instances of an object type. E.g., All apples are clean
should be translated as <state>(clean(X)) :- object(X, apple).

  2. 2.

a property of a named object. E.g., apple1 is red should be translated as
<state>(red(apple1)).

  3. 3.

a relationship between two or more named objects. E.g., apple1 is on plate2
should be translated as <state>(on(apple1, plate2)).

  4. 4.

a relationship between two or more unnamed objects. E.g., cups’ handles are
free should be translated as <state>(free(X, Y)) :- cup_handle(X, Y).

  5. 5.

a relationship between all unnamed objects of an object type with a named
object. E.g., apples are on table1 should be translated as <state>(on(X,
table1)) :- object(X, apple).

  6. 6.

a bijective relationship between instances of two object types of the same
cardinality. E.g., 3 balls for 3 kids should be translated as
<state>(map(ball, play, kid)).

  7. 7.

a 2 dimensional grid of cells with coordinates. E.g. cell_1_1, cell_1_2,
cell_2_1, cell_2_2 should be translated as <state>(cell_grid(1, 1, cell_1_1)),
…, <state>(cell_grid(2, 2, cell_2_2)).

For the example task description in Figure 1, the LLM generates the same
intermediate representation as in Figure 4.1 except the cardinality
information.

The steps for extracting cardinalities, named objects and rules can be
performed with three separate LLM calls (referred to as TIC-G3), or together
in one LLM call (referred to as TIC-G1). While extracted named objects are
programmatically passed on to the next prompt for rule extraction in TIC-G3,
in TIC-G1, the LLM is instructed to extract named objects and use them for
rule extraction. Refer to Appendix B for complete LLM prompts of both
implementation options.

| GPT-4 | GPT 3.5 | PaLM2
---|---|---|---
Domain | LLM+P | TIC-IC | TIC-G3 | TIC-G1 | LLM+P | TIC-IC | TIC-G3 | TIC-G1 | LLM+P | TIC-IC | TIC-G3 | TIC-G1
Barman | 100 | 100 | 100 | 98.33 | 98.33 | 100 | 0 | 0 | 75 | 100 | 100 | 80
Blocksworld | 88.33 | 100 | 100 | 100 | 71.67 | 100 | 91.67 | 93.33 | 10 | 100 | 100 | 100
Floortile | 0 | 100 | 100 | 100 | 26.67 | 100 | 100 | 68.33 | 0 | 90 | 90 | 60
Grippers | 95 | 100 | 100 | 98.33 | 95 | 100 | 40 | 36.67 | 95 | 100 | 95 | 95
Storage | 40 | 100 | 100 | 98.33 | 25 | 100 | 0 | 0 | 0 | 100 | 60 | 6.67
Termes | 53.33 | 100 | 98.33 | 98.33 | 71.67 | 100 | 15 | 63.33 | 0 | 100 | 93.33 | 0
Tyreworld | 71.67 | 100 | 95 | 95 | 55 | 100 | 8.33 | 0 | 35 | 100 | 0 | 0
Table 1:  % Accuracy of PDDL generation on LLM+P dataset averaged over 3 runs (variance $\leq 1$) | GPT-4 | GPT 3.5 | PaLM2
---|---|---|---
Domain | LLM+P | TIC-IC | TIC-G3 | TIC-G1 | LLM+P | TIC-IC | TIC-G3 | TIC-G1 | LLM+P | TIC-IC | TIC-G3 | TIC-G1
Barman | 100 | 100 | 100 | 96.67 | 53.33 | 100 | 0 | 0 | 0 | 100 | 66.67 | 66.67
Blocksworld | 100 | 100 | 100 | 100 | 51.67 | 95 | 55 | 63.33 | 0 | 100 | 95 | 95
Floortile | 0 | 100 | 100 | 100 | 0 | 100 | 83.33 | 48.33 | 0 | 90 | 90 | 45
Grippers | 0 | 100 | 96.67 | 96.67 | 0 | 90 | 15 | 35 | 0 | 100 | 60 | 60
Storage | 0 | 100 | 93.33 | 68.33 | 0 | 80 | 0 | 0 | 0 | 100 | 50 | 0
Termes | 0 | 100 | 100 | 96.67 | 5 | 100 | 0 | 21.67 | 0 | 100 | 65 | 0
Tyreworld | 0 | 100 | 93.33 | 88.33 | 0 | 90 | 0 | 0 | 0 | 95 | 0 | 0
Table 2:  % Accuracy of PDDL generation on Language Variation dataset averaged
over 3 runs (variance $\leq 1$)

##  5 Experiments

Through experiments we address the following questions. 1. Does introducing an
intermediate representation and a logical reasoner improve the accuracy of
PDDL generation? 2. Does the TIC approach generalize to language variations in
the natural language task description? 3. How does the LLM perform with
generic translation instructions for intermediate representation?

###  5.1 Experiment setup

We evaluate our TIC approach on seven planning domains that have been taken
directly from LLM+P Liu et al. (2023) and are standard planning benchmarks.
Each of the domains include a domain description PDDL and 20 automatically
generated planning tasks from Seipp et al. (2022), with each task consisting
of a natural language description and a corresponding ground truth PDDL. For
evaluating the in-context example based approach, we also include an example
task description and a corresponding intermediate representation that serves
as the in-context example for each domain. In addition, we also created a new
dataset derived from the LLM+P dataset by introducing language variation in
each of the natural language task descriptions for the seven domains. We used
GPT-3.5 to paraphrase the original task descriptions and call this the
Language Variation dataset. We evaluate the TIC approach on the Language
Variation dataset without modifying the in-context learning examples or the
generic prompt. We use Clingo Gebser et al. (2019) python
package111https://pypi.org/project/clingo/ as the ASP solver. For each of the
tasks, we generate the task PDDL and compare it with the ground truth PDDL
using an automated process. Note that the task PDDL comparison algorithm does
not count the number of mismatches but determines whether two PDDLs are
equivalent or not (see Appendix D for details). We solely present the PDDL
generation accuracy since plan generation using a classical planner is
deterministic and precise provided that the PDDL is accurate for the given
dataset. Below are details of the different experiments conducted. We run the
experiments on GPT-4, GPT-3.5 Turbo and PaLM2 (chat-bison) models.

  * •

LLM+P : Evaluating the LLM+P approach of directly generating task PDDL using
an LLM, as a baseline

  * •

TIC-IC : Evaluating our TIC In-Context example based approach

  * •

TIC-G3 : Evaluating our TIC Generic Prompt based approach with three LLM calls

  * •

TIC-G1 : Evaluating our TIC Generic Prompt based approach with one LLM call

###  5.2 Results and analysis

Tables 1 and 2 present our experiment results. TIC-IC outperforms LLM+P
approach on both the LLM+P dataset and the Language Variation dataset in all
seven domains. In particular, TIC-IC achieves $\approx$100% accuracy on the
LLM+P dataset even in complex domains such as Floortile, Storage and Termes
with GPT-3.5 Turbo, GPT-4 and PaLM2. This highlights TIC’s ability to generate
highly accurate PDDLs using logically interpretable intermediate
representations of the natural language task descriptions. The reason for
slightly lower accuracy of TIC-IC in Floortile with PaLM2 is because the chat-
bison model truncates the output of 2 out of 20 problems. We also observe that
TIC-IC results in a high accuracy on the Language Variation dataset with GPT-4
and PaLM2.

In general, the lower accuracy on the Language Variation dataset compared to
the LLM+P dataset is probably due to more complex and longer sentences in the
task descriptions of the former dataset. See Appendix E for examples.

As we move on to the generic prompt based approach, TIC-G3 generalizes well
overall on GPT-4. It achieves $\geq 95\%$ accuracy on all seven domains (with
100% on five domains) of the LLM+P dataset and $\geq 93.33\%$ on all seven
domains (with 100% on four domains) of the Language Variation dataset. In
general, the lower accuracy of TIC-G1 as compared to TIC-G3 may be due to the
more complex instructions to LLM for extracting cardinalities, objects and
rules in a single step. We also observe a drop in accuracy of TIC-G3 and
TIC-G1 for some domains on PaLM2 and GPT-3.5 models when compared to GPT-4.
This may be due to the failure of PaLM2 and GPT-3.5 in abiding by the generic
instructions in some cases. Some of the common errors by these LLMs are in
extracting bijective map relationships and intensional rules as well as in
generating syntactically correct ASP rules.

##  6 Conclusion

In this paper we studied the problem of generating plans from natural language
planning task requests. We presented the Translate-Infer-Compile (TIC)
approach to generate task PDDLs automatically from natural language planning
task descriptions with high accuracy. Thus, TIC addresses the limitations of
LLMs in generating accurate plans in the presence of a large number of actions
by enabling use of external planners. The core idea of TIC lies in the
introduction of a logically interpretable intermediate representation of the
task description. TIC combines the strengths of LLMs and symbolic reasoning to
address LLMs’ limitation in directly generating task PDDLs from task
descriptions. TIC not only outperforms similar approaches with respect to the
accuracy of task PDDL generation but is also more general.

Although, in this work, we have focused on generating planning task PDDLs, it
is worth noting that the TIC approach is generic enough to be applicable for
use cases that need to support a natural language interface on top of
structured tasks such as mathematical problems (see also Tool-integrated
Reasoning Agents (ToRA) Gou et al. (2023)), API calls (see also ToolFormer
Schick et al. (2023), LangChain Chase (2022), and Gorilla Patil et al.
(2023)), database queries (see also Spider dataset Yu et al. (2018) and the
report on LLM’s accuracy on SQL databases Sequeda et al. (2023)) etc.

While the ability to do logical reasoning helps us achieve high accuracy, it
comes with a price of modeling the domain. However, since the domain modeling
is an offline task, we believe that it can be accelerated with semi-automatic
techniques, possibly using LLMs among others, e.g., Ishay et al. (2023). In
this context of knowledge engineering, we believe that Description Logics
Baader et al. (2003) as a formalism and the standard Web Ontology Language
(OWL) OWL-WG (2012) may lend themselves as alternatives to ASP.

## References

  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, …, and Andy Zeng.  Do as i can, not as i say: Grounding language in robotic affordances, 2022.
  * Alkhazraji et al. [2020] Yusra Alkhazraji, Matthias Frorath, Markus Grützner, Malte Helmert, Thomas Liebetraut, Robert Mattmüller, Manuela Ortlieb, Jendrik Seipp, Tobias Springenberg, Philip Stahl, and Jan Wülfing.  Pyperplan.  https://doi.org/10.5281/zenodo.3700819, 2020.
  * Baader et al. [2003] Franz Baader, Diego Calvanese, Deborah L. McGuinness, Daniele Nardi, and Peter F. Patel-Schneider, editors.  The Description Logic Handbook: Theory, Implementation, and Applications.  Cambridge University Press, 2003.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, …, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * Chase [2022] Harrison Chase.  Langchain.  https://github.com/langchain-ai/langchain, 2022.
  * Gebser et al. [2019] Martin Gebser, Roland Kaminski, Benjamin Kaufmann, and Torsten Schaub.  Multi-shot ASP solving with clingo.  Theory Pract. Log. Program., 19(1):27–82, 2019.
  * Gelfond and Lifschitz [1988] Michael Gelfond and Vladimir Lifschitz.  The stable model semantics for logic programming.  In Robert A. Kowalski and Kenneth A. Bowen, editors, Logic Programming, Proceedings of the Fifth International Conference and Symposium, Seattle, Washington, USA, August 15-19, 1988 (2 Volumes), pages 1070–1080. MIT Press, 1988.
  * Ghallab et al. [1998] M. Ghallab, A. Howe, C. Knoblock, D. Mcdermott, A. Ram, M. Veloso, D. Weld, and D. Wilkins.  PDDL—The Planning Domain Definition Language, 1998.
  * Ghallab et al. [2004] Malik Ghallab, Dana S. Nau, and Paolo Traverso.  Automated planning - theory and practice.  Elsevier, 2004.
  * Gou et al. [2023] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen.  Tora: A tool-integrated reasoning agent for mathematical problem solving, 2023.
  * Guan et al. [2023] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati.  Leveraging pre-trained large language models to construct and utilize world models for model-based task planning, 2023.
  * Haslum et al. [2019] Patrik Haslum, Nir Lipovetzky, Daniele Magazzeni, and Christian Muise.  An Introduction to the Planning Domain Definition Language.  Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2019.
  * Helmert [2006a] M. Helmert.  The fast downward planning system.  Journal of Artificial Intelligence Research, 26:191–246, July 2006.
  * Helmert [2006b] Malte Helmert.  The fast downward planning system.  J. Artif. Intell. Res., 26:191–246, 2006.
  * Huang et al. [2022] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter.  Inner monologue: Embodied reasoning through planning with language models, 2022.
  * Ishay et al. [2023] Adam Ishay, Zhun Yang, and Joohyung Lee.  Leveraging large language models to generate answer set programs.  In Pierre Marquis, Tran Cao Son, and Gabriele Kern-Isberner, editors, Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, KR 2023, Rhodes, Greece, September 2-8, 2023, pages 374–383, 2023.
  * Lifschitz [2008] Vladimir Lifschitz.  What is answer set programming?  In Dieter Fox and Carla P. Gomes, editors, Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pages 1594–1597. AAAI Press, 2008.
  * Liu et al. [2023] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone.  Llm+p: Empowering large language models with optimal planning proficiency, 2023.
  * Lyu et al. [2023] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch.  Faithful chain-of-thought reasoning, 2023.
  * Nye et al. [2021] Maxwell Nye, Michael Henry Tessler, Joshua B. Tenenbaum, and Brenden M. Lake.  Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning, 2021.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  arXiv:2303.08774, 2023.
  * OWL-WG [2012] W3C OWL-WG.  Owl 2 web ontology language document overview (second edition).  https://www.w3.org/TR/owl2-overview/, 2012.
  * Pallagani et al. [2022] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia.  Plansformer: Generating symbolic plans using transformers, 2022.
  * Patil et al. [2023] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.  Gorilla: Large language model connected with massive apis, 2023.
  * Rana et al. [2023] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf.  Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning, 2023.
  * Schick et al. [2023] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools, 2023.
  * Seipp et al. [2022] Jendrik Seipp, Álvaro Torralba, and Jörg Hoffmann.  Pddl generators.  https://doi.org/10.5281/zenodo.6382173, March 2022.
  * Sequeda et al. [2023] Juan Sequeda, Dean Allemang, and Bryon Jacob.  A benchmark to understand the role of knowledge graphs on large language model’s accuracy for question answering on enterprise sql databases, 2023.
  * Silver et al. [2022] Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling.  PDDL planning with pretrained large language models.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
  * Sohrabi [2019] Shirin Sohrabi.  AI planning for enterprise: Putting theory into practice.  In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 6408–6410. ijcai.org, 2019.
  * Valmeekam et al. [2023a] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati.  On the planning abilities of large language models : A critical investigation, 2023.
  * Valmeekam et al. [2023b] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati.  Large language models still can’t plan (a benchmark for llms on planning and reasoning about change), 2023.
  * Xie et al. [2023] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh.  Translating natural language to planning goals with large-language models, 2023.
  * Yang et al. [2023] Zhun Yang, Adam Ishay, and Joohyung Lee.  Coupling large language models with logic programming for robust and general reasoning from text.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5186–5219. Association for Computational Linguistics, 2023.
  * Yu et al. [2018] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  CoRR, abs/1809.08887, 2018.
  * Zeng et al. [2022] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.  Socratic models: Composing zero-shot multimodal reasoning with language, 2022.
  * Zhou et al. [2023] Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma.  Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning, 2023.

##  Appendix A LLM Prompt for Generating Intermediate Representation using In-
Context Example

LLM Prompt for the Barman domain for generating intermediate representation
using domain specific in-context example I want you to create ASP Logic
Program representation of a paragraph. An example paragraph is: You have 1
shaker with 3 levels, 4 shot glasses, 3 dispensers for 3 ingredients. The
shaker and shot glasses are clean, empty, and on the table. Your left and
right hands are empty. The first ingredient of cocktail1 is ingredient3. The
second ingredient of cocktail1 is ingredient1. The first ingredient of
cocktail2 is ingredient1. The second ingredient of cocktail2 is ingredient2.
Your goal is to make 3 cocktails. shot1 contains cocktail1. shot2 contains
cocktail2. shot3 contains cocktail1. The ASP Logic Program representation of
the example paragraph is:

    cardinality(shaker, 1).
    cardinality(level, 3).
    cardinality(shot, 4).
    cardinality(dispenser, 3).
    cardinality(ingredient, 3).
    cardinality(cocktail, 2).
    init(clean(X)) :- object(X, shaker).
    init(clean(X)) :-  object(X, shot).
    init(empty(X)) :- object(X, shaker).
    init(empty(X)) :- object(X, shot).
    init(ontable(X)) :- object(X, shaker).
    init(ontable(X)) :- object(X, shot).
    init(handempty(left)).
    init(handempty(right)).
    init(map(dispenser, dispenses, ingredient)).
    init(cocktail_part1(cocktail1, ingredient3)).
    init(cocktail_part2(cocktail1, ingredient1)).
    init(cocktail_part1(cocktail2, ingredient1)).
    init(cocktail_part2(cocktail2, ingredient2)).
    goal(contains(shot1, cocktail1)).
    goal(contains(shot2, cocktail2)).
    goal(contains(shot3, cocktail1)).

Now I have a new paragraph:  You have 1 shaker with 3 levels, 5 shot glasses,
3 dispensers for 3 ingredients. The shaker and shot glasses are clean, empty,
and on the table. Your left and right hands are empty. The first ingredient of
cocktail1 is ingredient2. The second ingredient of cocktail1 is ingredient1.
The first ingredient of cocktail2 is ingredient1. The second ingredient of
cocktail2 is ingredient2. The first ingredient of cocktail3 is ingredient1.
The second ingredient of cocktail3 is ingredient3. The first ingredient of
cocktail4 is ingredient3. The second ingredient of cocktail4 is ingredient2.
Your goal is to make 4 cocktails. shot1 contains cocktail1. shot2 contains
cocktail4. shot3 contains cocktail3. shot4 contains cocktail2.  Provide me
with the ASP Logic Program representation of the new paragraph directly
without further explanations.

##  Appendix B Generating Intermediate Representation using Generic LLM
Prompts

Text You have 2 tables, 5 plates, 4 apples, and 3 balls for 3 kids. apple1 is
on plate_2. apple2 is on plate_1. orange1 is on plate_1. orange3 is on
plate_2. ball1 is in first drawer of first table. ball3 is in second drawer of
first table. apple3 is in first drawer of second table. apple2 is in second
drawer of second table. The apples are red. There are three oranges. The
plates are clean. The kids are inside the hall. cell_0_1 cell_0_2 cell_1_1
cell_1_2 cell_2_1 cell_2_2 cell_3_1 cell_3_2 There are 8 cells, each cell has
a robot. Your goal is to move the oranges to empty plates. orange1 should be
on plate_3, orange2 should be on plate_4, orange3 should be on plate_5. No
orange should be unplated. The plates are not clean. Each kid should paint a
ball. The kids are inside the hall.

Object types

    [
     {’type’: ’apple’, ’description’: ’The set of apples.’},
     {’type’: ’orange’, ’description’: ’The set of oranges.’},
     {’type’: ’plate’, ’description’: ’The set of plates.’},
     {’type’: ’table’, ’description’: ’The set of tables.’},
     {’type’: ’ball’, ’description’: ’The set of balls.’},
     {’type’: ’kid’, ’description’: ’The set of kids.’},
     {’type’: ’room’, ’description’: ’The set of rooms.’},
     {’type’: ’cell’, ’description’: ’The set of cells.’},
     {’type’: ’robot’, ’description’: ’The set of robots.’}
    ]

Predicates

    [
    {’predicate’: ’on’, ’arity’: 2, ’argument types’: [’apple’, ’plate’], ’description’:
    ’on(a, b) means apple a is on plate b.’},

    {’predicate’: ’red’, ’arity’: 1, ’argument types’: [’apple’], ’description’:
    ’red(a) means apple a is red.’},

    {’predicate’: ’clean’, ’arity’: 1, ’argument types’: [’plate’], ’description’:
    ’clean(a) means plate a is clean.’},

    {’predicate’: ’not_clean’, ’arity’: 1, ’argument types’: [’plate’], ’description’:
    ’not_clean(a) means plate a is not clean.’},

    {’predicate’: ’empty’, ’arity’: 1, ’argument types’: [’plate’], ’description’:
    ’empty(a) means plate a is empty. A plate is empty if there is no apple on it.’},

    {’predicate’: ’cell_grid’, ’arity’: 3, ’argument types’: [’row’, ’column’, ’cell’],
    ’description’: ’cell_grid(a, b, c) means cell a is on row b and column c.’},

    {’predicate’: ’unplated’, ’arity’: 0, ’argument types’: [], ’description’:
    ’unplated means at least one orange is not on the plate.’},

    {’predicate’: ’play’, ’arity’: 2, ’argument types’: [’ball’, ’kid’], ’description’:
    ’play is a bijective map between balls and kids. play(a, b)
    means ball a is played with by kid b.’},

    {’predicate’: ’cell_robot_map’, ’arity’: 2, ’argument types’: [’cell’, ’robot’],
    ’description’: ’cell_robot_map is a bijective map between cells and robots.
    cell_robot_map(a, b) means cell a has robot b.’},

    {’predicate’: ’in_1’, ’arity’: 2, ’argument types’: [’apple or ball’, ’table’],
    ’description’: ’in_1(a, b) means apple or ball a is in first drawer of table b.’},

    {’predicate’: ’in_2’, ’arity’: 2, ’argument types’: [’apple or ball’, ’table’],
    ’description’: ’in_2(a, b) means apple or ball a is in second drawer of table b.’},

    {’predicate’: ’inside’, ’arity’: 2, ’argument types’: [’kid’, ’room’],
    ’description’: ’inside(a, hall) means kid a is inside hall.’}
    ]

LLM Prompt for Cardinality Extraction Given a list of object types and their
descriptions as JSON. The provided text describes a planning task. The
planning task describes an init state and a goal state. I want you to extract
information from the provided task description by following below steps: Step
1. Detect the init state description and the goal state description. The init
state description is the text from start to begin of goal description. The
goal description typically starts with ’Your goal is …’. Step 2. Extract for
each object type T the explicitly mentioned count, width, and height of
instances of T from the init state description. Object types: [AS ABOVE] Text
: [AS ABOVE] Answer:

    {’table’: 2, ’apple’: 4, ’orange’: 3, ’plate’: 5, ’ball’: 3, ’kid’: 3, ’cell’: 8}

Object types: ⟨Task domain object types⟩ Text: ⟨Task description⟩ Answer:

LLM Prompt for named objects extraction Given a list of object types with
their descriptions as JSON. Extract the instances of each object type from the
provided text. Follow below steps in order to do so. Step 1. Instances appear
as singular nouns in text. Ignore plural nouns. Step 2. Consider only the
instances that are explicitly mentioned in the provided text. Do not output
instances that are not mentioned in the text. Step 3. If an instance id
contains spaces or hyphens, replace them by underscores. Step 4. If an
instance id begins with an upper case, then convert it to lower case. Object
types: [AS ABOVE] Text: [AS ABOVE] Answer:

    {’apple’: [’apple1’, ’apple2’], ’orange’: [], ’plate’: [’plate_1’, ’plate_2’,
    ’plate_3’, ’plate_4’, ’plate_5’], ’table’: [’table1’, ’table2’], ’ball’: [’ball1’,
    ’ball3’],  ’room’: [’hall’], ’kid’: [], ’robot’: [], ’cell’: [’cell_0_1’,
    ’cell_0_2’, ’cell_1_1’, ’cell_1_2’, ’cell_2_1’, ’cell_2_2’, ’cell_3_1’,
    ’cell_3_2’]}

Object types: ⟨Task domain object types⟩ Text: ⟨Task description⟩ Answer:

LLM Prompt for Rules Extraction Given (1) A JSON with object types as keys and
list of objects as values. (2) A list of predicates. The predicates of arity
one represent properties of objects. The predicates of arity two or more
represent relationships of two or more objects. (3) A text describing a
planning task. A planning task description consists of initial state and goal
state descriptions. Translate the provided text following below steps: Step 1.
Extract the initial state and goal state descriptions. Step 2. Translate each
state description by following below rules as applicable. ’⟨state⟩’ is a
placeholder for ’init’ or ’goal’ for sentences of initial state or goal state
respectively. While doing so, use only the provided predicates and respect
their arity and type of arguments. Step 3. Output the result as a valid ASP
Logic Program without any comments or additional text. State sentence
translation rules: Rule 1. Represent properties of sets of unnamed objects of
the same type as ASP Logic Program rules. For example, translate ’apples are
clean’ as ’⟨state⟩(clean(X)) :- object(X, apple).’, where ’apple’ is an object
type and ’clean’ is a predicate of arity 1 and argument type apple. Rule 2.
Represent properties of named objects as ASP Logic Program facts. For example,
translate sentence ’apple1 is red’ as ’⟨state⟩(red(apple1)).’, where ’apple1’
is an object and ’red’ is a predicate of arity 1 and argument type apple. Rule
3. Represent a relationship all unnamed objects of an object type with a named
object as a ASP Logic Program rule. For example, translate ’apples are on the
table’ as ’⟨state⟩(on(X, table)) :- object(X, apple).’ where ’apple’ is an
object type, ’table’ is an object of type ’furniture’ and ’on’ is a predicate
of arity 2 and argument types [’apple’, ’furniture’]. Rule 4. Represent
relationships between two or more named objects as ASP Logic Program facts.
For example, translate sentence ’apple1 is on plate2’ as ’⟨state⟩(on(apple1,
plate2)).’, where ’apple1’ and ’plate2’ are objects of type ’apple’ and
’plate’ resp, and ’on’ is a predicate of arity 2 and argument types [’fruit,
’plate’]. Rule 5. Represent relationships between two or more unnamed objects
as ASP Logic Program rules. For example, translate sentence ’cups handles are
free’ as ’⟨state⟩(free(X, Y)) :- cup_handle(X, Y).’, where ’cup_handle’ is a
predicate of arity 2 and argument types [’cup’, ’handle’] and ’free’ is a
predicate of arity 2 and argument types [’cup’, ’handle’]. Rule 6. Represent a
bijective map between two sets of unnamed objects as ASP Logic Program facts.
For example, translate sentence ’3 balls for 3 kids’ as ’⟨state⟩(map(ball,
play, kid)).’, where ’ball’ and ’kid’ are object types, and ’play’ is a
predicate of arity 2 with argument types [’ball’, ’kid’]. For example,
translate sentence ’each cell has a ball’ as ’⟨state⟩(map(cell, cell_ball_map,
ball)).’, where ’cell’ and ’ball’ are object types, and ’cell_ball_map’ is a
bijective map predicate of arity 2 with argument types [’cell’, ’ball’]. Rule
7. Represent objects that denote a cell on a two dimensional grid ’cell_grid’
as ASP Logic Program facts. For example, translate objects ’cell_2_3’ as
’⟨state⟩(cell_grid(2, 3, cell_2_3)).’, where ’cell_2_3’ is an object and
’cell_grid’ is a provided tertiary predicate.

LLM Prompt for Rules Extraction Ctd. Below an example of objects, predicates,
text and the answer. Objects:

    {’apple’: [’apple1’, ’apple2’], ’orange’: [], ’plate’: [’plate_1’, ’plate_2’,
    ’plate_3’, ’plate_4’, ’plate_5’], ’table’: [’table1’, ’table2’],
    ’ball’: [’ball1’, ’ball3’], ’room’: [’hall’], ’kid’: [], ’robot’: [],
    ’cell’: [’cell_0_1’, ’cell_0_2’, ’cell_1_1’, ’cell_1_2’, ’cell_2_1’, ’cell_2_2’,
    ’cell_3_1’, ’cell_3_2’]}

Predicates: [AS ABOVE] Text: [AS ABOVE] Answer:

    init(on(apple1, plate_2)).
    init(on(apple2, plate_1)).
    init(on(orange1, plate_1)).
    init(on(orange3, plate_2)).
    init(in_1(ball1, table1)).
    init(in_2(ball3, table1)).
    init(in_1(apple3, table2)).
    init(in_2(apple2, table2)).
    init(red(X)) :- object(X, apple).
    init(map(ball, play, kid).
    init(clean(X)) :- object(X, plate).
    init(inside(X, hall)) :- object(X, kid).
    init(cell_grid(0, 1, cell_0_1)).
    init(cell_grid(0, 2, cell_0_2)).
    init(cell_grid(1, 1, cell_1_1)).
    init(cell_grid(1, 2, cell_1_2)).
    init(cell_grid(2, 1, cell_2_1)).
    init(cell_grid(2, 2, cell_2_2)).
    init(cell_grid(3, 1, cell_3_1)).
    init(cell_grid(3, 2, cell_3_2)).
    init(map(cell, cell_robot_map, robot)).
    goal(on(orange1, plate_3)).
    goal(on(orange2, plate_4)).
    goal(on(orange3, plate_5)).
    goal(not_clean(X)) :- object(X, plate).
    goal(map(kid, paint, ball)).
    goal(inside(X, hall)) :- object(X, kid).
    goal(not unplated)).

Objects: [AS EXTRACTED IN THE OBJECT EXTRACTION STEP] Predicates: ⟨Predicates
of the task domain⟩ Text: ⟨Task description⟩ Answer:

Generic LLM Prompt for generating intermediate representation in one step
Given (1) A JSON list of object types with their descriptions. (2) A list of
predicates. The predicates of arity one represent properties of objects. The
predicates of arity two or more represent relationships of two or more
objects. (3) A text describing a planning task. A planning task description
consists of initial state and goal state descriptions. Translate the provided
text following below steps. Step 1. Detect the initial state and goal state
descriptions. Step 2. Extract for each object type T the explicitly mentioned
count, width, and height of instances of T from the init state description.
Step 3. Extract all instances of each object type from the provided text.
Instances appear as singular nouns in the text. Ignore the plural nouns. In
order to do so, consider only the instances that are explicitly mentioned in
the provided text. Do not output instances that are not mentioned in the text.
If an instance contains hyphens, replace the hyphens by underscores. If an
instance begins with an upper case, then lower case it. If an instance
contains spaces, replace the spaces by underscores. Step 4. Translate each
state description by following below rules as applicable. ’⟨state⟩’ is a
placeholder for ’init’ or ’goal’ for sentences of initial state or goal state
respectively. While doing so, use only the provided predicates and respect
their arity and type of arguments. Also use the object types and list of
objects extracted in Step 3. Step 5. Output the result as a valid ASP Program
without any comments or additional text. State sentence translation rules:
[SAME AS ABOVE] Object types: [SAME AS ABOVE] Predicates: [SAME AS ABOVE]
Text: [SAME AS ABOVE] Answer: [SAME AS ABOVE] Objects types: ⟨Object types of
the task domain⟩ Predicates: ⟨Predicates of the task domain⟩ Answer:

##  Appendix C Domain Knowledge as ASP Rules

###  C.1 Barman

    init(@make_seq(N-1, l, next, 1)) :- cardinality(level, N).
    first_level(X) :- object(X, level), not init(next(_, X)).

    init(shaker_empty_level(X, Y)) :- object(X, shaker), first_level(Y).
    init(shaker_level(X, Y)) :- object(X, shaker), first_level(Y).

    object(X, hand) :- init(handempty(X)).

    object(L, level) :- init(next(L, _)).
    object(L, level) :- init(next(_, L)).

    object(X, cocktail) :- init(cocktail_part1(X, _)).
    object(X, cocktail) :- init(cocktail_part2(X, _)).
    object(X, ingredient) :- init(cocktail_part1(_, X)).
    object(X, ingredient) :- init(cocktail_part2(_, X)).

    object(X, shot) :- goal(contains(X, _)).

###  C.2 Blocksworld

    object(B, block) :- init(on(B, _)).
    object(B, block) :- init(on(_, B)).
    object(B, block) :- init(on_table(B)).
    object(B, block) :- init(clear(B)).

    object(B, block) :- goal(on(B, _)).
    object(B, block) :- goal(on(_, B)).
    object(B, block) :- goal(on_table(B)).
    object(B, block) :- goal(clear(B)).

    object(B, object) :- object(B, block).

###  C.3 Floortile

    floortile_grid(R,C,Z1) :- init(floortile_grid(R,C,Z1)).
    init(up(Z1,Z2)) :- floortile_grid(R,C,Z1), floortile_grid(R-1,C,Z2).
    init(down(Z2,Z1)) :- floortile_grid(R,C,Z1), floortile_grid(R-1,C,Z2).
    init(right(Z1,Z2)) :- floortile_grid(R,C,Z1), floortile_grid(R,C-1,Z2).
    init(left(Z2,Z1)) :- floortile_grid(R,C,Z1), floortile_grid(R,C-1,Z2).
    init(robot_has(robot1, white)).
    init(robot_has(robot2, black)).

    object(Z, tile) :- floortile_grid(_,_,Z).

    object(X, color) :- init(robot_has(_, X)).
    object(X, robot) :- init(robot_has(X, _)).

    object(X, color) :- init(available_color(X)).
    object(X, robot) :- init(robot_at(X, _)).
    object(X, tile) :- init(robot_at(_, X)).

    object(X, color) :- goal(painted(_, X))
    object(X, tile) :- goal(painted(X, _)).

    init(clear(T)) :- object(T, tile), not init(robot_at(_, T)).

    init(min_cost_metric("=(total-cost) 0)")).

    init(available_color(X)) :- init(robot_has(_, X)).
    init(available_color(X)) :- goal(painted(_, X)).

###  C.4 Grippers

    object(X, robot) :- init(at_robby(X, _)).
    object(X, room) :- init(at_robby(_, X)).
    object(X, ball) :- init(at(X, _)).
    object(X, room) :- init(at(_, X)).
    object(X, ball) :- goal(at(X, _)).
    object(X, room) :- goal(at(_, X)).
    object(X, object) :- object(X, ball).
    object(X, gripper):- init(free(_, X)).
    object(X, robot):- init(free(X, _)).
    object(X, gripper) :- object(X, left_gripper).
    object(X, gripper) :- object(X, right_gripper).

    robot_left_gripper_map(X,Y) :- init(robot_left_gripper_map(X,Y)).
    robot_right_gripper_map(X,Y) :- init(robot_right_gripper_map(X,Y)).

###  C.5 Storage

    object(X, storearea) :- init(on(_, X)).
    object(X, crate) :- init(on(X, _)).
    object(X, hoist) :- init(at(X, _)).
    object(X, storearea) :- object(X, depot_storearea).
    depot_storearea_grid(R,C,Z) :- init(depot_storearea_grid(R,C,Z)).

    adjacent(Z1,Z2) :- depot_storearea_grid(R,C,Z1), depot_storearea_grid(R-1,C,Z2).
    adjacent(Z2,Z1) :- depot_storearea_grid(R,C,Z1), depot_storearea_grid(R-1,C,Z2).
    adjacent(Z1,Z2) :- depot_storearea_grid(R,C,Z1), depot_storearea_grid(R,C-1,Z2).
    adjacent(Z2,Z1) :- depot_storearea_grid(R,C,Z1), depot_storearea_grid(R,C-1,Z2).

    init(connected(X, Y)) :- init(connected(Y, X)).

    goal(connected(X, Y)) :- goal(connected(Y, X)).

###  C.6 Termes

    init(@make_seq(N, n, succ, 0)) :- init(max_height(N)).
    object(N, numb) :- init(succ(N, _)).
    object(N, numb) :- init(succ(_, N)).
    termes_pos_grid(R,C,Z) :- init(termes_pos_grid(R,C,Z)).

    object(Z, position) :- termes_pos_grid(_,_,Z).
    init(height(Z, n0)) :- termes_pos_grid(_,_,Z).

    init(neighbor(Z1,Z2)) :- termes_pos_grid(R,C,Z1), termes_pos_grid(R-1,C,Z2).
    init(neighbor(Z2,Z1)) :- termes_pos_grid(R,C,Z1), termes_pos_grid(R-1,C,Z2).
    init(neighbor(Z1,Z2)) :- termes_pos_grid(R,C,Z1), termes_pos_grid(R,C-1,Z2).
    init(neighbor(Z2,Z1)) :- termes_pos_grid(R,C,Z1), termes_pos_grid(R,C-1,Z2).

    0 { goal(height(Z, n0)) } 1 :- termes_pos_grid(_,_,Z).
    :- termes_pos_grid(_,_,Z), #count {X : goal(height(Z, X))} != 1.

###  C.7 Tyreworld

    init(@make_map(T1, P, T2, N)) :- init(map(T1, P, T2)), cardinality(T1, N).
    object(@gen_objects(N, 0, intact_tyre), intact_tyre) :- cardinality(intact_tyre, N).
    object(@gen_objects(N, 0, flat_tyre), flat_tyre) :- cardinality(flat_tyre, N).
    object(@gen_objects(N, 0, hub), hub) :- cardinality(hub, N).
    object(@gen_objects(N, 0, nut), nut) :- cardinality(nut, N).
    goal(@make_map(T1, P, T2, N)) :- goal(map(T1, P, T2)), cardinality(T1, N).

    object(X, container) :- init(in(_, X)).
    object(X, wheel) :- object(X, flat_tyre).
    object(X, wheel) :- object(X, intact_tyre).
    object(X, tool):- object(X, jack).
    object(X, tool):- object(X, pump).
    object(X, tool):- object(X, wrench).
    init(intact(X)) :- object(X, intact_tyre).

    object(X, container) :- init(in(_, X)).

    object(X, hub) :- init(on(_, X)).

    object(X, nut) :- init(tight(X,_)).
    object(X, hub) :- init(tight(_,X)).

##  Appendix D PDDL Comparision

Following algorithm illustrate the main steps of for comparing two PDDLs. We
use this to compare the TIC generated task PDDL with the ground truth task
PDDL.

1:taskPDDLTypes $\leftarrow$ getPDDLTypes(taskPDDL)

2:correctPDDLTypes $\leftarrow$ getPDDLTypes(correctPDDL)

3:if set(taskPDDLTypes) $\neq$ set(correctPDDLTypes) then

4: return $False$

5:end if

6:taskPDDLObjects $\leftarrow$ getPDDLObjects(taskPDDL)

7:correctPDDLObjects $\leftarrow$ getPDDLObjects(correctPDDL)

8:if len(taskPDDLObjects) $\neq$ len(correctPDDLObjects) then

9: return $False$

10:end if

11:taskInitStateAtoms $\leftarrow$ getInitStateAtoms(taskPDDL)

12:correctInitStateAtoms $\leftarrow$ getInitStateAtoms(correctPDDL)

13:if len(taskInitStateAtoms) $\neq$ len(correctInitStateAtoms) then

14: return $False$

15:end if

16:taskGoaltStateAtoms $\leftarrow$ getGoalStateAtoms(taskPDDL)

17:correctGoalStateAtoms $\leftarrow$ getGoalStateAtoms(correctPDDL)

18:if len(taskGoalStateAtoms) $\neq$ len(correctGoalStateAtoms) then

19: return $False$

20:end if

21:taskObjects $\leftarrow$ []

22:correctObjects $\leftarrow$ []

23:for all t $\in$ sort(taskPDDLTypes) do

24: correctObjectsOfType $\leftarrow$ getObjectsOfType(correctPDDL, t)

25: typePerm $\leftarrow$ permutations(correctObjectsOfType)

26: append typePerm to correctObjects

27: taskObjectsOfType $\leftarrow$ getObjectsOfType(taskPDDL, t)

28: append taskObjectsOfType to taskObjects

29:end for

30:for all perm $\in$ cartesianProduct(correctObjects) do //each perm is a
list containing one permutation of objects for each object type

31: if checkStateEquivalence(taskObjects, perm, taskInitStateAtoms,
correctInitStateAtoms) and

32:checkStateEquivalence(taskObjects, perm, taskGoaltStateAtoms,
correctGoalStateAtoms then

33: return $True$

34: end if

35:end for

36:return $False$

Algorithm 1 Compare PDDLs

checkStateEquivalence of two states $s$ and $t$ is true if state $s$ can be
simulated by state $t$ and vice versa, otherwise false. A state $s$ is
simulated by another state $t$ if each atom in $s$ can be simulated by an atom
in state $t$. That is, there exists a mapping $m$ from objects of state $s$ to
objects of state $t$ such that for every atom $a\in s$ there exists an atom
$b\in t$ such that $m(a)=b$.

##  Appendix E Examples of Language Variation

Below we provide a few examples of Language Variation of task descriptions.
Please refer to supplementary material for the entire Language Variation
dataset.

### E.1

Barman task description from original dataset You have 1 shaker with 3 levels,
4 shot glasses, 3 dispensers for 3 ingredients. The shaker and shot glasses
are clean, empty, and on the table. Your left and right hands are empty. The
first ingredient of cocktail1 is ingredient2. The second ingredient of
cocktail1 is ingredient1. The first ingredient of cocktail2 is ingredient2.
The second ingredient of cocktail2 is ingredient3. The first ingredient of
cocktail3 is ingredient1. The second ingredient of cocktail3 is ingredient2.
Your goal is to make 3 cocktails. shot1 contains cocktail1. shot2 contains
cocktail3. shot3 contains cocktail2.

Barman task description from Language Variation dataset There is a shaker with
3 levels, 4 shot glasses, and 3 dispensers for 3 ingredients. The shaker and
the shot glasses are on the table. The shaker and the shot glasses are clean
and empty. Your left and right hands are empty. There are 3 cocktails to be
made, each with 2 ingredients. The first cocktail requires ingredient2 as the
first ingredient and ingredient1 as the second ingredient. The second cocktail
requires ingredient2 as the first ingredient and ingredient3 as the second
ingredient. The third cocktail requires ingredient1 as the first ingredient
and ingredient2 as the second ingredient. Your task is to make the 3 cocktails
and pour them into shot glasses. The first cocktail should be poured into shot
glass 1, the second cocktail into shot glass 3, and the third cocktail into
shot glass 2.

### E.2

Floortile task description from original dataset You have 5 rows and 3 columns
of unpainted floor tiles. tile-0-1 tile-0-2 tile-0-3 tile-1-1 tile-1-2
tile-1-3 tile-2-1 tile-2-2 tile-2-3 tile-3-1 tile-3-2 tile-3-3 tile-4-1
tile-4-2 tile-4-3 You have 2 robots. Each robot can paint in color white or
black. robot2 is at tile-1-1. robot1 is at tile-2-3. Your goal is to paint the
grid in the following pattern: tile-1-1 is white; tile-1-2 is black; tile-1-3
is white; tile-2-1 is black; tile-2-2 is white; tile-2-3 is black; tile-3-1 is
white; tile-3-2 is black; tile-3-3 is white; tile-4-1 is black; tile-4-2 is
white; tile-4-3 is black.

Floortile task description from Language Variation dataset There are 15
unpainted floor tiles arranged in 5 rows and 3 columns. tile-0-1 tile-0-2
tile-0-3 tile-1-1 tile-1-2 tile-1-3 tile-2-1 tile-2-2 tile-2-3 tile-3-1
tile-3-2 tile-3-3 tile-4-1 tile-4-2 tile-4-3 Two robots are available to paint
the tiles either white or black. Robot 2 is currently positioned at tile-1-1
while robot 1 is at tile-2-3. The objective is to paint the tiles in a
specific pattern where tile-1-1 is white, tile-1-2 is black, tile-1-3 is
white, tile-2-1 is black, tile-2-2 is white, tile-2-3 is black, tile-3-1 is
white, tile-3-2 is black, tile-3-3 is white, tile-4-1 is black, tile-4-2 is
white, and tile-4-3 is black.

### E.3

Grippers task description from original dataset You control 3 robots, each
robot has a left gripper and a right gripper. There are 4 rooms and 7 balls.
robot3 is in room3. robot2 is in room1. robot1 is in room4. ball1 is in room4.
ball7 is in room3. ball3 is in room3. ball4 is in room2. ball6 is in room1.
ball2 is in room3. ball5 is in room2. The robots’ grippers are free. Your goal
is to transport the balls to their destinations. ball1 should be in room1.
ball2 should be in room4. ball3 should be in room3. ball4 should be in room4.
ball5 should be in room3. ball6 should be in room2. ball7 should be in room1.

Grippers task description from Language Variation dataset There are 3 robots.
Each robot has a left gripper and a right gripper. There are 4 rooms and 7
balls. Robot 3 is in room 3, robot 2 is in room 1, and robot 1 is in room 4.
The balls are distributed among the rooms as follows: ball 1 is in room 4,
ball 7 is in room 3, ball 3 is in room 3, ball 4 is in room 2, ball 6 is in
room 1, ball 2 is in room 3, and ball 5 are in room 2. The robots’ grippers
are currently not holding anything. The objective is to move the balls to
their designated rooms: ball 1 to room 1, ball 2 to room 4, ball 3 to room 3,
ball 4 to room 4, ball 5 to room 3, ball 6 to room 2, and ball 7 to room 1.