Recent advancements in Large Language Models (LLMs) have sparked a revolution across various research fields.

In particular, the integration of common-sense knowledge from LLMs into robot task and motion planning has been proven to be a game-changer, elevating performance in terms of explainability and downstream task efficiency to unprecedented heights. However, managing the vast knowledge encapsulated within these large models has posed challenges, often resulting in infeasible plans generated by LLM-based planning systems due to hallucinations or missing domain information. To overcome these challenges and obtain even greater planning feasibility and computational efficiency, we propose a novel LLM-driven task planning approach called DELTA. For achieving better grounding from environmental topology into actionable knowledge, DELTA leverages the power of scene graphs as environment representations within LLMs, enabling the fast generation of precise planning problem descriptions. For obtaining higher planning performance, we use LLMs to decompose the long-term task goals into an autoregressive sequence of sub-goals for an automated task planner to solve.
Our contribution enables a more efficient and fully automatic task planning pipeline, achieving higher planning success rates and significantly shorter planning times compared to the state of the art.

§ INTRODUCTION

With the enormous progress in the research field of Natural Language Processing (NLP), various powerful Large Language Models (LLMs) have been developed and are capable of producing human-like texts, programming code, and service compositions etc. [1, 2, 3, 4, 5].
Nowadays with always more robots co-operating with humans in industrial and household settings [6, 7], where planning considering context becomes important [8], e.g., performing household tasks such as house cleaning as shown in Fig. <ref>, many researchers use LLMs for solving robot Task And Motion Planning (TAMP) problems [9, 10, 11, 12, 13, 14, 15]. While directly using pre-trained LLMs to generate action plans for the robots tends to result in extremely low success rates in generating executable plans and completing the goals [16, 17], most of them use LLMs to extract common-sense knowledge to improve the performance of classical automated task planning approaches with respect to plan correctness, executability, and feasibility [9, 12, 15]. Several approaches use LLMs to generate task specifications defined in formal language, e.g., the domain and problem files programmed in the Planning Domain Definition Language (PDDL) [18], that can be solved by the off-the-shelf TAMP algorithms [11, 15].

However, previous TAMP approaches were cumbersome as they required vast manual knowledge engineering and input from human experts, inducing practically impossible domain adaptations.
On the other hand, none of the approaches above tackle long-term task planning problems, which are particularly difficult to solve with the growing problem complexity.

For robots solving long-term task sequences in large and complex environments, having efficient environment representations is a crucial prerequisite for the robot to understand the semantic information [8].

When using mid-level perceptual representations such as 2D semantic segmentation, the robot should learn the complex mapping from mid-level representations to more appropriate high-level abstractions, e.g., environment topology and semantic relations between objects [19]. This can be solved using high-level representations such as scene graphs [20]. For tackling task planning problems in such environments, researchers have found that scene graphs can serve as compact and informative spatial representations and can improve planning efficiency [21, 19, 14].

An example of the long-term task decomposition upon the Shelbiana environment [22]. A scene graph is pre-built from the environment with the floor, room, and item layers. The edges refer to the semantic relationships. Using the scene graph as the environment representation, a human user queries a Large Language Model (LLM) with a goal description to extract the relevant items and decompose the goal into multiple sub-goals. An automated task planner generates a task plan with respect to the sub-goals for the robot to execute.

As it emerges from the state of the art, utilizing LLMs and automated task planning techniques to solve long-term robot task planning problems, with structured representations of large environments, still remains an open research topic.

To this end, we propose DELTA: Decomposed Efficient Long-term TAsk planning for mobile robots using LLMs, which is the first to fill the aforementioned vacancy.
DELTA first feeds scene graphs into LLMs to generate the necessary domain and problem specifications in formal planning language, then decomposes the long-term task goals into multiple sub-ones using LLMs. The corresponding sub-problems are then solved autoregressively with an automated task planner.
In summary, we present the following key contributions:

i) We introduce a novel combination of LLMs and scene graphs that enables the extraction of actionable and semantic knowledge from LLMs and its grounding into the environmental topology. Thanks to one-shot prompting, DELTA is capable of solving complex planning problems in unseen domains.

ii) With the LLM-driven task decomposition strategy and the usage of formal planning language, compared to representative LLM-based baselines, DELTA is able to complete long-term tasks with higher success rates, near-optimal plan quality, and significantly shorter planning time. Table <ref> shows a comparison of the capabilities of different approaches. Further details of the baselines can be found in Sec. <ref>.

Approach     Grounding in
formal planning
language     Structured
representation     Tackling

LLM+P [11]

SayPlan [14]

DELTA (ours)

Capabilities of different LLM-based approaches

§ RELATED WORK

§.§ 3D Scene Graphs

A 3D scene graph is a recent 3D scene representation used to model large real-world environments as a graph structure.

They were first introduced by Armeni [22] as a hierarchical model to connect buildings, rooms, objects, and humans in multiple layers.  Following this introduction, Rosinol [23, 24] and Hughes [25] investigated the construction of 3D scene graphs from sensor data.
While other approaches rather focus on modeling semantic relationships between objects
[26, 27, 28, 29, 30] from 3D point clouds.
With the rise of large language models, 3D scene graphs also become open-vocabulary
[31, 32, 33], offering a deeper understanding of room types, objects, and relationships.
Recently, 3D scene graphs have started to be integrated into robotics systems. Applications include navigation [34, 31, 32] as well as task and motion planning [21, 14].

§.§ LLM-based Robot Task and Motion Planning

Integrating LLMs into robot TAMP approaches has gained significant attention. Particularly, the wealth of LLMs in semantic and common-sense knowledge allows LLMs to proficiently understand Natural Language (NL) instructions and perform few-shot or even zero-shot planning [35].
A common strategy for using LLMs in robot TAMP includes using LLMs for capturing and utilizing the collective knowledge embedded in text data to understand the environment in terms of common practices and norms [9, 12, 10].
However, despite the capability of LLMs to interpret the environments and generate reasonable responses, they still struggle with analyzing complex spatial relationships and processing detailed environmental features [16]. Consequently, the LLM's output about the task and motion plan is often not executable for the robots. Therefore, many researchers developed various approaches to ground the NL output of LLMs into executable and affordable action plans [13, 9, 15], or ground actionable knowledge into formal planning language, such as PDDL [11, 36].
However, most of these approaches require pre-provided domain descriptions, e.g., PDDL domain files, handcrafted by human experts. Using LLMs to generate domain descriptions with formal planning language, which also generalizes to new domain knowledge, still remains an open research topic.

For robots operating in large environments, semantic understanding becomes a crucial factor in successful navigation. LLMs unlock the capability of reasoning over semantic relations embedded in large scenes and allow to capture of those relations from different scene representations, e,g., semantic maps [37], landmarks [38, 39], as well as scene graphs [14, 19].

However, most of the LLM-based robot TAMP approaches only focus on short-term tasks, e.g., object rearrangement and object goal navigation. Common experiments in large environments are also semantically simple. Popular examples such as “make a cup of coffee and bring it to my desk” only comprise two sub-tasks “make coffee” and “bring coffee to my desk”. The capability of LLMs to handle long-term tasks in large environments is not fully exploited.

While decomposing a long-term task into multiple sub-tasks via classical machine learning methods can lead to a significant reduction of planning time [6], completing such a job with LLMs is still unexplored in the state-of-the-art.

The architecture of DELTA with five steps: Domain Generation, Scene Graph Pruning, Problem Generation, Goal Decomposition, and Autoregressive Sub-Task Planning.

§ APPROACH

§.§ Problem Statement

We focus on solving long-term robot task planning problems with LLMs. We consider a mobile robot that moves in household environments and performs different long-term tasks. Given a 3D scene graph defined in programming code as environment representation and NL descriptions of domain and problem specifications, the LLM will generate the necessary PDDL planning files, and decompose the long-term task goal in the problem instances into a sequence of sub-goals for generating corresponding sub-problems autoregressively, such that an automated task planner can solve.

In order to distinguish the object keyword in PDDL from objects in scene graphs, in the following, we refer to the scene graphs objects as items. Additionally, we define agent as an object type in the domain file and robot as an instance of agent in the problem file. Furthermore, we assume full observability of all nodes in the scene graphs.

caption=Action “mop floor” defined in PDDL,
(:action mop_floor
:parameters (?a - agent ?i - item ?r - room)
:precondition (and
(agent_at ?a ?r)
(item_is_mop ?i)
(item_pickable ?i)
(agent_has_item ?a ?i)
(mop_clean ?i)
(not(floor_clean ?r))
:effect (and
(floor_clean ?r)
(not(mop_clean ?i))
(not(battery_full ?a))

§.§ Architecture

DELTA consists of five steps, i.e., domain generation, scene graph pruning, problem generation, long-term goal decomposition, and autoregressive sub-task planning. Fig. <ref> illustrates the system architecture of DELTA.

§.§.§ Domain Generation

An LLM takes an NL prompt describing the domain knowledge as input and generates a domain description file encoded in formal planning language, e.g., PDDL, correspondingly in a one-shot fashion. The prompt consists of three main parts, the role, the example, and the instruction. First, the LLM is assigned a role. Then an example of the domain description in both NL and formal language is provided, which includes the necessary object types and the action knowledge, as well as the pre-conditions and effects of the action. For instance, the “mop floor” action can be described as “For mopping floor, an agent, a room and a mop are necessary, with the mop being pickable. Before the action, the agent is located in the room and has the mop in hand, the mop is clean, and the floor is not clean. After the action, the floor is clean, but the mop is not clean anymore, and the agent's battery will no longer be full.” The corresponding action definition in PDDL is formulated as in Listing <ref>.=-1

Subsequently, the instruction briefly introduces the requirements for generating a new domain file,

i.e., description of new object types and new actions.

An overview of the prompt and the information flow of this stage is shown as follows:

[breakable, title=Prompt Structure for Domain Generation, title filled=false, colback=black!5!white, colframe=black!75!black, left=2pt, right=2pt, top=2pt, bottom=2pt]
Role: You are an excellent PDDL domain file generator. Given a description of action knowledge in NL, you can generate a PDDL domain file.

Example: A robot in a household environment can perform the following example actions with pre-conditions and effects in NL.
The corresponding action definitions in a PDDL domain file look like: example_domain.pddl.

Instruction: A new domain includes the following new object types and actions described in NL. Please generate a corresponding new PDDL domain file.

§.§.§ Scene Graph Pruning

The scene graphs have a hierarchical structure with three layers of nodes: floor, room, and item layers. Fig. <ref> visualizes an example of the scene graph's layout. The room nodes are annotated with their neighboring rooms, and the item nodes contain several attributes, e.g., accessibility, states, and affordable actions.
In particularly large environments, scene graphs can contain a large number of items, where not all items are relevant for accomplishing different tasks. E.g., a bed plays obviously no role in the task of making coffee. Therefore, we prune the items in the scene graphs with LLMs according to different tasks by only keeping the relevant ones while still maintaining the floors and rooms as well as their connections.

Similarly, we also provide an example in the prompt. Firstly, a list of items from an example scene graph is extracted, and then with a goal description in NL, a subset of relevant items is given as the target. Finally, in the instruction, another goal description and item list are given for querying.

On the one hand, the scene graph pruning allows a reduction of input tokens for the LLM, and thus achieving a fast response time in generating the problem files. On the other hand, the more concise the information provided to the LLMs, the less likely that the LLMs generate output with mistakes and hallucinations [16].

caption=Goal states of the house cleaning problem formulated in PDDL,
(item_disposed cola_can)
(item_disposed banana_peel)
(floor_clean living_room)
(floor_clean kitchen)
(mop_clean mop)

§.§.§ Problem Generation

The prompt of this step has the same structure as the previous one.
Similarly, the role is given as a problem file generator. The example includes a scene graph, a goal description in NL, and the predicates from the previously generated domain file,

and a desired problem file. The scene graph serves as the environment representation.

Thereafter, the instruction presents the request to generate a new problem file using a new scene graph, a new goal description, and the object types and predicates from the previously generated domain file. An overview of the prompt and the information flow of this stage is shown as follows:

[breakable, title=Prompt Structure for Problem Generation, title filled=false, colback=black!5!white, colframe=black!75!black, left=2pt, right=2pt, top=2pt, bottom=2pt]
Role: You are an excellent PDDL problem file generator. Given a scene graph representation and a goal description, you can generate a PDDL problem file.

Example: A scene graph represents a scene in a hierarchical way, it can be programmed as a nested Python dictionary.

An example looks like: example_scene_graph.py.

Given an example goal description in NL, and using the predicates defined in example_domain.pddl, a corresponding PDDL problem file looks like: example_problem.pddl.

Instruction: Now given a new query_scene_graph.py and using the predicates in the query_domain.pddl generated previously, please generate a new PDDL problem file.

In the generated problem file, the connections of rooms in the scene graph can be expressed using the neighbor predicate. For instance, if kitchen is connected with corridor, the relationship can be formulated as (neighbor kitchen corridor) and (neighbor corridor kitchen), since the rooms are connected bi-directionally.
Similarly, the attributes of items can be defined with the predicates from the previously generated domain file, such as (item_at mop bathroom_1) and (item_accessible cola_can).
The goal description in NL will also be translated into PDDL predicates. For example, the overall goal of a house cleaning problem given by the human user in Fig. <ref> can be formulated as shown in Listing <ref>.

§.§.§ Goal Decomposition

To improve the computational efficiency and reduce the complexity of the planning problem, the long-term goal defined in the problem file will be decomposed into a sequence of sub-goals.

The prompt has the same structure. The example consists of a long-term goal description and a target set of decomposed sub-goals, both formulated in PDDL. The instruction requests to decompose the long-term goal with the previously generated problem file. An overview of the prompt is shown as follows:

[breakable, title=Prompt Structure for Goal Decomposition, title filled=false, colback=black!5!white, colframe=black!75!black, left=2pt, right=2pt, top=2pt, bottom=2pt]
Role: You are an excellent assistant in decomposing long-term goals. Given a PDDL problem file, you can decompose the long-term goal in a sequence of sub-goals.

Example: Here is an example PDDL problem file: example_problem.pddl. For simplifying the planning problem, the long-term goal defined here can be decomposed into a sequence of sub-goals in NL. Using the predicates defined in example_domain.pddl, the long-term goal can be decomposed into: sub-goal 1, ..., sub-goal n in PDDL.

Instruction: Given the query_problem.pddl generated previously, please decompose the long-term goal into a sequence of sub-goals considering the predicates and actions from the previously generated query_domain.pddl.

Taking action knowledge into account in goal decomposition is essential. For instance, knowing that the action mop_floor requires mop_clean and results in not (mop_clean) as shown in Listing <ref>, which infers that one cannot mop the floor in another room continuously since the mop turns dirty after mopping the previous room. Thus, cleaning the mop before mopping the next room should be considered when decomposing the goal, as shown in Fig. <ref>.

§.§.§ Autoregressive Sub-Task Planning

After having the generated problem file and the sequence of sub-goals defined in PDDL, one can query an automated task planner to solve the sub-problems individually. Algorithm <ref> illustrates how the sub-problems are generated and solved autoregressively.

The algorithm takes the automated task planner $\Pi$ and the previously generated domain file $d$, problem file $p_0$, and the sequence of PDDL sub-goals $G_{PDDL}$ as inputs.

The initial state $s_{1, PDDL}$ of the first sub-problem $p_1$ is identical to the initial state $s_{0, PDDL}$ from the original problem $p_0$, thus, the first sub-problem file $p_1$ can be simply formulated by replacing the goal $g_{0, PDDL}$ from the original problem $p_0$ with the first sub-goal $g_{1, PDDL}$ from the sequence $G_{PDDL}$.

In fact, the final state resulting from the plan's execution of each solvable sub-problem is exactly the initial state of the next sub-problem. Therefore, as shown in the for-loop in L. <ref>-<ref>, after solving each sub-problem $p$ with the planner $\Pi$ and the domain file $d$, we obtain a sub-plan $\pi'$ and the final state $s'_{PDDL}$ (L. <ref>). $s'_{PDDL}$ will then be assigned to $s_{PDDL}$ for the next sub-problem.
The following sub-problems can be formulated by replacing the initial state and goal (L. <ref>) and solved in the same way autoregressively until the last sub-problem is solved. The final task plan $\pi$ can be obtained by concatenating all sub-plans (L. <ref>). The task plans only consist of executable atomic actions.

$\Pi$, $d$, $p_0$, $G_{PDDL}$
$\pi \leftarrow \varnothing$
extract $s_{0, PDDL}, g_{0, PDDL}$ from $p_0$
$s_{PDDL} \leftarrow s_{0, PDDL}$
$g_{PDDL} \in G_{PDDL}$
$p \leftarrow$ replace $s_{0, PDDL}, g_{0, PDDL}$ in $p_0$ with $s_{PDDL}, g_{PDDL}$
$\pi', s'_{PDDL} \leftarrow \Pi(d, p)$
$\pi \leftarrow$ concat$(\pi, \pi')$
$s_{PDDL} \leftarrow s'_{PDDL}$

Autoregressive Sub-Task Planning

2*Domain     2*Scene     2*Model     2cSuccess Rate $[\%]$     3cPlan Length     2cPlanning Time $[s]$     2cExpanded Nodes

4-5 6-8 9-10 11-12             Orig.     Decomp.     Orig.     Decomp.     GT     Orig.     Decomp.     Orig.     Decomp.

Assembly     6*Parole     LLM-As-Planner     $0$     -     $\times$     -     6*47     -     -     -     -

LLM+P     $82$     -     $\textbf{47}$     -         $29.07$     -     $\textbf{1,379,036}$     -

SayPlan*     $0$     -     $\times$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $70$     $94$     $\textbf{47}$     $\textbf{47}$         $29.93$     $0.0121$     $\textbf{1,379,036}$     $581$

DELTA (GPT-4)     $96$     $\textbf{100}$     $\textbf{47}$     $\textbf{47}$         $28.91$     $\textbf{0.0120}$     $\textbf{1,379,036}$     $\textbf{576}$

DELTA (GPT-4-turbo)     $\textbf{100}$     $\textbf{100}$     $\textbf{47}$     $\textbf{47}$         $\textbf{28.86}$     $\textbf{0.0120}$     $\textbf{1,379,036}$     $\textbf{576}$

2-12     6*Allensville     LLM-As-Planner     $0$     -     $\times$     -     6*41     -     -     -     -

LLM+P     $8$     -     $\textbf{41}$     -         $52.47$     -     $\textbf{1,585,185}$     -

SayPlan*     $70$     -     $43$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $2$     $86$     $\textbf{41}$     $\textbf{41}$         $52.26$     $0.0138$     $\textbf{1,585,185}$     $\textbf{637}$

DELTA (GPT-4)     $76$     $96$     $\textbf{41}$     $\textbf{41}$         $51.99$     $0.0137$     $\textbf{1,585,185}$     $\textbf{637}$

DELTA (GPT-4-turbo)     $\textbf{92}$     $\textbf{100}$     $\textbf{41}$     $\textbf{41}$         $\textbf{51.79}$     $\textbf{0.0137}$     $\textbf{1,585,185}$     $\textbf{637}$

2-12     6*Shelbiana     LLM-As-Planner     $0$     -     $\times$     -     6*42     -     -     -     -

LLM+P     $82$     -     $\textbf{42}$     -         $49.67$     -     $\textbf{1,317,615}$     -

SayPlan*     $8$     -     $43$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $50$     $96$     $\textbf{42}$     $\textbf{45}$         $49.20$     $0.0149$     $\textbf{1,317,615}$     $703$

DELTA (GPT-4)     $88$     $90$     $\textbf{42}$     $\textbf{45}$         $49.23$     $\textbf{0.0148}$     $\textbf{1,317,615}$     $\textbf{682}$

DELTA (GPT-4-turbo)     $\textbf{98}$     $\textbf{100}$     $\textbf{42}$     $\textbf{45}$         $\textbf{48.58}$     $\textbf{0.0148}$     $\textbf{1,317,615}$     $\textbf{682}$

Table Setup     6*Parole     LLM-As-Planner     $0$     -     $\times$     -     6*33     -     -     -     -

LLM+P     $0$     -     $\times$     -         $\times$     -     $\times$     -

SayPlan*     $38$     -     $34$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4-turbo)     $\textbf{92}$     $\textbf{96}$     $\textbf{33}$     $\textbf{35}$         $\textbf{1.71}$     $\textbf{0.0098}$     $\textbf{109,208}$     $\textbf{404}$

2-12     6*Allensville     LLM-As-Planner     $0$     -     $\times$     -     6*39     -     -     -     -

LLM+P     $0$     -     $\times$     -         $\times$     -     $\times$     -

SayPlan*     $0$     -     $\times$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4-turbo)     $\textbf{82}$     $\textbf{88}$     $\textbf{39}$     $\textbf{39}$         $\textbf{40.57}$     $\textbf{0.0111}$     $\textbf{1,584,206}$     $\textbf{436}$

2-12     6*Shelbiana     LLM-As-Planner     $0$     -     $\times$     -     6*39     -     -     -     -

LLM+P     $0$     -     $\times$     -         $\times$     -     $\times$     -

SayPlan*     $0$     -     $\times$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4-turbo)     $\textbf{84}$     $\textbf{86}$     $\textbf{39}$     $\textbf{39}$         $\textbf{51.79}$     $\textbf{0.0141}$     $\textbf{1,826,112}$     $\textbf{567}$

Cleaning     6*Parole     LLM-As-Planner     $0$     -     $\times$     -     6*41     -     -     -     -

LLM+P     $0$     -     $\times$     -         $\times$     -     $\times$     -

SayPlan*     $8$     -     $\textbf{41}$     $0$         -     -     -     -

DELTA (GPT-3.5-turbo)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4)     $6$     $6$     $\textbf{41}$     $\textbf{45}$         $\textbf{5.77}$     $\textbf{0.0109}$     $\textbf{368,728}$     $\textbf{529}$

DELTA (GPT-4-turbo)     $\textbf{86}$     $\textbf{86}$     $\textbf{41}$     $\textbf{45}$         $11.48$     $0.0119$     $752,305$     $636$

2-12     6*Allensville     LLM-As-Planner     $0$     -     $\times$     -     6*39     -     -     -     -

LLM+P     $0$     -     $\times$     -         $\times$     -     $\times$     -

SayPlan*     $16$     -     $\textbf{39}$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4)     $16$     $16$     $\textbf{39}$     $\textbf{39}$         $19.91$     $0.0119$     $893,402$     $\textbf{607}$

DELTA (GPT-4-turbo)     $\textbf{46}$     $\textbf{74}$     $\textbf{39}$     $\textbf{39}$         $\textbf{17.90}$     $\textbf{0.0125}$     $\textbf{861,908}$     $689$

2-12     6*Shelbiana     LLM-As-Planner     $0$     -     $\times$     -     6*43     -     -     -     -

LLM+P     $0$     -     $\times$     -         $\times$     -     $\times$     -

SayPlan*     $4$     -     $\textbf{43}$     -         -     -     -     -

DELTA (GPT-3.5-turbo)     $0$     $0$     $\times$     $\times$         $\times$     $\times$     $\times$     $\times$

DELTA (GPT-4)     $16$     $16$     $\textbf{43}$     $\textbf{44}$         $57.14$     $\textbf{0.0121}$     $\textbf{2,187,261}$     $\textbf{602}$

DELTA (GPT-4-turbo)     $\textbf{66}$     $\textbf{94}$     $\textbf{43}$     $\textbf{44}$         $\textbf{53.78}$     $0.0132$     $\textbf{2,187,261}$     $743$

Comparison of all metrics of different approaches throughout all domains and scenes. Each metric is reported for both original (orig.) and decomposed (decomp.) problems. The ground truth (GT) plan lengths of each experiment are also provided, implying the optimal value. “-” means that the metric is not relevant for the current model, and “$\times$” implies that the metric is not applicable due to failure cases. The LLM used for the baselines is GPT-4-turbo. The results of plan length, planning time, and expanded nodes are averaged over the successful cases.

§ EVALUATION

§.§ Evaluation Domains

We designed four domains, one with a short-term task and the others with long-term tasks. The Laundry domain with a short-term task serves as the example for one-shot prompting, where the robot is asked to bring the dirty clothes and detergent to the washing machine and then bring them to the bedroom after washed.
The PC Assembly domain requires the robot to gather six different PC parts, i.e., a mainboard, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Random-Access Memory (RAM), a Solid-State-Drive (SSD) and a Power Supply Unit (PSU), distributed in the environment and bring them to the living room for assembly. It can be decomposed into six transportation and assembly sub-tasks.
In the Dining Table Setup domain, the robot should collect a plate, a fork, a knife, a spoon, a glass, and a flower distributed in different rooms, and place them on the dining table. Similarly, it also consists of six transportation sub-tasks.
In the House Cleaning domain, the robot should perform a long-term task to first dispose of a cola can, a banana peel, and a rotting apple in the rubbish bin in the kitchen, then mop the floor in the kitchen and living room and clean the mop immediately after cleaning each room. Finally, the robot will return to the hub to recharge the battery. The robot can only load one item at a time.

§.§ Baselines

We compare DELTA with three representative LLM-based robot task planning approaches: LLM-As-Planner, LLM+P [11], and SayPlan [14].

LLM-As-Planner is a naive approach. The LLM is queried with a prompt that comprises all information, including domain knowledge, environment, and goal descriptions, to generate a high-level task plan directly.

LLM+P [11] takes a problem description as input and generates a PDDL problem file. Subsequently, with a user-provided PDDL domain file, an automated task planner generates a high-level task plan. LLM+P can be treated as a subset of DELTA, which only has the problem generation step without pruning the scene graph and decomposing the long-term goal. In the following experiments, we also provide a pre-defined PDDL domain file as input.

SayPlan [14] has two stages: the semantic search stage first determines a minimal sufficient scene graph using an LLM by expanding room nodes with relevant items, then the iterative replanning stage queries an LLM to generate a high-level plan using the searched graph and can replan based on environmental feedback. Since there is no available open-source code of SayPlan, we implemented a simplified version, where we omit the semantic search stage by directly providing a pre-pruned minimal sufficient scene graph. In the following, we denote the simplified version as SayPlan*.
Since our approach focuses on high-level task planning and does not require replanning, for a fair comparison, we also use SayPlan* to generate high-level plans, and the number of replanning iterations of SayPlan* is set to $1$.

Illustration of the Shelbiana scene and the corresponding 3D scene graph used in the experiments [22]. The scene graph has three layers: floor (red), room (green), and item (blue) layers. Note that not all item nodes are visualized.

§.§ Dataset

We use four scene graphs from the 3D Scene Graph dataset [22]: Kemblesville ($9$ rooms, $16$ items) is paired with the Laundry domain as an example for one-shot prompting Allensville ($11$ rooms and $42$ items), Parole ($7$ rooms, $31$ items) and Shelbiana ($12$ rooms, $34$ items) are used to evaluate the rest three domains. Fig. <ref> illustrates the Shelbiana scene and the corresponding scene graph.

We implement the scene graphs as Python nested dictionaries.

§.§ Implementation and Parameters

We evaluate DELTA with three different pre-trained LLMs: GPT-3.5-turbo (version 1106), GPT-4 (version 0613) and GPT-4-turbo (version 1106-preview), with temperature parameter set as $0$ and top_p as $1$. The other baselines are evaluated with GPT-4-turbo. No LLM is fine-tuned.

We use Fast Downward (FD) [40] automated task planner with the default search configuration seq-opt-lmcut and the timeout of $60 s$.
Moreover, we use PDDLGym [41] to obtain the world states, and the plan validation tool VAL [42] to validate the correctness and executability of plans created by the FD planner. Each experiment is repeated with $50$ trials, resulting in $450$ trials crosswise evaluated with $3$ domains and $3$ scenes in total. All approaches are implemented in Python $3.8.13$. We run the experiments on a standard PC with an Intel Xeon W CPU at $3.40$ GHz and $32$ GB RAM.

Performance analysis of DELTA with different LLMs. Each step, success, and failure type is annotated with the number of trials. Problem Generation and Goal Decomposition are decoupled since the planning of original and decomposed problems are independent and executed parallelly with the same number of trials outgoing from Scene Graph Pruning.

Model     Domain Generation     Scene Graph Pruning     Problem Generation     Goal Decomposition     Total Time

GPT-3.5-turbo     $18.53 \pm 0.94$     $1.48 \pm 0.31$     $11.65 \pm 0.70$     $5.22 \pm 0.61$     $36.89 \pm 1.67$

GPT-4             $119.39 \pm 120.24$     $16.93 \pm 84.20$     $56.18 \pm 56.18$     $28.08 \pm 7.28$     $220.58 \pm 139.17$

GPT-4-turbo       $59.22 \pm 23.45$     $3.16 \pm 1.06$     $35.26 \pm 13.83$     $24.08 \pm 7.89$     $121.73 \pm 28.50$

Comparison of the response time of different LLMs in DELTA with House Cleaning domain and Shelbiana scene. The numbers are shown in $mean \pm std$ with unit $[s]$.

§.§ Results

We evaluate the approaches with four metrics: success rate, plan length, resulting planning time of the automated task planner, and the number of expanded nodes (nodes in a search tree created and explored during the process of searching for a solution to a planning problem, a lower number of expanded nodes implies a higher efficiency) reported by the automated task planner. A trial is successful if the VAL tool reports that the generated plan is valid, i.e., correct and executable.
Moreover, each metric is reported for both original and decomposed problems. The decomposed plan length shows the length of the concatenated plan, and the decomposed planning time refers to the total time of solving all sub-problems. Table <ref> shows the results of all experiments.

Unsurprisingly, LLM-As-Planner performs the worst among all approaches, which achieved no successful case at all. There is no doubt that the state-of-the-art LLMs still lack ability in graph and physical reasoning [17], therefore, directly using LLMs to generate executable high-level plans is extremely challenging and risky for long-term tasks without grounding with actionable knowledge.

By grounding natural language into formal planning language, LLM+P is able to achieve up to $82\%$ of success rate in PC Assembly domain and also able to reach the optimal plan length and number of expanded nodes. Solving the generated problem files also leads to a comparable planning time to our approach. However, LLM+P never succeed in Dining Table Setup and House Cleaning domains. Two leading failure cases are planner timeout ($239$ failures out of $450$ trials) and undeclared items ($86$ failures).

LLM+P consumes the original scene graphs containing over $30$ items. Although LLMs are mostly able to transfer the items from scene graphs to the PDDL problem files, the difficulty for the automated planner to solve problems increases exponentially with the number of items, resulting in exceeding the planner's timeout. On the other hand, the longer the input prompt, the more likely that the LLMs neglect important information, such as semantic attributes of items, and forget to translate them into PDDL predicates, which leads to incomplete problem files, such that the automated task planner fails to solve.

Compared to other domains, PC Assembly is semantically the simplest one, where all sub-tasks are similar transportation tasks, such that the LLMs are not required to generate unseen predicates other than “item_pickable”. Furthermore, Parole and Shelbiana scenes contain the least items. These explain the higher success rates of LLM+P in the corresponding experiments in comparison with the others.

SayPlan* expresses action knowledge as environment functions and environment states in its prompt and directly generates a high-level task plan. Table <ref> shows that, although SayPlan* has its highest success rate of $70\%$ in PC Assembly domain with Allensville scene, in many other experiments, its success rates are mostly lower than $10\%$. Despite using scene graphs as structured environment representations, without grounding action and environment knowledge into formal planning language, it is unable to guarantee the correctness and executability of the generated plan. The absence of proficient graph-based reasoning capability of LLMs often leads to infeasible actions, e.g., going to an unconnected room or picking an item while already holding one, which is particularly more critical in tackling long-term tasks.
Moreover, since we simplify SayPlan* by directly providing the pre-pruned minimal sufficient scene graphs, one can expect that the results of its full approach will not outperform those of the simplified version.

Finally, DELTA with GPT-4-turbo has achieved the highest success rate in both original and decomposed cases in all experiments. It also mostly achieves the shortest planning time, except for Parole and Shelbiana scenes in House Cleaning domain, where DELTA with GPT-4-turbo is merely $0.001 s$ slower than the same approach with GPT-4. In Parole scene, the planning time of DELTA with GPT-4-turbo in the original problem is ca. $2$ times slower than that with GPT-4. It is because GPT-4-turbo detailedly defines the action pre-conditions in the domain files, including negations, such as (not (floor_clean ?r)) as one of the pre-conditions of the mop_floor action. Although the uninstantiated predicates are automatically considered as negations, when defining them in PDDL actions, the planner needs to go through more states when solving a problem, which results in increased planning time as well as the number of expanded nodes.
However, DELTA with GPT-4-turbo also has a remarkably higher success rate of $86\%$ than that with GPT-4 of $6\%$ in the same experiment.

The differences between DELTA and LLM+P lie in the domain generation, scene graph pruning, and goal decomposition. Since the quality of the generated domain files cannot exceed those provided by human experts, therefore, the comparison of success rate in original problems between LLM+P and DELTA in Table <ref> implies that, having sparse scene graphs,

i.e., pruning irrelevant items, notably reduces the risk of LLMs producing unnecessary and possibly incorrect predicates translated from the irrelevant items. As such, the success rate can be increased due to fewer potential errors, and the planning time can be reduced since it needs to check fewer predicates.

On the other hand, decomposing the long-term goal leads to a significant reduction of planning time and expanded nodes, e.g., over $4,000$ times faster in planning time and over $3,600$ time less expanded nodes in House Cleaning domain with Shelbiana scene, which enables a considerable enhancement of the planning efficiency.

Ablation Study: we also evaluate DELTA with different LLMs, i.e., GPT-3.5-turbo, GPT-4 and GPT-4-turbo. The results from the Dining Table Setup and House Cleaning domains shown in Table <ref> imply an enormous performance gap between GPT-4-turbo and the lower versions, where GPT-4 barely succeeds, and GPT-3.5-turbo completely fails.
Additionally, Fig. <ref> shows an analysis of the success and failure cases for a better understanding of the performance of DELTA with different LLMs. Tendentiously, the lower the GPT's version, the more likely the LLM generates failures in the early steps of the experiments. Leading factors are $147$ failures of incorrect action that occurred when generating domain files with GPT-3.5-turbo as shown in Fig. <ref>, and $239$ out of $443$ trials failed when pruning scene graphs with GPT-4 as shown in Fig. <ref>.

Table <ref> lists a few representative examples of the response time of different LLMs in House Cleaning domain and Shelbiana scene. Interestingly, despite the low success rate in all experiments, GPT-3.5-turbo responses almost $6$ times faster than GPT-4 and over $3$ times faster than GPT-4-turbo.

§ CONCLUSION

We proposed DELTA, a novel task planning approach for mobile robots, which uses 3D scene graphs as environment representations, and utilizes LLMs to prune the irrelevant items in the scene graphs,

generate PDDL planning files,

and decompose the long-term task goals with Natural Language (NL) instructions. The grounding of NL and structured environment representations into formal planning language as well as the goal decomposition enables a significant enhancement of efficiency in automated task planning, and allows to achieve a considerably faster planning time and higher success rate compared to various baselines. For future work, we plan to validate our approach on a robot in a real-world environment and extend it to dynamic environments.

§ ACKNOWLEDGMENTS

This work was partly supported by the EU Horizon 2020 research and innovation program under grant agreement No. 101017274 (DARKO).

[1]
J. Achiam et al., “Gpt-4 technical report,” arXiv preprint
arXiv:2303.08774, 2023.

[2]
A. Chowdhery et al., “Palm: Scaling language modeling with pathways,”
Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113,

[3]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep
bidirectional transformers for language understanding,” arXiv preprint
arXiv:1810.04805, 2018.

[4]
H. Touvron et al., “Llama: Open and efficient foundation language
models,” arXiv preprint arXiv:2302.13971, 2023.

[5]
M. Aiello and I. Georgievski, “Service composition in the chatgpt era,”
Service Oriented Computing and Applications, pp. 1–6, 2023.

[6]
Y. Liu, L. Palmieri, I. Georgievski, and M. Aiello, “Human-flow-aware
long-term mobile robot task planning based on hierarchical reinforcement
learning,” IEEE Robotics and Automation Letters, 2023.

[7]
N. Aboki, I. Georgievski, and M. Aiello, “Automating a telepresence robot for
human detection, tracking, and following,” in Annual Conference
Towards Autonomous Robotic Systems.1em plus 0.5em minus 0.4emSpringer, 2023.

[8]
Y. Goel, N. Vaskevicius, L. Palmieri, N. Chebrolu, K. O. Arras, and
C. Stachniss, “Semantically informed mpc for context-aware robot
exploration,” in 2023 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS).1em plus 0.5em minus 0.4emIEEE,
2023, pp. 11 218–11 225.

[9]
W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot
planners: Extracting actionable knowledge for embodied agents,” in
International Conference on Machine Learning.1em plus 0.5em
minus 0.4emPMLR, 2022, pp. 9118–9147.

[10]
C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su,
“Llm-planner: Few-shot grounded planning for embodied agents with large
language models,” in Proceedings of the IEEE/CVF International
Conference on Computer Vision, 2023, pp. 2998–3009.

[11]
B. Liu et al., “Llm+ p: Empowering large language models with optimal
planning proficiency,” arXiv preprint arXiv:2304.11477, 2023.

[12]
Y. Ding, X. Zhang, C. Paxton, and S. Zhang, “Task and motion planning with
large language models for object rearrangement,” arXiv preprint
arXiv:2303.06247, 2023.

[13]
M. Ahn et al., “Do as i can, not as i say: Grounding language in
robotic affordances,” arXiv preprint arXiv:2204.01691, 2022.

[14]
K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf,
“Sayplan: Grounding large language models using 3d scene graphs for scalable
robot task planning,” in Conference on Robot Learning.1em plus
0.5em minus 0.4emPMLR, 2023, pp. 23–72.

[15]
Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan, “Autotamp: Autoregressive
task and motion planning with llms as translators and checkers,” arXiv
preprint arXiv:2306.06531, 2023.

[16]
V. Pallagani et al., “On the prospects of incorporating large language
models (llms) in automated planning and scheduling (aps),” arXiv
preprint arXiv:2401.02500, 2024.

[17]
K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati, “Large language
models still can't plan (a benchmark for llms on planning and reasoning about
change),” arXiv preprint arXiv:2206.10498, 2022.

[18]
D. McDermott et al., “Pddl-the planning domain definition language,”

[19]
G. Chalvatzaki, A. Younes, D. Nandha, A. T. Le, L. F. Ribeiro, and I. Gurevych,
“Learning to reason over scene graphs: a case study of finetuning gpt-2 into
a robot language model for grounded task planning,” Frontiers in
Robotics and AI, vol. 10, 2023.

[20]
Z. Ravichandran, L. Peng, N. Hughes, J. D. Griffith, and L. Carlone,
“Hierarchical representations and explicit memory: Learning effective
navigation policies on 3d scene graphs using graph neural networks,” in
2022 International Conference on Robotics and Automation (ICRA).1em plus 0.5em minus 0.4emIEEE, 2022, pp. 9272–9279.

[21]
C. Agia et al., “Taskography: Evaluating robot task planning over large
3d scene graphs,” in Conference on Robot Learning.1em plus
0.5em minus 0.4emPMLR, 2022.

[22]
I. Armeni et al., “3d scene graph: A structure for unified semantics,
3d space, and camera,” in Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV), 2019.

[23]
A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone, “3d dynamic scene
graphs: Actionable spatial perception with places, objects, and humans,” in
Robotics: Science and Systems (RSS), 2020.

[24]
A. Rosinol et al., “Kimera: From slam to spatial perception with 3d
dynamic scene graphs,” The International Journal of Robotics
Research, vol. 40, no. 12-14, pp. 1510–1546, 2021.

[25]
N. Hughes, Y. Chang, and L. Carlone, “Hydra: A real-time spatial perception
system for 3D scene graph construction and optimization,” 2022.

[26]
J. Wald, H. Dhamo, N. Navab, and F. Tombari, “Learning 3d semantic scene
graphs from 3d indoor reconstructions,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[27]
S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari, “Scenegraphfusion:
Incremental 3d scene graph prediction from rgb-d sequences,” in
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), June 2021, pp. 7515–7525.

[28]
Z. Wang, B. Cheng, L. Zhao, D. Xu, Y. Tang, and L. Sheng, “Vl-sat:
Visual-linguistic semantics assisted training for 3d semantic scene graph
prediction in point cloud,” in Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR), June 2023, pp.
21 560–21 569.

[29]
S. Koch, P. Hermosilla, N. Vaskevicius, M. Colosi, and T. Ropinski, “Sgrec3d:
Self-supervised 3d scene graph learning via object-level scene
reconstruction,” in Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV), 2024.

[30]
S. KKoch, P. Hermosilla, N. Vaskevicius, M. Colosi, and
T. Ropinski, “Lang3dsg: Language-based contrastive pre-training for 3d scene
graph prediction,” arXiv preprint arXiv:2310.16494, 2023.

[31]
Q. Gu et al., “Conceptgraphs: Open-vocabulary 3d scene graphs for
perception and planning,” arXiv preprint arXiv:2309.16650, 2023.

[32]
H. Chang et al., “Context-aware entity grounding with open-vocabulary
3d scene graphs,” in 7th Annual Conference on Robot Learning, 2023.

[33]
S. Koch, N. Vaskevicius, M. Colosi, P. Hermosilla, and T. Ropinski, “Open3dsg:
Open-vocabulary 3d scene graphs from point clouds with queryable objects and
open-set relationships,” arXiv preprint arXiv:2402.12259, 2024.

[34]
Z. Seymour, N. C. Mithun, H.-P. Chiu, S. Samarasekera, and R. Kumar,
“Graphmapper: Efficient visual navigation by scene graph generation,” in
2022 26th International Conference on Pattern Recognition
(ICPR).1em plus 0.5em minus 0.4emIEEE, 2022, pp. 4146–4153.

[35]
T. Brown et al., “Language models are few-shot learners,”
Advances in neural information processing systems, vol. 33, pp.
1877–1901, 2020.

[36]
T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano-Pérez,
and L. P. Kaelbling, “Pddl planning with pretrained large language models,”
in NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.

[37]
B. Chen et al., “Open-vocabulary queryable scene representations for
real world planning,” in 2023 IEEE International Conference on
Robotics and Automation (ICRA).1em plus 0.5em minus 0.4emIEEE,
2023, pp. 11 509–11 522.

[38]
D. Shah, B. Osiński, S. Levine et al., “Lm-nav: Robotic navigation
with large pre-trained models of language, vision, and action,” in
Conference on Robot Learning.1em plus 0.5em minus 0.4emPMLR, 2023, pp. 492–504.

[39]
S. Wang et al., “Less is more: Generating grounded navigation
instructions from landmarks,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2022, pp.
15 428–15 438.

[40]
M. Helmert, “The Fast Downward Planning System,” Journal of
Artificial Intelligence Research, vol. 26, pp. 191–246, 2006.

[41]
T. Silver and R. Chitnis, “Pddlgym: Gym environments from pddl problems,” in
International Conference on Automated Planning and Scheduling (ICAPS)
PRL Workshop, 2020.

[42]
R. Howey, D. Long, and M. Fox, “VAL: automatic plan validation, continuous
effects and mixed initiative planning using PDDL,” in 16th IEEE
International Conference on Tools with Artificial Intelligence.1em
plus 0.5em minus 0.4emIEEE, 2004, pp. 294–301.


# Neural Speaker Embeddings for Ultrasound-based Silent Speech Interfaces

###### Abstract

Articulatory-to-acoustic mapping seeks to reconstruct speech from a recording
of the articulatory movements, for example, an ultrasound video. Just like
speech signals, these recordings represent not only the linguistic content,
but are also highly specific to the actual speaker. Hence, due to the lack of
multi-speaker data sets, researchers have so far concentrated on speaker-
dependent modeling. Here, we present multi-speaker experiments using the
recently published TaL80 corpus. To model speaker characteristics, we adjusted
the x-vector framework popular in speech processing to operate with ultrasound
tongue videos. Next, we performed speaker recognition experiments using 50
speakers from the corpus. Then, we created speaker embedding vectors and
evaluated them on the remaining speakers. Finally, we examined how the
embedding vector influences the accuracy of our ultrasound-to-speech
conversion network in a multi-speaker scenario. In the experiments we attained
speaker recognition error rates below 3%, and we also found that the embedding
vectors generalize nicely to unseen speakers. Our first attempt to apply them
in a multi-speaker silent speech framework brought about a marginal reduction
in the error rate of the spectral estimation step.

Index Terms: silent speech interface, articulatory-to-acoustic mapping,
speaker embedding, x-vector

##  1 Introduction

Silent Speech Interfaces (SSI) aim to convert silent (mouthed) articulation to
audible speech [1, 2, 3]. Such SSI systems could aid the communication of the
speaking impaired (e.g. patients after laryngectomy). The theoretical
background is provided by articulatory-to-acoustic mapping (AAM), where
articulatory data is recorded while the subject is speaking, and machine
learning methods (typically deep neural networks (DNNs)) are applied to
predict the speech signal from the articulatory input. The set of articulatory
acquisition devices includes ultrasound tongue imaging (UTI) [4, 5, 6, 7, 8],
Magnetic Resonance Imaging (MRI) [9], electromagnetic articulography (EMA)
[10, 11, 12], permanent magnetic articulography (PMA) [13, 14, 15], surface
electromyography (sEMG) [16, 17, 18], electro-optical stomatography (EOS)
[19], lip videos [20, 21], or a multimodal combination of the above [22].

Although there are lots of studies on generating intelligible speech from the
above biosignals, most of these were conducted on relatively small databases
of just a single or a small number of speakers [1, 2, 3]. Meanwhile, all of
the articulatory tracking devices are obviously highly sensitive to the actual
speaker’s anatomy. A further source of variance may come from the possible
misalignment of the recording equipment. For example, for ultrasound
recordings, the probe fixing headset has to be remounted onto the speaker for
each recording session. This inevitably causes the recorded ultrasound videos
to become slightly misaligned between each recording session.

Although the properly working cross-session and cross-speaker methodologies
are still missing, there have already been studies in this direction. Kim et
al. investigated speaker-independent Silent Speech Recognition (SSR) using EMA
with 12 healthy and laryngectomized speakers [11]. For EMG-based recognition,
several signal normalization and model adaptation methods were investigated by
Maier-Hein et al. [16]. Janke et al. studied session-independent sEMG over 16
sessions of a speaker, and the results showed that sEMG is quite robust to
minor changes in the electrode placement [17]. Wand et al. utilized domain-
adversarial DNN training to increase the session-independency of their EMG-
based speech recognizer [18]. For EOS-based SSR with a small vocabulary, Stone
and Birkholz found the speaker-independent average word accuracy to be
relatively stable, varying between 56–62% [19].

Ultrasound-based SSI systems, however, might be less robust, as slight changes
in probe positioning causes shifts and rotations in the resulting image. To
this end, we examined on the session dependency of UTI-based direct speech
synthesis, and we proposed a simple session adaptation method [23]. Ribeiro et
al. experimented with the classification of UTI images of phonetic segments
[24], and found that speaker-dependent systems perform much better than
speaker-independent ones, but adding speaker information in a simple form
(e.g. mean ultrasound image) helps the model generalize to unseen speakers.
The same authors reported that unsupervised model adaptation can improve the
results for silent speech (but not for modal speech) [6]. They also performed
multi-speaker recognition and synthesis experiments where they applied
$x$-vectors for speaker conditioning – but they extracted the $x$-vectors from
the acoustic data and not from the ultrasound [25].

###  1.1 Speaker embedding vectors

Just like the above-mentioned biosignals, speech signals also contain speaker-
specific factors that hurt the cross-speaker performance of speech processing
systems. Various DNN training and adaptation methods have been proposed as
remedies, perhaps the simplest being is to use auxiliary input features that
encapsulate information on speaker characteristics. The classic such
representation was the $i$-vector [26], but its role has recently been taken
by the $x$-vector [27]. The $x$-vector can be created by training a DNN for
speaker classification, and then the activation values of an upper hidden
layer are used as the speaker embedding vector in other tasks. Our goal here
is to adjust this $x$-vector scheme to ultrasound tongue videos. To this aim,
we train a DNN for speaker classification, using 3D blocks of adjacent
ultrasound frames as input. First, we report the speaker recognition accuracy
of this network in 5.1. Next, we evaluate the speaker embedding vector
provided by this network on a separate set of speakers via very simple speaker
recognition experiments in 5.2. Finally, we attempt to apply the embedding
vectors as auxiliary input for a second network which is trained to perform
speech synthesis, more precisely, to estimate spectral vectors from the
ultrasound frames (5.3). Before the experiments, brief descriptions are given
regarding our SSI framework, $x$-vectors and the experimental conditions in
Sections 2, 3 and 4, respectively.

##  2 The SSI framework

We presented our approach for creating an ultrasound-based SSI in [5], so we
just give a brief overview here. As Fig 1 shows, the input to our system is a
sequence of ultrasound tongue imaging (UTI) frames, and the target sequence is
a speech signal. This is a sequence-to-sequence mapping problem, which could
be addressed by sophisticated encoder-decoder networks that would not even
require aligned training data [25]. However, as we have synchronized input-
output samples, most authors apply simpler networks that perform the mapping
frame by frame [28, 5]. The optimal output representation is also a matter of
choice. While training DNNs that generate speech signals directly is feasible
[29], it would require large amounts of training data. Alternatively, one can
use a dense spectral representation as the training target that can be
converted to speech [4]. Recently, speech synthesis technology has introduced
neural vocoders for synthesizing speech from spectrograms [29]. The main
advantage of applying these in the SSI task is that that we can borrow large
pre-trained networks for the speech synthesis step, and we have to deal only
with the ultrasound-to-spectrum mapping task. In a recent study we used
WaveGlow [5], and we obtained higher quality speech than with standard
vocoders [4].

Figure 1: Schematic diagram of the UTI-to-speech conversion process applied in
our SSI model.

As WaveGlow requires a mel-spectrogram with 80 spectral components as input,
our DNN was trained to estimate such spectral vectors from the UTI video in a
frame-by-frame manner. The input of our network is a 3D array of consecutive
images, and the output is a 80-dimensional spectral vector. The convolutional
(3D-CNN) network structure that we applied here [30] was the same as the
lower, ’frame-level’ part of the $x$-vector network, so we delay its
presentation to the next section. As here the task is to estimate spectral
vectors, we used a linear output layer and the network was trained to minimize
the mean-squared error (MSE) of the regression task.

##  3 Speaker embedding vectors for ultrasound tongue imaging

The $x$-vector concept was introduced by Snyder et al., motivated by the goal
of replacing the previous Gaussian-based $i$-vector approach with a purely
neural solution [27]. The basis behind the $x$-vector is a DNN that is trained
to discriminate speakers. The network structure is unusual in the sense that
it consists of three main parts. The lower layers – typically a time-delay
network (TDNN [31]) – operate on the level of frames. Then, the subsequent
temporal pooling layer aggregates statistics over the frames of a given speech
segment or utterance. This layer may collect only the mean values [32], the
mean and the standard deviation [27, 33, 34] or it may even apply a
sophisticated attention mechanism [35, 36]. The aggregated values are
processed further by several fully connected layers. As these layer operate on
the segment level, after training they can produce a fixed-size speaker
embedding vector even from utterances of variable lengths. The embedding
vector is typically obtained as the linear activation output of the fully
connected layer right below or two layers below the softmax output [27].

Figure 2: Illustration of the UTI-based $x$-vector network.

We adjusted the $x$-vector DNN to operate with a sequence of ultrasound images
instead of a sequence of speech feature vectors as follows (see also Fig 2).

The frame-level part has the same structure as the network we apply in the
spectral estimation step of Fig 1 [30]. Its input consists of 21 consecutive
images, which are processed by a 3D convolutional layer in 5-image blocks,
using a relatively large stride size of 4 along the time axis. The subsequent
convolutional layers apply the 3D convolution in a decomposed “(2+1)D” form,
first processing these blocks locally, and aggregating their content along
time only at higher layers. This architecture was motivated by the findings of
Tran et al. for video recognition [37], and also by the success of TDNNs in
speech recognition [31, 38]. The top layer of the frame-level part is a dense
layer that produces a local output vector at each frame position (ie., for the
center frame of the input block).

The statistical pooling layer performs a simple average pooling. We leave more
complex solutions such as standard deviation pooling or attention-based
weighting for future work.

The segment-level part consists of two fully connected layers and a softmax
output layer that has one output neuron for each speaker in the training set.
The neural speaker embedding vector for a given input segment or utterance is
extracted from one of these dense layers (see the experiments in Section 5).

##  4 Experimental Set-Up

In the experiments we used the TaL80 corpus [25], which contains ultrasound,
speech and lip video recordings from 81 speakers. Apart from the silent speech
experiments, the speech signals were also recorded in parallel with the
ultrasound, and here we used these synchronized ultrasound and speech tracks.
The ultrasound was recorded using Articulate Instruments’ Micro system that
captures a midsaggital view of the tongue at a frame rate of 82 fps. The raw
ultrasound images contain 64 x 842 data values, which were resized to 64 x 128
pixels. More details about the recording process can be found in [25].

We divided the data into two sets, so we used 50 speakers to create the
$x$-vector network, and we held out 31 speakers to train and evaluate the SSI
network. Although the $x$-vector network is able to handle inputs with
different sizes, for technical simplicity we chose to work with short uniform
2-second long chunks from each recording (similar to Shon et al. [34]). To
train the $x$-vector network, we extracted such 2-second (164 frames) chunks
from each training file of the 50-speaker subset of the corpus, resulting in
76 chunks from each speaker on the average (minimum 65, maximum 85).
Altogether we obtained 3800 such blocks, from which 2298 were used for
training, 357 for development, and 1145 blocks for testing.

The network used for the spectral estimation task in the SSI and the frame-
level part of the $x$-vector network were structurally identical, consisting
of 4 3D-convolutional layers (with a MaxPooling layer after every second
convolution), and a fully connected layer that aggregates the convolutional
outputs (the exact model parameters can be found in our earlier work [30]).
The fully connected hidden layer was followed by a linear output layer for the
spectral estimation task, while in the $x$-vector network the fully connected
layer outputs were aggregated along time by average pooling. The segment-level
part consisted of two fully connected layers (FC#1 and FC#2 in Fig 2) with
sizes of 500 and 250, and the softmax output layer contained 50 neurons,
corresponding to the 50 speakers of the train set. Both networks were trained
using the Adam optimizer with a learning rate of 0.0002, using a batch size of
100 for the SSI network, and batch sizes of 4-16 for the $x$-vector network
(using smaller batch sizes for longer segments). The nonlinearity applied in
all hidden layers was the swish function.

##  5 Results and Discussion

Table 1: Speaker recognition error rates for the 50-speaker set as a function of the input segment duration. Segment length |  Aggregated | Error rate
---|---|---
(frames & seconds) | frames | (Dev Set)
21 (0.25 sec) | 1 | 3.16%
41 (0.50 sec) | 21 | 2.87%
82 (1.0 sec) | 62 | 2.10%
164 (2.0 sec) | 144 | 1.96%

###  5.1 Training the $x$-vector network

In the first experiment we were interested to measure the speaker
classification accuracy of our $x$-vector network, and how it is influenced by
the input duration. The motivation for the aggregation step in the original
$x$-vector model is that normally we have not just a single frame but whole
utterances (at least a word) from a given speaker, and that certain phonetic
segments may be less speaker-discriminative than others [34] – the most
trivial example are the silent parts. But the situation is different when the
input is an ultrasound tongue video, as the recording device always returns an
image of the tongue, even when the speaker remains silent, and thus longer
segments may not be necessary. In the first experiment we gradually increased
the size of the input segment from 21 frames (when the lower part of the
network returns one frame-level output, so effectively no temporal pooling
occurs) to 164 frames (2 seconds).

Table 1 shows the speaker recognition error rates on the development set of
the 50-speaker corpus for different durations of temporal pooling. As we
expected, we obtained a very low error rate already with just a single-frame
output, which shows that the UTI videos are very speaker-specific. Gradually
increasing the segment length and thus the number of aggregated frames
improves the results, but the improvements are smaller than those obtained,
for example, in language recognition [39].

Figure 3: Normalized histogram of the cosine distances for randomly chosen
same-speaker and different-speaker $x$-vector pairs from the 31-speaker set.

###  5.2 Validating the $x$-vectors on an independent speaker set

The second experiment sought to test the generalization ability of the
$x$-vector. We were worried about overfitting the training speakers, as the
popular $x$-vector implementations are trained with orders of magnitudes more
training data and typically with more than a thousand speakers [27]. Thus, to
validate our $x$-vector DNN, we performed a speaker recognition experiment on
the dataset of the 31 speakers not used during $x$-vector training.

Using the speaker embeddings to recognize or verify a new set of speakers is
not trivial, as the speakers will be different from those seen during
training. For optimal performance, the embedding vectors are typically post-
processed by factor analysis or dimension reduction methods [27, 40]. A simple
and fast, though somewhat sub-optimal solution is to calculate the similarity
score of two speakers based on the cosine similarity of their embedding
vectors [40]. According to this, we performed a simple speaker recognition
experiment with a 1-nearest neighbor classifier using the cosine distance as
follows. We fused the development and the test sets of the 31 speakers, and
calculated the $x$-vectors for each segment. Then we performed leave-one-out
classification, that is, to each vector in this set we found the closest one
(excluding itself). The classification was considered correct if the speaker
IDs of the two segments were identical. We intentionally used the simplest
possible 1-NN classifier, as it is very sensitive to the accuracy of the
distance function applied, and hence to the accuracy of the underlying
$x$-vectors.

Snyder et al. defined the $x$-vector as the linear output of the first fully
connected layer after statistical pooling [27]. However, some authors
questioned whether this is the optimal strategy, and also whether dropping the
nonlinearity is required [34]. So we also examined which fully connected layer
gives the best result, and if the nonlinearity is necessary or not. The
results are shown in Table 3. As can be seen, all embedding extraction methods
resulted in very low speaker recognition error rates around 1-2%. The best
result was obtained when the speaker embedding was produced by the lower fully
connected layer (FC#2) without the nonlinearity. This coincides with the
findings of Snyder et al. [27] and Shon et al. [34].

To further demonstrate the speaker discriminative abilities of the $x$-vectors
on a new set of speakers, a histogram of the cosine distances is shown in Fig
3 for 10000 randomly selected same-speaker and different-speaker vector pairs
from the 31-speaker dev+test subset. Although the distributions overlap
slightly, this figure also suggests that the embedding vectors behave as
expected, that is, vectors from the same speaker are closer to each other than
vectors from different speakers.

Table 2: Speaker recognition error rates for the held-out 31 speakers using
1-NN leave-one-out testing.

Embedding layer | FC#1 | FC#2
---|---|---
Activation | swish | linear | swish | linear
Error rate | 0.96% | 0.96% | 2.03% | 0.70%

Table 3: Mean squared error (MSE) for the SSI spectral estimation task in the
single-speaker and multi-speaker scenarios.

SSI Train+Test |  Size of training | MSE
---|---|---
configuration |  set (frames) | Dev | Test
single-speaker | 254306 | 0.256 | 0.265
multi-speaker | 305040 | 0.603 | 0.669
multi-spk + Xvec | 0.589 | 0.653

###  5.3 Applying the speaker embedding in speech synthesis

In the last experiment we attempted to use the $x$-vector as an auxiliary
input for the spectral estimator network of our SSI system (cf. Fig 1). As the
baseline, we trained the net with the single-speaker TaL1 corpus. The obtained
mean squared error rates (MSE) shown in Fig 3 are around 0.26, which results
in a speech signal with a mel-cepstral distortion (MCD) of 3.12 after speech
synthesis. This corresponds to a low-quality, but intelligible speech [5]. For
multi-speaker training and testing we used samples from the 31-speaker subset.
We did not use all the available data from the 31 speakers to demonstrate how
the accuracy of the SSI network drops when switching to a multi-speaker
scenario with a similar amount of training data. Table 3 shows that the multi-
speaker setup led to a drastically larger MSE. We note that the multi-speaker
scenario is still speaker-dependent in the sense that the training,
development and test sets are from the same speakers. Even larger performance
drops can be expected in a speaker-independent configuration [24].

Finally, we re-trained the multi-speaker model with the $x$-vector as
auxiliary input. However, the optimal way of combining the ultrasound input
with the $x$-vector is not trivial. As the ultrasound images are processed by
convolution, simple concatenation was not an option. We decided to inject the
$x$-vector into the network only after the convolutional layers, which were
initialized by transfer learning from the multi-speaker model. However, this
solution might be suboptimal and requires further studies. We mention that
Ribeiro et al. used the mean ultrasound frame to represent the speaker, so
they could add this image to the CNN input as a second channel [24].

The bottom row of Table 3 shows that although the introduction of the
$x$-vector resulted in a consistent improvement for both the development and
the test sets, this improvement is marginal. Ribeiro et al. reported similarly
small gains from using the speaker mean as the speaker-characteristic input
[24].

##  6 Conclusions

Here, we adjusted the $x$-vector framework of speech processing to ultrasound
tongue videos to create a speaker-characteristic embedding vector. We modified
the network architecture to process videos as input, and trained the network
for speaker recognition. We obtained very low speaker recognition error rates,
and our embedding vectors also seem to generalize well to new speakers.
However, our first attempts to apply the embedding vector in a multi-speaker
SSI scenario resulted in just a minimal improvement, showing that further
studies are required on the proper application of the $x$-vector in this
field.

##  7 Acknowledgements

This study was supported by grant NKFIH-1279-2/2020 of the Ministry for
Innovation and Technology, Hungary, and by the Ministry of Innovation and the
NRDI Office within the framework of the Artificial Intelligence National
Laboratory Programme and also through project FK 124584. The GPU card used was
donated by the NVIDIA Corporation.

## References

  * [1] B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, and J. S. Brumberg, “Silent speech interfaces,” _Speech Communication_ , vol. 52, no. 4, pp. 270–287, 2010.
  * [2] T. Schultz, M. Wand, T. Hueber, D. J. Krusienski, C. Herff, and J. S. Brumberg, “Biosignal-based spoken communication: A survey,” _IEEE/ACM Trans. ASLP_ , vol. 25, no. 12, pp. 2257–2271, 2017.
  * [3] J. A. Gonzalez-Lopez, A. Gomez-Alanis, J. M. Martin Donas, J. L. Perez-Cordoba, and A. M. Gomez, “Silent Speech Interfaces for Speech Restoration: A Review,” _IEEE Access_ , vol. 8, pp. 177 995–178 021, 2020\.
  * [4] T. G. Csapó, T. Grósz, G. Gosztolya, L. Tóth, and A. Markó, “DNN-based ultrasound-to-speech conversion for a silent speech interface,” in _Proc. Interspeech_ , 2017, pp. 3672–3676.
  * [5] T. G. Csapó, C. Zainkó, L. Tóth, G. Gosztolya, and A. Markó, “Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis,” in _Proc. Interspeech_ , 2020, pp. 2727–2731.
  * [6] M. S. Ribeiro, A. Eshky, K. Richmond, and S. Renals, “Silent versus modal multi-speaker speech recognition from ultrasound and video,” in _submitted to Interspeech_ , 2021.
  * [7] A. H. Shandiz, L. Tóth, G. Gosztolya, A. Markó, and T. G. Csapó, “Improving neural silent speech interface models by adversarial training,” in _The International Conference on Artificial Intelligence and Computer Vision_. Springer, 2021, pp. 430–440.
  * [8] A. H. Shandiz and L. Tóth, “Voice activity detection for ultrasound-based silent speech interfaces using convolutional neural networks,” _arXiv preprint arXiv:2105.13718_ , 2021.
  * [9] Y. Yu, A. H. Shandiz, and L. Tóth, “Reconstructing speech from real-time articulatory mri using neural vocoders,” _arXiv preprint arXiv:2104.11598_ , 2021.
  * [10] J. Wang, A. Samal, J. Green, and F. Rudzicz, “Sentence recognition from articulatory movements for silent speech interfaces,” in _Proc. ICASSP_ , 2012, pp. 4985–4988.
  * [11] M. Kim, B. Cao, T. Mau, and J. Wang, “Speaker-Independent Silent Speech Recognition From Flesh-Point Articulatory Movements Using an LSTM Neural Network,” _IEEE/ACM Trans. ASLP_ , vol. 25, no. 12, pp. 2323–2336, 2017\.
  * [12] F. Taguchi and T. Kaburagi, “Articulatory-to-speech conversion using bi-directional long short-term memory,” in _Proc. Interspeech_ , 2018, pp. 2499–2503.
  * [13] M. J. Fagan, S. R. Ell, J. M. Gilbert, E. Sarrazin, and P. M. Chapman, “Development of a (silent) speech recognition system for patients following laryngectomy,” _Medical Engineering and Physics_ , vol. 30, no. 4, pp. 419–425, 2008.
  * [14] J. A. Gonzalez, L. A. Cheah, A. M. Gomez, P. D. Green, J. M. Gilbert, S. R. Ell, R. K. Moore, and E. Holdsworth, “Direct speech reconstruction from articulatory sensor data by machine learning,” _IEEE/ACM Trans. ASLP_ , vol. 25, no. 12, pp. 2362–2374, 2017.
  * [15] J. A. Gonzalez-Lopez, M. Gonzalez-Atienza, A. Gomez-Alanis, J. L. Perez-Cordoba, and P. D. Green, “Multi-view Temporal Alignment for Non-parallel Articulatory-to-Acoustic Speech Synthesis,” in _Proc. IberSPEECH_ , 2021, pp. 230–234.
  * [16] L. Maier-Hein, F. Metze, T. Schultz, and A. Waibel, “Session independent non-audible speech recognition using surface electromyography,” in _Proc. ASRU_ , 2005, pp. 331–336.
  * [17] M. Janke, M. Wand, K. Nakamura, and T. Schultz, “Further investigations on EMG-to-speech conversion,” in _Proc. ICASSP_ , 2012, pp. 365–368.
  * [18] M. Wand, T. Schultz, and J. Schmidhuber, “Domain-Adversarial Training for Session Independent EMG-based Speech Recognition,” in _Proc. Interspeech_ , 2018, pp. 3167–3171.
  * [19] S. Stone and P. Birkholz, “Cross-speaker silent-speech command word recognition using electro-optical stomatography,” in _Proc. ICASSP_ , online, 2020, pp. 7849–7853.
  * [20] A. Ephrat and S. Peleg, “Vid2speech: Speech reconstruction from silent video,” in _Proc. ICASSP_ , 2017, pp. 5095–5099.
  * [21] D. Michelsanti, O. Slizovskaia, G. Haro, E. Gómez, Z.-H. Tan, and J. Jensen, “Vocoder-Based Speech Synthesis from Silent Videos,” in _Proc. Interspeech_ , 2020, pp. 3530–3534.
  * [22] J. Freitas, A. J. Ferreira, M. A. T. Figueiredo, A. J. S. Teixeira, and M. S. Dias, “Enhancing multimodal silent speech interfaces with feature selection,” in _Proc. Interspeech_ , 2014, pp. 1169–1173.
  * [23] G. Gosztolya, T. Grósz, L. Tóth, A. Markó, and T. G. Csapó, “Applying DNN Adaptation to Reduce the Session Dependency of Ultrasound Tongue Imaging-Based Silent Speech Interfaces,” _Acta Polytechnica Hungarica_ , vol. 17, no. 7, pp. 109–124, 2020.
  * [24] M. S. Ribeiro, A. Eshky, K. Richmond, and S. Renals, “Speaker-independent Classification of Phonetic Segments from Raw Ultrasound in Child Speech,” in _Proc. ICASSP_ , 2019, pp. 1328–1332.
  * [25] M. Ribeiro, J. Sanger, J.-X. Zhang, A. Eshky, A. Wrench, K. Richmond, and S. Renals, “Tal: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos,” _arXiv preprint arXiv:2011.09804_ , 2020\.
  * [26] A. Senior and I. Lopez-Moreno, “Improving DNN speaker-independence with I-vector inputs,” in _Proc. ICASSP_ , 2014, pp. 225–229.
  * [27] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in _Proc. ICASSP_ , 2018, pp. 5329–5333.
  * [28] E. Tatulli and T. Hueber, “Feature extraction using multimodal convolutional neural networks for visual speech recognition,” in _Proceedings of ICASSP_ , 2017, pp. 2971–2975.
  * [29] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A flow-based generative network for speech synthesis,” in _Proc. ICASSP_ , 2019, pp. 3617–3621.
  * [30] L. Tóth and A. H. Shandiz, “3D convolutional neural networks for ultrasound-based silent speech interfaces,” in _Proc. ICAISC_. Springer, 2020, pp. 159–169.
  * [31] V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in _Proc. Interspeech_ , 2015, pp. 3214–3218.
  * [32] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu, “Deep Speaker: an End-to-End Neural Speaker Embedding System,” _arXiv e-prints_ , p. arXiv:1705.02304, 2017.
  * [33] D. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur, “Deep neural network-based speaker embeddings for end-to-end speaker verification,” in _Proc. (SLT)_ , 2016, pp. 165–170.
  * [34] S. Shon, H. Tang, and J. R. Glass, “Frame-level speaker embeddings for text-independent speaker recognition and analysis of end-to-end model,” in _Proc. SLT_ , 2018, pp. 1007–1013.
  * [35] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive statistics pooling for deep speaker embedding,” in _Proc. Interspeech_ , 2018, pp. 2252–2256.
  * [36] Z. Y., T. Ko, D. Snyder, B. Mak, and D. Povey, “Self-attentive speaker embeddings for text-independent speaker verification,” in _Proc. Interspeech_ , 2018, pp. 3573–3577.
  * [37] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in _Proc. CVPR_ , 2018.
  * [38] L. Tóth, “Combining time- and frequency-domain convolution in convolutional neural network-based phone recognition,” in _Proc. ICASSP_ , 2014, pp. 190–194.
  * [39] D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, D. Povey, and S. Khudanpur, “Spoken language recognition using x-vectors,” in _Proc. Odyssey 2018 The Speaker and Language Recognition Workshop_ , 2018, pp. 105–111.
  * [40] N. Dehak, R. Dehak, P. Kenny, N. Brummer, P. Ouellet, and P. Dumouchel, “Support vector machines versus fast scoring in the low-dimensional total variability space for speaker verification,” in _Proc. Interspeech_ , 2009, pp. 1559–1562.