Language agents powered by large language models (LLMs) are increasingly valuable as decision-making tools in domains such as gaming and programming.
However, these agents often face challenges in achieving high-level goals without detailed instructions and in adapting to environments where feedback is delayed.
In this paper, we present , a novel automatic approach designed to enhance agents' capabilities to achieve high-level goals with limited human prior and environmental feedback.
The core concept of involves adaptively breaking down a high-level goal into a tree structure of more practical subgoals during the interaction with environments while identifying the most useful subgoals and progressively updating this structure.
Experimental results demonstrate that significantly enhances the performance of language agents across various tasks, including competitive, cooperative, and deferred feedback environments.[Project page: <https://selfgoal-agent.github.io>.]

§ INTRODUCTION

The advancement of large language models (LLMs) [1, 2, 3] has enabled the construction of autonomous language agents (or LLM-based agents) to solve complex tasks in dynamic environments without task-specific training.
In reality, these autonomous agents are often tasked with very broad, high-level goals, such as winning the most money or succeeding in a competition, whose ambiguous nature and delayed reward raise great challenges for autonomous task-solving.
More importantly, it is not practical to frequently train these models to adapt to new goals and tasks [4, 5, 6].
Therefore, a critical question arises:
How can we enable autonomous language agents to consistently achieve high-level goals without training?

Previous works focus on creating two types of auxiliary guidance in the instructions for language agents to achieve high-level goals in tasks: prior task decomposition and post-hoc experience summarization.
The former involves decomposing the task before acting, utilizing prior knowledge from LLMs to break down high-level goals into more tangible subgoals related to specific actions at hand [7, 4, 8, 9].
However, this line of work does not ground these subgoals into the environment during interaction, resulting in the loss of empirical guidance.
In contrast, the latter allows agents to interact directly with environments and summarize valuable experiences from history [10, 11, 12, 13], , “X contributes to Y”.
However, the difficulty of inducing rules from experience causes the guidance to be simple and unstructured, making it difficult to prioritize or adjust strategies effectively.

A natural solution to combine the best of both worlds is to dynamically decompose the task and its high-level goal during interaction with the environment.
This approach requires an agent to build and use guidelines that vary in detail and aspect.
A tree structure is ideal for this requirement, as it allows hierarchical organization, providing both broad overviews and detailed guidance as needed.
However, this approach presents two major challenges:

* Not all nodes are relevant to the current context during task execution, which requires selecting the most suited nodes to guide current actions.
For example, watch for bargains is a more prudent choice than bid on the most expensive item when budget is tight;

* The granularity of guidance provided by nodes increases with tree depth, yet the appropriate detail level varies across scenarios, making a fixed tree depth not general.
For example, a generic guideline like earn more money is not useful in auctions.

To tackle these challenges, we propose , a self-adaptive framework for a language agent to utilize both prior knowledge and environmental feedback to achieve high-level goals.
The main idea is to build a tree of textual subgoals, where agents choose appropriate ones as the guidelines to the prompt based on the situation.
Specifically, as shown in Figure <ref>, is featured with two main modules to operate a , which is constructed, updated, and utilized during task execution:

* Search Module is prompted to select the top-K most suited nodes of goals based on the provided current state and existing nodes in , which utilizes the prior knowledge of LLMs;

* Decomposition Module breaks down a goal node into a list of more concrete subgoals as subsequent leaves, ensuring an adaptive self-growth of . Note that we filter out the redundant nodes during decomposition based on the textual similarity between new ones and the existing nodes of goals;

* Act Module takes as input the selected subgoals as guidelines, and prompts LLMs for actions for the current state.

Extensive experiments in various competition and collaboration scenarios show that provides precise guidance for high-level goals and adapts to diverse environments, significantly improving language agent performance.

In summary, our contributions in this paper are as follows:

* We target the challenge of enabling autonomous language agents to consistently achieve high-level goals without the need for frequent retraining.

* We introduce , a self-adaptive framework that constructs, updates, and utilizes a to dynamically decompose a task's high-level goals into subgoals during interaction with the environment.

* We conduct extensive experiments in both collaborative and competitive scenarios where agents tend to deviate from their goals. The results demonstrate that significantly enhances the capability of language agents to adhere to high-level goals consistently.

§ RELATED WORK

Learning from Feedback
LLMs have become a promising tool for building goal-directed language agents [14].
With textual input that includes the world state, task, and interaction history, language agents are to decide the next action to achieve a goal [15, 16].
Studies have explored enhancing the reasoning and planning abilities of language agents through feedback from environments.
For example, Reflexion [17] enables an agent to reflect on its failures and devise a new plan that accounts for previous mistakes.
Similarly, Voyager [18] operates in Minecraft, developing a code-based skill library from detailed feedback on its failures.
Recent works [11, 19] analyze both failed and successful attempts, summarizing a memory of causal abstractions.
However, learnings directly from feedback are often too general and not systematic, making it difficult to prioritize strategies effectively.

LLMs for Decision Making
LLMs are increasingly used as policy models for decision-making in interactive environments such as robotics [20, 21, 22], textual games [23, 24, 25, 26], and social tasks [27].
However, the goals in these environments, like find a fruit in ScienceWorld [28], are often simple and specific.
For long-term, high-level goals, LLMs struggle to perform effectively [29, 30], and additional modules are needed for support[4].
In our work, we use a method that does not require updating LLM parameters, enabling language agents to consistently pursue high-level goals during interactions with environments.

Decomposition and Modularity
Decomposing complex decision-making tasks into sub-tasks is a traditional method that enhances LLM task-solving capabilities [31, 32].
Approaches like Hierarchical Task Networks leverage domain knowledge, including a hand-specified library of plans, to simplify complex problems [33].
Recently, some studies have assigned LLMs the role of decomposing goals.
For example, Decomposed Prompting [34] uses a few-shot prompting approach to tackle multi-step reasoning tasks by breaking them into a shared library of prompts.
OKR-Agent [4] utilizes self-collaboration and self-correction mechanisms, supported by hierarchical agents, to manage task complexities.
ADAPT [6] enables LLMs to recursively re-decompose goals based on feedback in decision-making tasks.
However, these approaches often decompose tasks before interaction with the environments, resulting in a lack of grounded, dynamic adjustment.
To address this, we aim to combine modular goal decomposition with learning from environmental feedback.

§ METHODOLOGY

An overview of , illustrated with a bargaining example. The agent interacts with environments, and make actions based on environmental feedback and the dynamically constructs, utilizes and updates with Search and Decompose Modules.

When executing complex tasks with high-level goals (, forecast future stock prices), humans usually decompose it into specific detailed subgoals (, gather historical price data and adjust predictions based on recent market events) for effective execution [35].
Inspired from this idea, we propose in this paper, which is a non-parametric learning approach for language agents to exploit and achieve high-level goals.
conducts a top-down hierarchical decomposition of the high-level goal, with a tree of nodes representing useful guidance for decision-making.

In this section, we first outline how works in $\mathsection$<ref>.
Next, we explain the details of three key modules (Search, Decompose and Act) in that help maintain a tree of subgoals () in $\mathsection$<ref> and guide task execution.

§.§ Overview of

Problem Formulation: Tasks with High-level Goals
First, we formulate the features of our studied tasks, requiring an agent to interact with a dynamic environment and evaluated based on the achievement of the high-level goal.
We focus on the scenarios where an actor model $M_a$ aims to achieve a high-level goal $g_0$ in an environment $E$ through interaction.
The policy employed by $M_a$ is denoted as $\pi_\theta$.
At each timestep $t$, $\pi_\theta$ generates an action $a_t$, and the environment $E$ returns a state $s_t$.
This action-state pair $\{a_t, s_t\}$ is then utilized to update $\pi_\theta$.
Note that also supports accomplishing long-horizon tasks that do not always have immediate rewards.
In this case, only by completing the task $M_a$ will be evaluated with a score according to the achievement of the goal $g_0$.

Workflow of
is a non-parametric learning algorithm for language agents, i.e., without parameter update.
The workflow of is shown at Algorithm <ref>, which models $\pi_\theta = p$ by setting $p$ as the instruction prompt provided to $M_a$ (an LLM), i.e., $a_t = \texttt{LLM}(p_{t}, s_{t-1})$.
The policy $\pi_\theta$ is updated through the modifications to $p$, which is the modification of subgoal instructions $g_{i,j}$ ($i$-th layer, $j$-th leaf node) that best suit the current situation.
Concretely, is featured with two key modules, Decomposition and Search which construct and utilize a subgoal tree $\mathbb{T}$ respectively, namely , to interact with the environment.
Setting the high-level goal of the task as the root node in ,
Search Module finds the nodes that are helpful for the status quo, and Decomposition Module decomposes the node into subgoals as leaf nodes if it is not clear enough.

§.§ Details in

Search: Identifying Useful Subgoals for Current Situation
In the Search Module of , we ask the backbone LLM of the agent to identify the most appropriate subgoal for the current situation, e.g., “Select $K$ most useful sub-goals that will help you reach your main goal in the current situation ...” (see Appendix <ref> for the complete prompt).
We represent the the current state $s_t$ for timestep $t$ as a description of the dialogue history of the interaction with the environment.
We also find the leaf nodes of each branch in as the candidate subgoal list for LLMs to decide which ones that are useful.
The LLM then selects $K$ most suitable subgoals, followed by decomposition and the update of the instruction prompt $p_{t}$ at this step.

Decompose: Refine to Adapt to the Environment
Based on the current action-state pair $\{a_t, s_t\}$, is updated through decomposition if it is not specific enough for useful guidance to the agent.
We use the backbone LLM to break down the selected subgoal $g_{i,j}$ in the Search Module (initially set to $g_0$).
We prompt the LLM with the instruction such as “What subgoals can you derive from $\{g_{i,j}\}$, based on $\{a_t, s_t\}$”, which generates a new set of subgoals $G$ (see also Appendix <ref>).
To control the granularity of these subgoals, we apply a filtering mechanism that if the cosine similarity [36] between a new subgoal and existing subgoals exceeds $\xi$, the current node will not be updated.
Otherwise, we add the new subgoals under the current node, thus expanding the .
Moreover, a stopping mechanism is designed that if no new nodes are added to the for $N$ consecutive rounds, the update is stopped.

Act: Utilizing Subgoals to Take Actions
After getting the subgoals from that are found by as useful, the agent updates the instruction prompt $p_t$ for the LLM and takes action $a_t$ to interact with the environment.
The prompt of this step can also be found in Appendix <ref>.

achived below:

§.§ Overview of

starts from a root node and continues to grow during interaction with the environment until convergence.
The guidance (, nodes) become finer gradually as the tree grows, corresponding to human decision process.
During the iterative growth, a Generator produces new detailed guidance as candidates, while an Evaluator assesses them to determine if they are suitable for tree expansion.
After iteration for one round, a Searcher selects part of the nodes to guide the action at next step.
In this section, we will demonstrate how does improve decision making during interacting with environment and illustrate the workflow of different modules of .

§.§ Guided Decision Making

In an environment focused on a main goal $g_0$, an Actor $M_a$ have to achieve the goal through interacting with it.
During the interaction, the policy that $M_a$ takes is defined as $\pi_\theta$, where $\theta=\{M_a, prompt\}$.
$\pi_\theta$ updates in a non-parametric manner along with the change of $prompt$.
At each round, policy $\pi_\theta$ generate an action and the environment returns a state $s_t$.
This action-state pair $\{a_{t}, s_{t}\}$ is then used to refine the instruction and modify the policy as needed. The LLM receives a reward that aligns with the main goal $g_0$ at the end of each iteration.

An Example Run

In an auction scenario where the goal is to maximize your own profit, it's vital for agents to budget wisely.
As the bidding war begins, the may choose the guidance Monitor other bidders from existing nodes, based on current state.
This selection is followed by an update to $\pi_\theta$.
The actor $M_a$ then takes action $a_t$ to interact with the environment.
After bids are placed, the state $s_{t}$ is updated.
The refines the existing nodes with the new action-state pair $\{a_{t}, s_{t}\}$ to provide more specific guidance.
If competition intensifies, the updated might suggest Identify aggressive bidders or Look for signals of withdrawal.
This strategic advice helps prevent agents from continuously bidding without considering their profit margins.
Without such guidance, bidders may risk obtaining items with minimal or even negative returns.

§.§ Details of

starts from a root node (, the high-level goal) and continues to grow iteratively during interaction (See details in Algorithm <ref>).

Initially, before the first trial begins, the Generator breaks down the main goal based on the current environment, obtaining the first layer of subgoals.
These subgoals, usually representing high-level goals, are added under the main goal node as the tree's first layer.
Next, the Searcher selects subgoals from the tree as guidance, based on the current state, and updates the policy.
The Actor $M_a$ then makes decisions according to the updated policy.
After the first trial, the Generator decomposes the subgoal nodes selected in this round.
The Evaluator then assesses the decomposition, and the subgoals that pass the evaluation are added under the selected nodes, forming a new layer of the tree.
The Actor, Generator, Searcher, and Evaluator models collaborate through trials in a loop until the process concludes.

Searcher $S$
At time step $t$, for the goal tree $T_t$ with multiple goals, it is necessary to determine the most appropriate subgoals for the current state.
To achieve this, we construct a Searcher based on the frozen LLM.
The instructions provided to the Searcher consist of a description of the current state $s_t$ at time step $t$ and a list of all subgoals, which are derived from the lowest node structure of each branch in the goal tree.
Subsequently, the Searcher is prompted to select the most beneficial subgoal within a window width of 5, resulting in an update to $p_{s_t}$. This update is then employed to modify the policy parameters $\theta$.

Generator $G$
The Generator plays a key role in the expansion of the goal tree.
Built upon an LLM, we have devised instructions that prompt the Generator to decompose a goal into finer subgoals.
These instructions encompass the current goal and a description of the current environmental state (refer to the complete prompts in Figure <ref>,  <ref>).
Specifically, assuming a collection of subgoals $G_{t-1}=\left\{g_{1, t}, \ldots, g_{i, t}\right\}$ is selected from the Goal Tree $T_{t-1}$ at time step $t-1$, the Generator traverses the entire set.
Each selected goal is divided into new subgoals based on the current context.
If these new subgoals pass the assessment of the Evaluator, they are added as children under the selected node, thus updating $T_{t-1}$ into $T_t$.

Evaluator $E$
The Evaluator plays a crucial role in controlling the granularity of the generated subgoals.
For subgoals produced by the Generator, we need to establish selection criteria. We use similarity as the criterion in this case.
The Evaluator calculates the cosine similarity [36] between each new subgoal and all the lowest-level child nodes in the current tree.
If the similarity is above 0.8, the expansion of the current child node is not pursued; otherwise, it is added under the current node.
The threshold of 0.8 indicates the level of granularity, with higher values corresponding to finer granularity.

§ EXPERIMENTAL SETUP

§.§ Tasks and Environments

The categorization of studied tasks.

Task     Rounds     Task Type

Public Goods Game     Single     Competitive

Guess 2/3 of the Average     Single     Cooperative

First-price Auction     Multiple     Competitive

Bargaining     Multiple     Cooperative

We evaluate across 4 dynamic tasks with high-level goals, including Public Goods Game, Guess 2/3 of the Average, First-price Auction, and Bargaining, which are implemented by existing works [37, 38, 39].
As seen in Table <ref>, they are either single-round or multi-round games, requiring the collaboration or competition of multiple agents.
Note that agents in multi-round games will only receive delayed rewards at the end of the game.
In our experiments, we repeat single-round games for $T=20$ times and multi-round games for $T=10$ times for stable results.

Public Goods Game: GAMA-Bench
We use GAMA-Bench [37] as the implemented environment for this game.
Specifically, each of $N=5$ players privately decides the number of tokens contributed to a public pot.
The tokens in the pot are multiplied by a factor $R$ $(1 \leq R \leq N)$, and the created “public good” is distributed evenly among all players.
Players keep any tokens they do not contribute.
A simple calculation reveals that for each token a player contributes, their net
gain is $\frac{R}{N}-1$ (i.e., income-contribution).
Since this value is negative, it suggests that the most rational strategy for each player is to contribute no tokens.
This strategy results in a Nash equilibrium [40] in the game.
$N$ agents using the same backbone model and equipped with the same method (, CLIN or ) play games with each other to observe group behavior.
Following [37], we set $R=2$.

Guess 2/3 of the Average: GAMA-Bench
Using the implementation of GAMA-Bench [37], $N$ players independently choose a number between 0 and 100 [41],
and whoever has the number closest to two-thirds of the group's average wins the game.
This setup effectively tests players' theory-of-mind (ToM) abilities [42, 43].
In behavioral economics, the Cognitive Hierarchy Model [44] categorizes players as follows: Level-0 players choose numbers randomly. Level-1 players assume others are Level-0 and pick two-thirds of an expected mean of 50.
Level-$k$ players believe that the participants include levels $0$ to $k-1$, and therefore choose $(2/3)^k \times 50$. The optimal outcome is to choose 0 for all players, achieving a Nash equilibrium.
In this game, $N=5$ agents using same backbone model with the same prompting method (, ) play games with each other to observe group behavior.

First-price Auction: AucArena
We use AucArena [38] as the implementation of first-price auctions.
An auctioneer collects and announces the bids of all participants, revealing the current highest bid.
Participants must publicly make their decisions after privately considering their bids.
The auction comprises if $K=15$ items with values ranging from $2,000 to $10,000, with an increment of $2,000 between each item.
These items are presented in a randomized sequence, making the auction last for $K=15$ rounds.
$N=4$ agents participate in the auction as bidders.
Each agent aims to secure the highest profit by the end of the auction and thereby outperform all competitors.
In our experiment, we set the budget for each bidder at $20,000. We have an agent, enhanced by various methods (, ), using different backbone models to compete against three identical opponents powered by the same model (GPT-3.5 [2]).

Bargaining: DealOrNotDeal
We use DealOrNotDeal [39] to implement the bargaining over multiple issues.
$N=2$ agents, namely Alice and Bob, are presented with sets of items (e.g., books, hats, balls) and must negotiate their distribution.
Each agent is randomly assigned an integer value between 0 and 10 for each item, ensuring that the total value of all items for any agent does not exceed 10.
The bargaining goes on for $K=10$ rounds, and if the agents fail to agree on the distribution of items within 10 rounds, neither party profits.
The goal is to minimize profit discrepancies between the two agents.
We randomly select $M=50$ items for Alice and Bob to negotiate over.
The final profits at the end of the negotiation for Alice and Bob are defined as $P_{Alice}$ and $P_{Bob}$, respectively.
Note that, we alter the prompting methods of the agent behind Alice, and keep Bob fixed (GPT-3.5).

§.§ Agent Framework Baselines and Backbone LLMs

We adopt two types of agent frameworks providing guidance for achieving high-level goals in the above tasks.[Implementation details are in Appendix <ref>.]
One is task decomposition framework, including ReAct [16] and ADAPT [6].
ReAct enables agents to reason before acting, while ADAPT recursively plans and decomposes complex sub-tasks when the LLM cannot execute them.
Another is experience summarization framework, including Reflexion [17] and CLIN [11].
Reflexion prompts agents to reflect on failed task attempts and retry.
CLIN creates a memory of causal abstractions to assist trials in future by reflecting on past experiences, expressed as A [may/should] be necessary for B..

To drive these language agent frameworks, we use the following LLMs: GPT-3.5-Turbo (gpt-3.5-turbo-1106) [3] and GPT-4-Turbo (gpt-4-1106-preview) [3];
Gemini 1.0 Pro [45]; Mistral-7B-Instruct-v0.2 [46] and a Mixture of Experts (MoE) model Mixtral-8x7B-Instruct-v0.1 [47];
Qwen 1.5 (7B and 72B variants) [48].
The temperature is set to 0 to minimize randomness.

§.§ Metrics for Tasks

In GAMA-Bench's Public Goods Game [37], where $N$ players participating in repeated $T$ times,
the score $S_1$ for this game is then given by:
$S_1=\frac{1}{NT} \sum_{i j} C_{i,j}$,
where $C_{i,j} \in [0, 1]$ is the proposed contribution of player $i$ in round $j$.

In GAMA-Bench's Guess 2/3 of the Average Game [37], the score $S_2$ is calculated by
$S_{2}=100-\frac{1}{NT} \sum_{ij} C_{i,j}$,
where $C_{i,j}$ is the number chosen by player $i$ in round $j$.

In AucArena's First-price Auction [38], we use the TrueSkill Score [49, 50] (Appendix <ref>) to rank the profits of agents.
TrueSkill Score estimates dynamic skill levels $(\mu)$ through Bayesian statistics while considering the uncertainty $(\sigma)$ in their true skills.
Thus the performance score of an agent is defined as $S_{3} = \text{TrueSkill Score}$.
This method is commonly used in competitions such as online games or tournaments.

In DealOrNotDeal's Bargaining Game [39],
we calculate the absolute difference in their profits:
$S_{4}=\frac{|P_{Alice} - P_{Bob}|}{M}$,
where $P_{Alice}, P_{Bob}$ represents the profits at the end of the negotiation, and $M$ is the number of items to negotiate on. ($S_{4}$ can also be represented by TrueSkill Score for convenience.)

§ RESULTS AND ANALYSIS

§.§ Main Results

The main results across 4 scenarios are presented in Table <ref>.
Overall, our significantly outperforms all baseline frameworks in various environments containing high-level goals, where larger LLMs produce higher gains.
When diving into the generated guidelines and corresponding agents' behaviors, we find that some of those subgoals given by task decomposition methods like ReAct and ADAPT are no longer suited for the current situation.
For example, bid on the most expensive item is not useful when the budget is tight.
Moreover, task decomposition before interacting with the environment does not consider the practical experience, leading to broad and meaningless guidance.
For example, in Public Goods Game, ADAPT provides broad subgoals like It's important to strike a balance between contributing enough tokens to the public pot to earn a significant payoff while retaining enough tokens in my private collection for future rounds.
In contrast, post-hoc experience summarization methods, i.e., Reflexion and CLIN, tend to induce too detailed guidelines, lacking a correlation with the main goal and might deviating agents from their paths.
For example, CLIN produces subgoals focusing on minutiae, such as Considering the distribution of numbers chosen by opponents may be necessary to make an informed decision on your own selection.

In comparison, overcomes both of the shortcomings.
At each round, decomposes new nodes referring to existing guidance, aligning with the main goal as the game progresses.
For example, in Public Good Game, the initial subgoal is The player aims to contribute strategically based on their assessment of other players' behaviors and the overall distribution of tokens in the public pot.
If all players contribute less to the public pot during the game, absorbs the observation and refines existing nodes to If the player notices that the average contribution of the group has been increasing in recent rounds, they might choose to contribute fewer tokens in the current round to avoid over-contributing and potentially losing out on their own gain.
According to the new subgoal as a practical guideline, agents can dynamically adjust their contributions.[More details of are in Appendix <ref>.]

Interestingly, shows superior performance in smaller LLMs as well, while others can not due to the deficiency of induction and summarization capability of these models.
For example, CLIN is 0.7 inferior to Reflexion for Mistral-7B and 5.77 for Qwen-7B in Guess 2/3 of the Average, but brings improvements consistently.
This can be attributed to the logical, structural architecture of in .
At each time for decomposition, the model receives existing subgoals on the last layer of as clear references, making it easy for decomposition.

Competition between Different Agent Framework
Previous results are mostly evaluated against a fixed baseline (GPT-3.5).
To understand how these agent frameworks behave when competing with each other, we set an AucArena for a multi-agent comparison.
As shown in Table <ref>, has a clear advantage over baselines.
When looking closer at the bidding behaviors, we find that other methods tend to be overly cautious.
They often stop bidding or avoid participating once bidding starts, resulting in zero profits.
However, takes a different approach by bidding frequently in the early stages of the bidding war when competition is less intense.
This allows for purchasing high-priority items early on, avoiding fierce competition in the later stages.

§.§ Analysis of

Granularity control of the threshold $\xi$ in 's stopping mechanism.

How does the granularity of guidelines in affect task solving?

As discussed in $\mathsection$<ref>, adjusts to the dynamic environment by setting different depths, where subgoal nodes of deeper layers provide more detailed instructions.
Here, we explore how such granularity affects the performance of .
We use Auction and Bargaining environments as testbeds, and modify the level of subgoals by setting the threshold $\xi$ in the stopping mechanism as 0.6, 0.7, 0.8, and 0.9.
According to Figure <ref>, the agent's performance initially improves with increasing depth but eventually diminishes.
A shallow tree ($\xi=0.6$) lacks guidance details, thus leading to the poorest performance.
Yet, the deepest tree ($\xi=0.9$) does not show superior performance, probably because repetitive guidance interferes with model selection of useful guidance.
Redundant nodes increase the candidate set, making it difficult for the search module to select all the valuable nodes.
In fact, the search module always focuses on multiple nodes representing the same meaning, resulting in the loss of other helpful nodes.
This experiment confirms that more detailed instructions help language agents achieve high-level goals, but only with a balanced, adaptive depth of the guidance tree to mitigate the drawbacks of overly detailed guidance.

Ablation study of different search modules.

Ablation Study of Search Module

Can the Search Module in succeed in finding useful subgoal nodes?
We employ two methods as baselines to replace the original LLM-based search module, which is instantiated with GPT-3.5.
One baseline is random selection, where we randomly choose an node from the set of subgoal nodes.
The other is the selection based on embedding similarity, which selects the subgoals most similar to the current situation based on cosine similarity.
On multi-round games as Auction and Bargaining, we keep the Trueskill Score for evaluating the rankings of these methods.
As shown in Figure <ref>, the LLM search module gains a better score in both games.
Besides, similarity-based method performs worse than random selection in Bargaining, which could be the reason that the guidance is usually short, making it hard to capture semantic embeddings between subgoals and situations.
This experiment demonstrates the rationality of the LLM-based search module in 's design.

Ablation study of the model that generates , either by a stronger (GPT-4) or weaker (GPT-3.5) model. The rest of the agent framework is driven by GPT-3.5.

How does the quality of affect goal achievement?
To explore the influence of on , we conduct an experiment in Auction and Bargaining Games by replacing the model that constructs with GPT-4 or GPT-3.5 for comparison, while keeping the model that utilizes the tree fixed as GPT-3.5.
Results in Figure <ref> illustrate that higher-quality (from GPT-4) significantly boosts the performance of , with gains of +2.87 in Auction and +3.10 in Bargaining compared to one using GPT-3.5.
This improvement comes from more abundant and higher-quality guidance, generated by a strong model equipped with better understanding and summarizing capabilities. (We also conduct an ablation study about the influence of pruning on in Appendix <ref>.)

Patterns of model behavior in repeated games. (a): Fluctuations in contributions within the Public Goods game. The agent equipped with displays more rational behavior (, achieving a Nash equilibrium) by consistently contributing fewer tokens than other methods.
(b): Adjustments in number predictions within the Guessing Game. Our shows enhanced ToM abilities by converging to a guess of zero more quickly in each round.

Can improve the rationality in agents' behaviors?

Aside from the final performance gain, we are also interested in whether each agent behavior at every turn benefits from .
Therefore, we use two games from GAMA-Bench to examine the impact of on model behavior, where behavioral changes are easier to evaluate.
Here, we use LLMs with great improvement from , i.e., Mistral-7B for Public Goods Game and Qwen-72B for Guessing 2/3 Average Number Game.
We record patterns in the model's number predictions and token contributions by visualizing data from 20 repeated experiments.
Note that is updated across these 20 rounds of games.
With , agents in the Public Goods scenario consistently act more rationally compared to those using alternative methods, as illustrated in Figure <ref>(a).
For the Guessing Game, enhanced models showed smoother, more steadily declining curves, indicating quicker convergence to the Nash equilibrium, as depicted in Figure <ref>(b).

In the Bargaining task, Mistral-7B with CLIN or ADAPT gives guidance that is either too broad or too detailed resulting in large profit discrepency, whereas is successful.

§.§ Case Study

To illustrate how agents from different frameworks reason and plan in a dynamic environment, we conduct a case study using Mistral-7B, a small LLM, as the backbone in a bargaining game (Figure <ref>).
We find that 's emphasis on granularity control offers clear advantages.
provides agents with actionable guidance such as ask clarifying questions, prompting agents to pay early attention to their opponent's psychological assessment and different valuations of items.
After acquiring a partner's valuation, then gives guidance such as make concessions, leading the agent to propose a plan that gives up a particular item in exchange for minimizing the profit difference.

In contrast, CLIN advises agents to consider the preference of the partner, which leads agents to focus on the opponent's preferences but may result in plans that sacrifice their own interests to improve the other party's income.
ADAPT, which decomposes tasks beforehand, provides very broad advice such as equal allocation.
This generic advice aims to minimize the profit gap but may not be suitable for scenarios lacking knowledge of the partner's valuation.
Consequently, the model proposes allocation plans without first clarifying the partner's valuations, assuming that all participants have the same valuation for each item.

§ CONCLUSION

In this paper, we introduce , an agent framework that enhances the capabilities of LLMs for achieving high-level goals across various dynamic tasks and environments.
We demonstrate that significantly improves agent performance by dynamically generating and refining a hierarchical of contextual subgoals based on interactions with the environments.
Experiments show that this method is effective in both competitive and cooperative scenarios, outperforming baseline approaches.
Moreover, can be continually updated as agents with further engage with the environments, enabling them to navigate complex environments with greater precision and adaptability.
However, we also notice that although is effective for small models, there is still a demand for the understanding and summarizing capability of models, which might prevent from achieving its full effectiveness.

[1]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.

[2]
Chatgpt, 2022.

[3]
Gpt-4 technical report, 2024.

[4]
Yi Zheng, Chongyang Ma, Kanle Shi, and Haibin Huang.
Agents meet okr: An object and key results driven agent system with hierarchical self-collaboration and self-evaluation, 2023.

[5]
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal.
Decomposed prompting: A modular approach for solving complex tasks, 2023.

[6]
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot.
Adapt: As-needed decomposition and planning with language models, 2024.

[7]
Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Jankowski, Yanghua Xiao, and Deqing Yang.
Distilling script knowledge from large language models for constrained language planning.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4303–4325, Toronto, Canada, 2023. Association for Computational Linguistics.

[8]
Ishika Singh, David Traum, and Jesse Thomason.
Twostep: Multi-agent task planning using classical planners and large language models.
ArXiv preprint, abs/2403.17246, 2024.

[9]
Yuchen Liu, Luigi Palmieri, Sebastian Koch, Ilche Georgievski, and Marco Aiello.
Delta: Decomposed efficient long-term robot task planning using large language models, 2024.

[10]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
Self-refine: Iterative refinement with self-feedback, 2023.

[11]
Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark.
Clin: A continually learning language agent for rapid task adaptation and generalization, 2023.

[12]
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang.
Expel: Llm agents are experiential learners.
In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2024, February 20-27, 2024, Vancouver, Canada, pages 19632–19642. AAAI Press, 2024.

[13]
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings.
Refiner: Reasoning feedback on intermediate representations, 2024.

[14]
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.
Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.
In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 9118–9147. PMLR, 2022.

[15]
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren.
Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks, 2023.

[16]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023.

[17]
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning, 2023.

[18]
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar.
Voyager: An open-ended embodied agent with large language models.
ArXiv preprint, abs/2305.16291, 2023.

[19]
Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, and Roy Fox.
Skill set optimization: Reinforcing language model behavior via transferable skills, 2024.

[20]
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.
Do as i can, not as i say: Grounding language in robotic affordances, 2022.

[21]
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter.
Inner monologue: Embodied reasoning through planning with language models, 2022.

[22]
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone.
Llm+p: Empowering large language models with optimal planning proficiency, 2023.

[23]
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim.
Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2609–2634, Toronto, Canada, 2023. Association for Computational Linguistics.

[24]
Yikai Zhang, Siyu Yuan, Caiyu Hu, Kyle Richardson, Yanghua Xiao, and Jiangjie Chen.
Timearena: Shaping efficient multitasking language agents in a time-aware simulation.
ArXiv preprint, abs/2402.05733, 2024.

[25]
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su.
Travelplanner: A benchmark for real-world planning with language agents.
ArXiv preprint, abs/2402.01622, 2024.

[26]
Chengdong Ma, Ziran Yang, Minquan Gao, Hai Ci, Jun Gao, Xuehai Pan, and Yaodong Yang.
Red teaming game: A game-theoretic framework for red teaming language models, 2024.

[27]
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap.
Sotopia: Interactive evaluation for social intelligence in language agents, 2024.

[28]
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu.
ScienceWorld: Is your agent smarter than a 5th grader?
In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.

[29]
Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lee.
Successor feature landmarks for long-horizon goal-conditioned reinforcement learning.
In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 26963–26975, 2021.

[30]
Zhiao Huang, Fangchen Liu, and Hao Su.
Mapping state space using landmarks for universal goal reaching.
In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1940–1950, 2019.

[31]
Andrew G Barto and Sridhar Mahadevan.
Recent advances in hierarchical reinforcement learning.
Discrete event dynamic systems, 13:341–379, 2003.

[32]
Damien Pellier, Alexandre Albore, Humbert Fiorino, and Rafael Bailon-Ruiz.
Hddl 2.1: Towards defining a formalism and a semantics for temporal htn planning, 2023.

[33]
Kutluhan Erol, James Hendler, and Dana S Nau.
Htn planning: Complexity and expressivity.
In AAAI, volume 94, pages 1123–1128, 1994.

[34]
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal.
Decomposed prompting: A modular approach for solving complex tasks.
ArXiv preprint, abs/2210.02406, 2022.

[35]
Valerie Goffaux, Judith Peters, Julie Haubrechts, Christine Schiltz, Bernadette Jansma, and Rainer Goebel.
From coarse to fine? spatial and temporal dynamics of cortical face processing.
Cerebral Cortex, page 467–476, 2011.

[36]
Faisal Rahutomo, Teruaki Kitasuka, Masayoshi Aritsugi, et al.
Semantic cosine similarity.
In The 7th international student conference on advanced science and technology ICAST, volume 4, page 1. University of Seoul South Korea, 2012.

[37]
Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu.
How far are we on the decision-making of llms? evaluating llms' gaming ability in multi-agent environments.
ArXiv preprint, abs/2403.11807, 2024.

[38]
Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson.
Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena, 2023.

[39]
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra.
Deal or no deal? end-to-end learning of negotiation dialogues.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453, Copenhagen, Denmark, 2017. Association for Computational Linguistics.

[40]
Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou.
The complexity of computing a nash equilibrium.
Communications of the ACM, 52(2):89–97, 2009.

[41]
Alain Ledoux.
Concours résultats complets: Les victimes se sont plu à jouer le 14 d'atout.
Jeux & Stratégie, 2(10):10–11, 1981.

[42]
Michal Kosinski.
Theory of mind might have spontaneously emerged in large language models.
ArXiv preprint, abs/2302.02083, 2023.

[43]
Yuanyuan Mao, Shuang Liu, Pengshuai Zhao, Qin Ni, Xin Lin, and Liang He.
A review on machine theory of mind, 2023.

[44]
Colin F. Camerer, Ho Teck-Hua, and Chong Juin-Kuan.
A cognitive hierarchy model of games.
The Quarterly Journal of Economics, 119, 2004.

[45]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.
Gemini: a family of highly capable multimodal models.
ArXiv preprint, abs/2312.11805, 2023.

[46]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mistral 7b, 2023.

[47]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
ArXiv preprint, abs/2401.04088, 2024.

[48]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.
Qwen technical report, 2023.

[49]
Ralf Herbrich, Tom Minka, and Thore Graepel.
Trueskill™: a bayesian skill rating system.
Advances in neural information processing systems, 19, 2006.

[50]
Tom Minka, Ryan Cleven, and Yordan Zaykov.
Trueskill 2: An improved bayesian skill rating system.
Technical Report, 2018.

§ DETAILS

§.§ Does pruning the affect search quality?

We investigate whether pruning nodes not selected for a long time from the target tree affects the Search Module's decisions.
Pruning begins after the Decompose Module completes building the tree, and nodes unselected for more than five consecutive rounds will be deleted.
We assess the impact of pruning on GPT-3.5's performance in Auction and Bargaining.
As shown in Table <ref>, the TrueSkill Score with and without pruning are similar.
This suggests that nodes not chosen for extended periods do not compromise the Search Module's decision-making effectiveness.
This efficiency likely results from our Search Module using prior knowledge from LLM to identify and avoid selecting unnecessary nodes, akin to lazy deletion.
For efficiency, these redundant nodes are also removed every five rounds.

§.§ Implementation Details

We compare our with the following methods: ReAct [16], which induces an LLM actor to engage in preliminary reasoning about the task before initiating action, Reflexion [17], which encourages an LLM actor to re-assess unsuccessful task attempts before attempting the task again, CLIN [11], which leverages historical insights to deduce transition strategies, articulated as “A [may/should] be necessary for A”.
To adapt these methods to our experimental environment, we update the memory of the CLIN/Reflexion approach at each timestep within a single trial, whether it is a bid in the Auction environment, a dialogue round in the Negotiation environment, or a game round in GAMA-Bench.
Specifically, for Reflexion, the model uses historical steps from the current trial to generate verbal self-reflections.
These self-reflections are then added to long-term memory, providing valuable feedback for future trials. In the case of CLIN, we use the BASE method due to the absence of a training set in our environment.
The memory is updated at each step by prompting the model with historical steps from the current trial and all previous memories to generate an updated memory, which includes a new list of semi-structured causal abstractions.
This updated memory is then incorporated into the historical memories.

§.§ Details of TrueSkill Score

In a game with a population of $n$ players $\{1, \ldots, n\}$, consider a match where $k$ teams compete. The team assignments are specified by $k$ non-overlapping subsets $A_j \subset\{1, \ldots, n\}$ of the player population, with $A_i \cap A_j=\emptyset$ for $i \neq j$. The outcome $\mathbf{r}:=\left(r_1, \ldots, r_k\right) \in\{1, \ldots, k\}$ is defined by a rank $r_j$ for each team $j$, with $r=1$ indicating the winner and draws possible when $r_i=r_j$. Ranks are based on the game's scoring rules.

The probability $P(\mathbf{r} \mid \mathbf{s}, A)$ of the game outcome $\mathbf{r}$ is modeled given the skills $\mathbf{s}$ of the participating players and the team assignments $A:=\left\{A_1, \ldots, A_k\right\}$. From Bayes' rule, we get the posterior distribution

p(\mathbf{s} \mid \mathbf{r}, A)=\frac{P(\mathbf{r} \mid \mathbf{s}, A) p(\mathbf{s})}{P(\mathbf{r} \mid A)} .

We assume a factorizing Gaussian prior distribution, $p(\mathbf{s}):=\prod_{i=1}^n {N}\left(s_i ; \mu_i, \sigma_i^2\right)$. Each player $i$ is assumed to exhibit a performance $p_i \sim {N}\left(p_i ; s_i, \beta^2\right)$ in the game, centered around their skill $s_i$ with fixed variance $\beta^2$.

The performance $t_j$ of team $j$ is modeled as the sum of the performances of its members, $t_j:=\sum_{i \in A_j} p_i$. Teams are reordered in ascending order of rank, $r_{(1)} \leq r_{(2)} \leq \cdots \leq r_{(k)}$. Disregarding draws, the probability of a game outcome $\mathbf{r}$ is modeled as

P\left(\mathbf{r} \mid\left\{t_1, \ldots, t_k\right\}\right)=P\left(t_{r_{(1)}}>t_{r_{(2)}}>\cdots>t_{r_{(k)}}\right)

In other words, the order of performances determines the game outcome. If draws are allowed, the winning outcome $r_{(j)}<r_{(j+1)}$ requires $t_{r_{(j)}}>t_{r_{(j+1)}}+\varepsilon$ and the draw outcome $r_{(j)}=r_{(j+1)}$ requires $\left|t_{r_{(j)}}-t_{r_{(j+1)}}\right| \leq \varepsilon$, where $\varepsilon>0$ is a draw margin calculated from the assumed probability of a draw. ${ }^1$

To report skill estimates after each game, we use an online learning scheme called Gaussian density filtering. The posterior distribution is approximated to be Gaussian and is used as the prior distribution for the next game. If skills are expected to change over time, a Gaussian dynamics factor ${N}\left(s_{i, t+1} ; s_{i, t}, \gamma^2\right)$ can be introduced, leading to an additive variance component of $\gamma^2$ in the subsequent prior.

Consider a game with $k=3$ teams with team assignments $A_1=\{1\}, A_2=\{2,3\}$ and $A_3=\{4\}$. Assume that team 1 wins and teams 2 and 3 draw, i.e., $\mathbf{r}:=(1,2,2)$.
The function represented by a factor graph in our case, the joint distribution $p(\mathbf{s}, \mathbf{p}, \mathbf{t} \mid \mathbf{r}, A)$, is given by the product of all the potential functions associated with each factor. The structure of the factor graph provides information about the dependencies of the factors involved and serves as the foundation for efficient inference algorithms.
Referring back to Bayes' rule, the quantities of interest are the posterior distribution $p\left(s_i \mid \mathbf{r}, A\right)$ over skills given game outcome $\mathbf{r}$ and team assignments $A$. The $p\left(s_i \mid \mathbf{r}, A\right)$ are calculated from the joint distribution by integrating out the individual performances $\left\{p_i\right\}$ and the team performances $\left\{t_i\right\}$:

p(\mathbf{s} \mid \mathbf{r}, A)=\int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} p(\mathbf{s}, \mathbf{p}, \mathbf{t} \mid \mathbf{r}, A) d \mathbf{p} d \mathbf{t} .

§.§ Instruction Prompt Examples

The instruction prompts of three modules in are presented in Listing <ref>.

We implemented CLIN and Reflexion methods in our environments as presented in Listing <ref>.

§.§ Examples of GoalTree

Here, we provide examples of from four environments in Listing <ref>, with their main goals as follows:

* Public Goods: maximize your total token count by the end of the game;

* Guess 2/3 of the Average: choose a number that you believe will be closest to 2/3 of the average of all numbers chosen by players, including your selection;

* First-price Auction: secure the highest profit at the end of this auction, compared to all other bidders;

* Bargaining: minimize the profit gap between yourself and your partner in this negotiation, regardless of your own profit.


# Towards computer-assisted design of hole spin qubits in quantum dot devices

Ana Ciocoiu  Department of Electrical and Computer Engineering. University of
British Columbia, Vancouver, B.C., Canada.  Blusson Quantum Matter Institute,
University of British Columbia, Vancouver, B.C., Canada.  Mohammad Khalifa
Department of Electrical and Computer Engineering. University of British
Columbia, Vancouver, B.C., Canada.  Blusson Quantum Matter Institute,
University of British Columbia, Vancouver, B.C., Canada.  Joe Salfi
<EMAIL_ADDRESS>Department of Electrical and Computer Engineering.
University of British Columbia, Vancouver, B.C., Canada.  Blusson Quantum
Matter Institute, University of British Columbia, Vancouver, B.C., Canada.

###### Abstract

The design of scalable quantum computers will benefit from predictive models
for qubit performance that consider the design and layout of the qubit
devices. This approach, has recently been adopted for superconducting qubits,
but has received little attention for spin qubits in semiconductors. Here, we
employ models for both the device and the quantum mechanical states in the
valence band to theoretically investigate the properties of hole spin qubits
in laterally gated quantum dots in group-IV materials, which have received
significant recent attention. We find that device design impacts qubit
properties in unexpected ways. First, the presence of optimal operation points
where coherence times are long and qubits can be rapidly manipulated results
not only from gate-voltage-induced changes to the Rashba coefficient, but also
from gate-voltage-induced changes to quantum dot radius in real devices, which
impacts g-factor even when inversion symmetry is preserved (Rashba coupling is
zero). Second, the qubit electric manipulation is substantially higher in the
realistic anharmonic potential, by an order of magnitude, in the device design
we consider, compared with a harmonic potential assumption, because the
transverse electric dipole of the qubit depends on electrically mediated
couplings to many excited states. Finally, we show that the rapid electric
drive, compatible with long coherence times, can be achieved in the single-
hole regime, with and without strain. These results establish the need for
realistic description of both the device and the quantum mechanical states to
support the design of spin qubit devices, identify new ways to control hole
qubit properties.

Classical electronic circuit design employs simplified models extracted from
device-level simulations. Similarly, the design of large-scale quantum
circuits will benefit from a thorough understanding of how device design
impacts quantum circuit performance. This approach is beginning to be adopted
within the superconducting qubit community, including electromagnetic based
extraction of parameters in quantized circuit models in superconducting
devices, and development of design frameworks[1, 2, 3, 4]. The understanding
of spin qubits in semiconductors[5], which is at an earlier stage of
development[6, 7, 8], could benefit from a similar approach, given recent
rapid experimental progress [9, 10].

Spin qubits, implemented as quantum dots (QDs)[11, 12, 13, 14, 9, 10, 15, 16,
17, 10, 18] or dopants[19, 20, 21] in group-IV materials, are of interest
because they utilize the most common materials in the semiconductor industry,
and because isotope purification in group-IV materials permits the elimination
of the nuclear spin bath, enabling long spin coherence times[22, 19, 11]. Spin
qubits based on valence band holes have recently attracted significant
attention in Si[13, 12, 20, 21] and Ge[15, 16, 17, 10, 18]. This is due to the
strong spin-orbit interaction (SOI) experienced by spins in the valence band,
which enables electric qubit manipulation through, e.g., electric dipole spin
resonance (EDSR)[23], as well as long-ranged two-qubit operations mediated by
superconducting resonators[24] or mutual capacitances[25]. Holes also lack the
valley-orbit degree of freedom that complicates operation of electron spin
qubits in group-IV materials[26, 27]. Moreover, the $p$-type character of the
orbitals of valence band holes suppresses nuclear-spin-induced decoherence
compared to conduction band electrons[28].

For hole spin qubits, the presence of Rashba SOI that enables the desirable
electric control generally ties their properties to electric potentials, and
makes them more susceptible than electron spin qubits to decoherence and
relaxation from electric fluctuations and phonons. Indeed, reported
$T_{2}^{*}$ coherence times for hole spin QDs lies within 0.13-0.8 $\mu$s[15,
16, 18], compared to $\approx$ 120 $\mu$s [11] for electrons, while reported
Hahn-echo coherence times $T_{2}$ range from 0.2-1.9 $\mu$s [18, 13] for
holes, compared to $\approx$ 30 ms for electrons[11]. Theoretical studies have
identified sweet spots combining fast electric operation and long coherence
and relaxation times for hole spins bound to Si:B dopants and QDs in group-IV
materials[25, 27, 29, 30], and hole spin coherence times rivaling electron
spin in Si has been observed experimentally in Si:B dopants[20]. Understanding
achievable coherence times both experimentally and theoretically for hole QDs
is immensely important because coherence times set upper bounds on quantum
operation fidelities (e.g. for two-qubit gates), and increase the time between
costly error correction cycles anticipated in future fault tolerant quantum
computers. To date, models that account for realistic properties of holes in
the valence band do not take into account how devices are built[23, 25, 27,
29, 31, 30]. Qubit models that incorporate a realistic description of the
device and the valence band have yet to be reported.

In this paper, we theoretically investigate the properties of QD based hole
spin qubits in realistic gate-based devices, under different conditions for
strain and applied magnetic field direction. We compare the results obtained
for potentials in realistic devices with potentials used in previous work. We
find that dephasing from electric fields and electric qubit drive in real
devices is governed by two additional aspects of spin-orbit coupling relevant
to electronic devices that have yet to be identified in the literature. First,
the control of the Larmor frequency by varying a gate voltage occurs not only
because of changes in the Rashba spin-orbit coefficient induced by the
gate[32, 27], but also because of changes to the QD radius induced by the
gate, which modifies the g-factor even in zero electric field, when there is
inversion symmetry and no Rashba spin orbit interaction. We nevertheless find
gate voltages where the Larmor frequency is stable to electric fluctuations,
and where the qubit can be driven electrically due to a large transverse
electric dipole[25, 27, 29, 30]. These optimal operation points are found for
both out-of-plane and in-plane applied magnetic fields, and between unstrained
and highly strained (0.6% compressive strain) substrates. Second, the rate of
qubit manipulation for EDSR is very sensitive to the anharmonicity of the
electric potential. This arises from changes in the excited state spectrum,
the states responsible for mediating the electric drive. We find that taking
the anharmonicity into account enhances the EDSR rate by 40x compared to a
harmonic potential with the same curvature for the device geometries we
consider. We also find the qubit manipulation rate to be higher for the in-
plane magnetic field than the out-of-plane magnetic field, for fixed Larmor
frequency. Finally, we consider the charging of the quantum dot, showing that
that optimal operation point can be realized in the single hole regime by an
appropriate choice of gate voltages. Our study reveals that the out-of-plane
electric field and the occupation of holes are nearly independently controlled
by voltage differences and voltage averages, respectively, of plunger and
barrier gates in the gated QD.

Figure 1: Isometric (left) and front view (right) of the single QD platform
build using a Ge substrate. Gates are contained within a 30 nm thick Al2O3
dielectric, which has been partially made transparent for clarity. Spacing is
2.5 nm between Al2O3 interface and centre barrier, 2.5 nm between centre
barrier and plunger, and 5 nm between plunger and top barrier.

Although our model works for Si and Ge, we consider for specificity Ge QDs
defined by the gate pattern shown in Fig. 1. The structure consists of three
gate layers separated by Al2O3 dielectric. The plunger gate diameter is 90 nm.
The Al2O3 is 30 nm thick, with 15 nm between the gate and the active region of
the device, 2.5 nm of the dielectric spacing between the plunger and the
centre barrier. A hole QD forms beneath the plunger gate within the Ge when a
negative voltage is applied to the plunger. Positive voltages applied to the
barrier gates create a barrier to a reservoir (not shown). We assume the
substrate growth direction is along $\hat{z}\parallel(001)$.

Figure 2:  Comparison of extracted g-factor for a QD system with toy potential
and fixed radius of 16 nm (a) against realistic gate potential with barrier
voltage of 1.06V (b). QD radius variation as a function of applied voltage
(same barrier voltage as (b) ) and average z-directed field, $Ez$ is shown in
(c). All simulations use a zero-strain Ge substrate 15.4 nm thick and out-of-
plane magnetic field of 22 mT. We can notice that the ’sweet spot’ lies in
approximately the same range of electric field values in both (a) and (b),
despite significant radius variation across that range for the realistic
potential case.

We obtain the eigenstates of the QD using the three-dimensional Kohn-Luttinger
Hamiltonian for holes in the valence band. The valence band has $p$-like
symmetry ($L=1$). Heavy hole (HH) states are the states with $J=3/2$ and
projected angular momentum $m_{J}=\pm 3/2$ and light hole (LH) states are the
states with $J=3/2$ and projected angular momentum $m_{J}=\pm 1/2$. Split-off
hole states ($J=1/2$) are not considered in this work, which can be justified
in Ge because the split-off valence band is 300 meV away. The Hamiltonian
describing a single hole QD is:

$H=H_{LK}+H_{\epsilon}+H_{Z}+U(\mathbf{r})$ (1)

Here, $H_{LK}$ is the Luttinger-Kohn Hamiltonian for the valence band[33],
which accurately decribes both QDs[23] and acceptor dopants[34, 35],

$H_{LK}=\begin{bmatrix}P+Q&0&L&M\\\ 0&P+Q&M^{*}&-L^{*}\\\ L^{*}&M&P-Q&0\\\
M^{*}&-L&0&P-Q\end{bmatrix},$ (2)

where $P=\frac{\hbar^{2}\gamma_{1}}{2m_{0}}(k^{2}+k_{z}^{2})$,
$Q=\frac{-\hbar^{2}\gamma_{2}}{2m_{0}}(2k_{z}^{2}-k^{2})$,
$L=\frac{-\sqrt{3}\hbar^{2}\gamma_{3}}{2m_{0}}k_{-}k_{z}$,
$M=\frac{-\sqrt{3}\hbar^{2}}{2m_{0}}((\frac{\gamma_{2}+\gamma_{3}}{2})k_{-}^{2}+(\frac{\gamma_{3}-\gamma_{2}}{2}))k_{+}^{2}$,
and $m_{0}$ is the free electron mass, $\gamma_{1}$, $\gamma_{2}$,
$\gamma_{3}$ are Luttinger band-structure parameters with values of 13.15,
4.4, and 5.69 respectively[36]. The in-plane wave vector $k$ is given by
$k^{2}=k_{x}^{2}+k_{y}^{2}$ and $k_{\pm}=k_{x}\pm ik_{y}$. $H_{LK}$ is written
in the basis of HH and LH states:

$\left\\{\ket{\tfrac{3}{2},\tfrac{3}{2}}\ket{\tfrac{3}{2},-\tfrac{3}{2}}\ket{\tfrac{3}{2},\tfrac{1}{2}}\ket{\tfrac{3}{2},-\tfrac{1}{2}}\right\\}$
(3)

$H_{\epsilon}$ is a Bir-Pikus Hamiltonian for strain[34, 35], $U(\mathbf{r})$
is the electric potential energy, and $H_{Z}$ is the Zeeman coupling between
the hole and external magnetic field $\mathbf{B}$, given by
$H_{Z}=\mu_{B}\left(g_{1}(J_{x}B_{x}+J_{y}B_{y}+J_{z}B_{z})+g_{3}(J_{x}^{3}B_{x}+J_{y}^{3}B_{y}+J_{z}^{3}B_{z})\right)$,
$g_{1},g_{3}$ are the linear and cubic Lande g-factors, $\mu_{B}$ is the Bohr
magneton.

The voltages chosen for the gates define the potential energy $U(\mathbf{r})$
for the QD, which is obtained from a finite element electrostatic calculation
employing the gate pattern (see Fig. 1). The relative dielectric permittivity
of different materials is taken into account. We solve the full Hamiltonian
numerically in the finite difference approximation, using the external
potential $U(\mathbf{r})$, to find the eigenstates $\psi_{i}$, which are
spin-3/2 spinors, and their energies. The numerical solver uses infinite well
boundary conditions. Calculations are presented for unstrained and strained Ge
layers of 15.4 nm depth. The two lowest-energy eigenstates are, as expected,
mainly $+3/2$ and $-3/2$ heavy-hole states.

The coupling of spin qubits to electric fields is described by the
longitudinal and transverse spin-electric dipoles, $v$ and $\chi$,
respectively. These electric dipoles are constructed from integrals $p_{ijk}$,

$p_{ijk}=\int\psi_{i}(r)^{*}qx_{k}\psi_{j}(r)d^{3}r.$ (4)

that express couplings between states $i$ and $j$ due to an electric field
along the $k$ axis. The transverse electric dipole $v=p_{12k}$ determines the
Rabi frequency $\tau_{\rm Rabi}^{-1}=vE_{ac}/h$ of the oscillation between
qubit states $i=1$ and $j=2$ due to an applied electric field of magnitude
$E_{ac}$, oscillating at the qubit frequency, along the $k$ axis in space. The
electric dipole matrix elements with $i=j$ describe the shift in energy of
state $i$ due to an electric field $E$, and the longitudinal electric dipole
$\chi=p_{22k}-p_{11k}$ determines the change in qubit splitting
$\Delta\varepsilon=\chi E$ due to electric noise, which causes pure dephasing
in the presence of charge noise (fluctuating electric fields).

The eigenstates $\psi_{i}$ are computed numerically under two different
assumptions: first, using harmonic potentials that ignore gate voltage induced
changes in QD radius, and second, using potentials predicted from voltages
applied to the gates in Fig. 1. For the first, a 16 nm QD is modeled with a
laterally harmonic potential and independently tunable vertical electric field
$E_{z}$, $U(\mathbf{r})=\frac{1}{2}m_{p}{\omega_{0}}^{2}(x^{2}+y^{2})+qE_{z}z$
, where $m_{p}=m_{0}/(\gamma_{1}+\gamma_{2})$ is the in-plane effective mass
of the heavy holes and $\omega_{0}=\hbar/(m_{p}a_{0}^{2})$ is the harmonic
oscillator frequency, and $a_{0}$ is the quantum dot radius. The Hamiltonian
is solved numerically and the resulting g-factor, defined by the energy
difference between the two qubit states $g\mu_{B}B$, is plotted as a function
of vertical field $E_{z}$ in Fig. 2(a). The result of our finite numerical
solution to the Kohn-Luttinger equations is in qualitative agreement with the
variational calculation in ref. [27]. That is, we observe a local minimum in
the g-factor, occurring here at $E_{z}$=8 MV/m associated with the vanishing
of the longitudinal electric dipole that causes pure dephasing from electric
noise. This non-monotonic behavior of $g$ as a function of the gate field was
previously identified to arise from the non-monotonic behaviour of the Rashba
SOI coefficient[32, 27].

The effective qubit g-factor and QD radius for the real device potential are
plotted as a function of plunger gate voltage and average electric field in
Fig. 2 (b),(c). We define the average electric field as $\langle
E_{z}\rangle=\langle\Psi|-(d/dz)U(\mathbf{r})|\Psi\rangle$ since the hole
state will not experience a perfectly uniform electric field $E_{z}$ in
realistic devices. Similar to Fig 2(a), we observe a local minimum in the
g-factor, this time at an electric field 5.45 MV/m. In both cases, as
expected, the qubit basis states are found to be composed of mainly HH states
(albeit with non-zero LH-HH mixing), such that the system is operating near
the 2-D limit.

The observation of the local minimum at 5.45 MV/m instead of 8 MV/m means that
the optimal operation point has shifted upon consideration of the realistic
device potential. We have investigated the origin of the shift, and found it
to be as follows. In the perfect 2D limit, the g-factor changes with electric
field because of the dependence of the Rashba SOI on electric field. In the
case of a QD with in-plane confinement, changes in the in-plane QD radius
change the QD’s effective g-factor at fixed electric field, even when there is
inversion symmetry (in zero electric field). We observe in the device
potential that increasing gate voltage modifies not only the Rashba electric
field, as previous identified, but also reduces the QD radius (Fig. 2(b),(c)).
In comparison, the QD radius is assumed to be fixed in the simplified
potential (Fig 2 (a)). While the gate voltage modifies both the Rashba SOI
coefficient and the QD radius, the change in the Rashba SOI has more influence
on the overall curve shape, producing the minimum of the g-factor with the
gate voltage in Fig. 2(b) at 5.45 MV/m.

We now calculate the longitudinal electric dipole for the qubit states
$\psi_{i}$ in the real device, for two different values of strain, completely
unstrained materials, and strain of $-0.6$ % similar to the value expected for
Ge quantum wells in the literature[15]. The results for the longitudinal
electric dipole are shown in Fig. 3, which is found to have stronger variation
with electric field in the unstrained material. Similarly, the transverse
dipole is about 5x larger than in the strained Ge (results not shown).
Importantly, there exists a choice of gate voltage (electric field) where the
longitudinal dipole vanishes in both cases, demonstrating the existence of an
optimal operation point where dephasing vanishes to first order in electric
field noise amplitude. In other words, the presence of optimal operation
points is not tied to whether or not the material is strained.

Figure 3: Plot of the longitudinal dipole in Debye for both unstrained
($\epsilon_{x},\epsilon_{z}=0$) and strained
($\epsilon_{x}=-0.0063,\epsilon_{z}=0.0044$) Ge. Increased electric field
dependence of the dipole in the unstrained case can be seen. Note the
existence of an x-axis crossing in both cases. Figure 4: Plot of the zero,
one, and two-hole regimes as a function of realistic gate potential and
$\langle Ez\rangle$ for a barrier voltage of 1.06V. Note that the one-hole
regime resides in the plunger voltage range of $\thicksim$0.181-0.191 V,
giving the region a width of $\thicksim$10 mV. The sweet spot of Fig. 2(b)
resides within this one-hole regime. Out-of-plane magnetic field is 22mT, and
the substrate is 15.4 nm thick unstrained Ge.

Having found the optimal gate voltages where the longitudinal dipole vanishes,
we show that this can be obtained in the regime where a single hole occupies
the QD. To investigate this, we calculated the energy of the zero-hole, one-
hole and two-hole states by considering the Coulomb interactions on a single
QD[37]. We set the zero-hole energy $E_{0}$ as a reference point: then, the
one-hole energy $E_{1}$ is simply the ground state energy eigenvalue,
$\mu_{1}$, obtained from the numerical solver using the realistic gate
potential as $U(\mathbf{r})$. The two-hole energy $E_{2}$ is calculated as
$2\mu_{1}+U_{C}$, where $U_{C}$ is the Coulomb repulsion integral between two
holes residing in the same QD,

$U_{C}=\iint{|\psi({\bf
r_{1}})|}^{2}\frac{q}{4\pi\epsilon_{0}\epsilon_{r}|{\bf r_{1}}-{\bf
r_{2}}|}{|\psi({\bf r_{2}})|}^{2}d^{3}{\bf r_{1}}d^{3}{\bf r_{2}}$ (5)

and $\psi$ is the ground-state wavefunction of a single hole obtained from the
numerical solver. We have calculated the Coulomb repulsion integral by Monte-
Carlo integration. The results for $E_{0}$, $E_{1}$ and $E_{2}$ are plotted as
a function of applied gate voltage and average electric field in Fig. 4. The
one-hole regime occurs when the energy of the single-hole state is lower than
the zero and two-hole states, $E_{1}<E_{0}$ and $E_{1}<E_{2}$.

To line up the one-hole regime, where $E_{1}<E_{2}$ and $E_{1}<E_{0}$, with
the local minimum of the g-factor, we take advantage of the fact that the QD
experiences a potential produced by multiple in-plane gates. We employ the
intuition that the difference between the plunger and the barrier gate
voltages mostly controls electric field $\langle Ez\rangle$ experienced by the
hole, which should set the location of the sweet spot, and the mean value of
the plunger and barrier controls mainly occupation. Varying the barrier gate
voltage, we find that this shifts the plunger voltage where the g-factor
minimum is found in Fig. 2(b), and that the average value $\langle Ez\rangle$
of the electric field where the g-factor minimum is found is roughly constant,
in spite of the changing plunger gate voltage. This allows us to line up the
local minimum at 5.45 MV/m with the one-hole regime in Fig. 4.

As a consistency check, we examine the lever-arm of the plunger gate, which
indicates the change in electrochemical potential of the QD per unit change in
the electrostatic energy of the electrode[38]. The gate lever-arm is
calculated using the average value for $U_{C}$ in (5) over a sweep of the
plunger gate voltage at fixed barrier gates voltages, divided by the width of
the one-hole regime. It is found to be $\approx 0.72$, reflecting the
relatively small dielectric thickness in the device design (Fig. 1). As
expected, the lever-arm is higher than values from the literature where gates
are farther from the well. A lever-arm of $\approx 0.2$ was found for Ge QWs
capped with 22 nm GeSi and 20 nm Al2O3[15, 18]. Our predicted lever-arm is
similar to the lever-arm of 0.85 found for nanowire field-effect transistor
based QDs with effective oxide thicknesses of $<2$ nm [12].

Figure 5: $t_{Rabi}$ (blue) and $v$ (red) plotted as a function of applied
gate voltage and average electric field. The value of the driving field
$E_{AC}$ is 18 kV/m, and the out-of-plane magnetic field is 22mT. The
substrate is 15.4 nm thick unstrained Ge.

We have evaluated the transverse dipole defined in (4) in order to understand
the impact of the device on the qubit manipulation rate. The EDSR Rabi time is
obtained from the transverse dipole $v$ using

$t_{Rabi}=\frac{h}{E_{AC}|v|},$ (6)

where $E_{AC}$ is the strength of an oscillating in-plane electric field on
resonance with the qubit. The trends in $t_{Rabi}$ and $v$ are plotted as a
function of applied gate voltage and average electric field in Fig. 5.

Comparing these results to the result obtained for the harmonic potential
calibrated to the same curvature, we find that both the transverse and
longitudinal dipoles are underestimated by the harmonic potential model. Since
our harmonic potential is chosen to reproduce the same curvature at the bottom
of the anharmonic potential, the difference in the results must be due to the
long-distance behaviour of the potential. In particular, for realistic
devices, the potential $U(\mathbf{r})$ approaches a finite value far away from
the gates, and at this energy, the energy level density becomes very large.
This contrasts the harmonic potential typically used to describe the energy
spectrum of QDs[23], where the excited states are spaced equally apart. Though
clearly unphysical for sufficiently highly excited states and at large
distances where the potential diverges, the harmonic potential is convenient
because it is usually believed to describe the lowest energy states well. We
find this to not be the case for the longitudinal and transverse dipole. In
particular, the change to wavefunctions and energy level densities from the
treatment of the anharmonic potential impacts the EDSR matrix elements
greatly. In the framework of Schrieffer Wolff transformations [39, 40], the
higher excited state energies and smaller wavefunctions of the unbounded
parabolic potential would be anticipated to reduce the transverse dipole and
increase the EDSR time compared to the realistic potential.

We find the underestimate is a factor of $\thicksim$36 for the transverse
dipole, and $\thicksim$3 for the longitudinal dipole, for the device we have
considered. This is a remarkable quantitative difference between two
approaches that are so similar at first glance, and highlights the importance
of considering realistic device potentials to produce proper qubit performance
predictions. The increase by 36x of the transverse electric dipole compared to
a harmonic potential implies a ¿1000x increase in long-range two-qubit
coupling mediated by electric dipoles, which is relevant to circuit quantum
electrodynamics and direct electric dipole-dipole interactions[24, 25].

Figure 6: Qubit properties for in-plane magnetic field. (a) Plunger voltage
dependence of g-factor and corresponding Larmor frequency, $\omega_{L}$, for
different orientations of magnetic field $B=2$ T in xy-plane. $B$ is along
x-axis (y-axis) at $0^{\circ}$ ($90^{\circ}$); blue (red) line. The circles at
$30^{\circ},45^{\circ}$ and $60^{\circ}$ lines highlight the locations of the
sweet spot. (b) Plunger voltage dependence of electric dipole moment along
x-axis when magnetic field is along y-axis, and corresponding Rabi time for
driving field $E_{AC}=18$ kV/m.

Finally, we investigated the properties of the hole spin qubit for in-plane
magnetic fields that can be more readily tolerated by superconducting devices.
The in-plane g-factor is around 0.2, as shown in Fig. 6(a), which closely
matches the experimentally reported values in planar QDs [15, 41, 10, 42].
Compared to the out-of-plane g-factor, the small in-plane g-factor requires
two orders of magnitude larger magnetic field to operate at the same Larmor
frequency. This large anistropy is consistent with previous theoretical
predictions [43, 44], and has been demonstrated experimentally in planar QDs
[42]. Moreover, the vertical electric field dependence of the in-plane
g-factor depends on the magnetic field direction in the xy-plane. In
particular, the difference between 0∘ and 90∘, two equivalent axes in cubic
materials, is due to the absence of perfect circular symmetry in the realistic
confining potential. Interestingly, the g-factor sensitivity to the magnetic
field direction gives an extra degree of freedom for controlling the plunger
voltage and the electric field at which the sweet spot occurs, as illustrated
in Fig. 6(a). We emphasize here the importance of the cubic g-factor term for
in-plane magnetic field calculations, as the effect of the linear Zeeman term
is negligible compared to it. This applies for both the harmonic and the
realistic potentials. In contrast, the effective out-of-plane g-factor is
totally dominated by the linear Zeeman term. Fig. 6(b) shows the variation of
$v$ and the associated $t_{Rabi}$ versus vertical electric field when $B$ is
along y-axis. At low electric field, $v$ goes up to 0.8 Debye, which is about
7 times greater than when $B$ is out of plane for the same Larmor frequency.
Therefore, qubits defined by in-plane field can be manipulated faster.

Conclusion- The theoretical model in this paper is the first we are aware of
for describing qubits based on hole spin in the valence band that takes into
account a realistic description of the device, strain, and a realistic quantum
mechanical theory for qubits in the valence band. It is found that (a) optimal
operation points can still be identified for realistic devices where gate
voltage not only controls the Rashba coefficient, but also changes the QD
radius, and that the optimal operation points occur for strained and
unstrained devices, (b) the qubit manipulation rate is significantly
underestimated by harmonic potentials compared to realistic anharmonic device
potentials, (c) in-plane magnetic field direction allows for an extra degree
of freedom in tuning the location of the sweet spot and (d) in-plane magnetic
field leads to a sevenfold improvement in qubit manipulation time at the same
Larmor frequency. The increase by 36x of the transverse electric dipole
compared to a harmonic potential implies a $>1000$x increase in long-range
two-qubit coupling mediated by electric dipoles, a significant result. From a
high level, these results demonstrate a minimal model to evaluate the
properties of qubits from device layouts, and underscore the need to describe
both the device and quantum mechanical states of the valence band. In the
future, this work could be extended to describe multi-qubit circuits.

###### Acknowledgements.

This work was undertaken with support from the Stewart Blusson Quantum Matter
Institute, the National Science and Research Council of Canada through the
Discovery Grant scheme, the Canadian Foundation for Innovation through the
John Edwards Leaders Foundation scheme, and the Canada First Research
Excellence Fund, Quantum Materials and Future Technologies Program. AC
acknowledges financial support from NSERC Canada Graduate Scholarship and the
NSERC CREATE program in Quantum Computing. AC and MK acknowledge financial
support from the QMI QuEST fellowship program. The authors acknowledge CMC
Microsystems for the provision of computer aided design tools that were
essential to obtain the results presented here. This research was supported in
part through computational resources provided by Advanced Research Computing
at the University of British Columbia.

## References

  * Nigg  _et al._ [2012] S. E. Nigg, H. Paik, B. Vlastakis, G. Kirchmair, S. Shankar, L. Frunzio, M. H. Devoret, R. J. Schoelkopf, and S. M. Girvin, Black-box superconducting circuit quantization, Phys. Rev. Lett. 108, 240502 (2012).
  * Yan  _et al._ [2020] F. Yan, Y. Sung, P. Krantz, A. Kamal, D. K. Kim, J. L. Yoder, T. P. Orlando, S. Gustavsson, and W. D. Oliver, Engineering framework for optimizing superconducting qubit designs (2020), arXiv:2006.04130 [quant-ph] .
  * Minev  _et al._ [2021] Z. K. Minev, Z. Leghtas, S. O. Mundhada, L. Christakis, I. M. Pop, and M. H. Devoret, Energy-participation quantization of Josephson circuits, npj Quantum Information 7, 131 (2021), arXiv:2010.00620 [quant-ph] .
  * Minev  _et al._ [2021] Z. K. Minev, T. G. McConkey, M. Takita, A. D. Corcoles, and J. M. Gambetta, Circuit quantum electrodynamics (cqed) with modular quasi-lumped models (2021), arXiv:2103.10344 [quant-ph] .
  * Loss and DiVincenzo [1998] D. Loss and D. DiVincenzo, Quantum computation with quantum dots, Phys. Rev. A 57 (1998).
  * Friesen  _et al._ [2003] M. Friesen, P. Rugheimer, D. E. Savage, M. G. Lagally, D. W. van der Weide, R. Joynt, and M. A. Eriksson, Practical design and simulation of silicon-based quantum-dot qubits, Phys. Rev. B 67, 121301 (2003).
  * Mohiyaddin  _et al._ [2019] F. A. Mohiyaddin, G. Simion, N. I. D. Stuyck, R. Li, F. Ciubotaru, G. Eneman, F. M. Bufler, S. Kubicek, J. Jussot, B. Chan, T. Ivanov, A. Spessot, P. Matagne, J. Lee, B. Govoreanu, and I. P. Raduimec, Multiphysics simulation amp; design of silicon quantum dot qubit devices, in  _2019 IEEE International Electron Devices Meeting (IEDM)_ (2019) pp. 39.5.1–39.5.4.
  * Mohiyaddin  _et al._ [2020] F. A. Mohiyaddin, G. Simion, N. I. D. Stuyck, R. Li, A. Elsayed, M. Shehata, S. Kubicek, C. Godfrin, B. T. Chan, J. Jussot, F. C. ubotaru, S. Brebels, F. M. Bufler, G. Eneman, P. Weckx, P. Matagne, A. Spessot, B. Govoreanu, and I. P. Radu, Tcad-assisted multiphysics modeling amp; simulation for accelerating silicon quantum dot qubit design, in  _2020 International Conference on Simulation of Semiconductor Processes and Devices (SISPAD)_ (2020) pp. 253–256.
  * Watson  _et al._ [2018] T. F. Watson, S. G. J. Philips, E. Kawakami, D. R. Ward, P. Scarlino, M. Veldhorst, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, M. A. Eriksson, and L. M. K. Vandersypen, A programmable two-qubit quantum processor in silicon, Nature (London) 555, 633 (2018), arXiv:1708.04214 [cond-mat.mes-hall] .
  * Hendrickx  _et al._ [2021] N. W. Hendrickx _et al._ , A four-qubit germanium quantum processor, Nature (London) 581 (2021).
  * Veldhorst  _et al._ [2014] M. Veldhorst _et al._ , An addressable quantum dot qubit with fault-tolerant control-fidelity, Nat. Nanotech. 9 (2014).
  * Voisin  _et al._ [2016] B. Voisin, R. Maurand, S. Barraud, M. Vinet, X. Jehl, M. Sanquer, J. Renard, and S. De Franceschi, Electrical control of g‑factor in a few-hole silicon nanowire mosfet, Nano letters 16 (2016).
  * Maurand  _et al._ [2016] R. Maurand _et al._ , A CMOS silicon spin qubit, Nature Comms 7 (2016).
  * Veldhorst  _et al._ [2015] M. Veldhorst, C. H. Yang, J. C. C. Hwang, W. Huang, J. P. Dehollain, J. T. Muhonen, S. Simmons, A. Laucht, F. E. Hudson, K. M. Itoh, A. Morello, and A. S. Dzurak, A two-qubit logic gate in silicon, Nature (London) 526, 410 (2015), arXiv:1411.5760 [cond-mat.mes-hall] .
  * Hendrickx  _et al._ [2020a] N. Hendrickx, W. Lawrie, L. Petit, A. Sammak, G. Scappucci, and M. Veldhorst, A single-hole spin qubit, Nature communications 11, 1 (2020a).
  * Watzinger  _et al._ [2018] H. Watzinger _et al._ , A germanium hole spin qubit, Nature Comms 9 (2018).
  * Jirovec  _et al._ [2021] D. Jirovec _et al._ , A singlet-triplet hole spin qubit in planar Ge, Nature Mat. 20 (2021).
  * Hendrickx  _et al._ [2020b] N. W. Hendrickx _et al._ , Fast two-qubit logic with holes in germanium, Nature 577 (2020b).
  * Muhonen  _et al._ [2014] J. T. Muhonen, J. P. Dehollain, A. Laucht, F. E. Hudson, R. Kalra, T. Sekiguchi, K. M. Itoh, D. N. Jamieson, J. C. McCallum, A. S. Dzurak, and A. Morello, Storing quantum information for 30 seconds in a nanoelectronic device, Nature Nanotechnology 9, 986 (2014), arXiv:1402.7140 [cond-mat.mes-hall] .
  * Kobayashi  _et al._ [2021] T. Kobayashi _et al._ , Engineering long spin coherence times of spin-orbit qubits in silicon, Nature Mat. 20 (2021).
  * van der Heijden  _et al._ [2018] J. van der Heijden _et al._ , Readout and control of the spin-orbit states of two coupled acceptor atoms in a silicon transistor, Science Advances 4 (2018).
  * Itoh and Watanabe [2014] K. M. Itoh and H. Watanabe, Isotope engineering of silicon and diamond for quantum computing and sensing applications, MRS Commun. 4 (2014).
  * Bulaev and Loss [2007] D. V. Bulaev and D. Loss, Electric dipole spin resonance for heavy holes in quantum dots, Phys. Rev. Lett. 98, 097202 (2007).
  * Kloeffel  _et al._ [2013] C. Kloeffel, M. Trif, P. Stano, and D. Loss, Circuit qed with hole-spin qubits in ge/si nanowire quantum dots, Phys. Rev. B 88, 241405 (2013).
  * Salfi  _et al._ [2016] J. Salfi, J. A. Mol, D. Culcer, and S. Rogge, Charge-insensitive single-atom spin-orbit qubit in silicon, Phys. Rev. Lett. 116, 246801 (2016).
  * Veldhorst  _et al._ [2018] M. Veldhorst _et al._ , Silicon CMOS architecture for a spin-based quantum computer, Nature Comms. 8 (2018).
  * Wang  _et al._ [2021] Z. Wang _et al._ , Optimal operation points for ultrafast, highly coherent Ge hole spin-orbit qubits, npj Quantum Information 7 (2021).
  * Testelin  _et al._ [2009] C. Testelin, F. Bernardot, B. Eble, and M. Chamarro, Hole–spin dephasing time associated with hyperfine interaction in quantum dots, Phys. Rev. B 79, 195440 (2009).
  * Bosco  _et al._ [2021] S. Bosco, B. Hetenyi, and D. Loss, Hole spin qubits in finfets with fully tunable spin-orbit coupling and sweet spots for charge noise, PRX Quantum 2, 010348 (2021).
  * Bosco and Loss [2021] S. Bosco and D. Loss, Fully tunable hyperfine interactions of hole spin qubits in si and ge quantum dots, Phys. Rev. Lett. 127, 190501 (2021).
  * Terrazos  _et al._ [2021] L. Terrazos _et al._ , Theory of hole-spin qubits in strained germanium quantum dots, Physical Rev. B 103 (2021).
  * Marcellina  _et al._ [2017] E. Marcellina, A. R. Hamilton, R. Winkler, and D. Culcer, Spin-orbit interactions in inversion-asymmetric two-dimensional hole systems: A variational analysis, Phys. Rev. B 95, 075305 (2017).
  * Luttinger and Kohn [1955] J. M. Luttinger and W. Kohn, Motion of electrons and holes in perturbed periodic fields, Phys. Rev. 97, 869 (1955).
  * Bir  _et al._ [1963a] G. Bir, E. Butekov, and G. Pikus, Spin and combined resonance on acceptor centres in ge and si type crystals—i, Journal of Physics and Chemistry of Solids 12 (1963a).
  * Bir  _et al._ [1963b] G. Bir, E. Butekov, and G. Pikus, Spin and combined resonance on acceptor centres in ge and si type crystals—ii, Journal of Physics and Chemistry of Solids 12 (1963b).
  * Lawaetz [1971] P. Lawaetz, Valence-band parameters in cubic semiconductors, Phys. Rev. B 4, 3460 (1971).
  * Yang  _et al._ [2011] S. Yang _et al._ , Generic hubbard model description of semiconductor quantum-dot spin qubits, Phys. Rev. B 83 (2011).
  * Kiyama  _et al._ [2018] H. Kiyama _et al._ , Single-electron charge sensing in self-assembled quantum dots, Scientific reports 8, 13188 (2018).
  * Schrieffer and Wolff [1966] J. R. Schrieffer and P. A. Wolff, Relation between the anderson and kondo hamiltonians, Physical review 149, 491 (1966).
  * Winkler [2003] R. Winkler,  _Spin-orbit coupling effects in two-dimensional electron and hole systems_ , Vol. 191;191.; (Springer, New York;Berlin;, 2003).
  * Hendrickx  _et al._ [2020c] N. Hendrickx, D. Franke, A. Sammak, G. Scappucci, and M. Veldhorst, Fast two-qubit logic with holes in germanium, Nature 577, 487 (2020c).
  * Miller  _et al._ [2021] A. J. Miller, M. Brickson, W. J. Hardy, C.-Y. Liu, J.-Y. Li, A. Baczewski, M. P. Lilly, T.-M. Lu, and D. R. Luhman, Effective out-of-plane g-factor in strained-ge/sige quantum dots, arXiv preprint arXiv:2102.01758 (2021).
  * Nenashev  _et al._ [2003] A. Nenashev, A. Dvurechenskii, and A. Zinovieva, Wave functions and g factor of holes in ge/si quantum dots, Physical review B 67, 205301 (2003).
  * Maier  _et al._ [2013] F. Maier, C. Kloeffel, and D. Loss, Tunable g factor and phonon-mediated hole spin relaxation in ge/si nanowire quantum dots, Physical Review B 87, 161305 (2013).