# A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large
Language Models

Chengxing Xie
Department of Computer Science
The University of Hong Kong
<EMAIL_ADDRESS>
&Difan Zou
Department of Computer Science
Institute of Data Science
The University of Hong Kong
<EMAIL_ADDRESS>
This work was done when Chengxing Xie was an intern at HKU.

###### Abstract

Recent studies have highlighted their proficiency in some simple tasks like
writing and coding through various reasoning strategies. However, LLM agents
still struggle with tasks that require comprehensive planning, a process that
challenges current models and remains a critical research issue. In this
study, we concentrate on travel planning, a Multi-Phases planning problem,
that involves multiple interconnected stages, such as outlining, information
gathering, and planning, often characterized by the need to manage various
constraints and uncertainties. Existing reasoning approaches have struggled to
effectively address this complex task [54]. Our research aims to address this
challenge by developing a human-like planning framework for LLM agents, i.e.,
guiding the LLM agent to simulate various steps that humans take when solving
Multi-Phases problems. Specifically, we implement several strategies to enable
LLM agents to generate a coherent outline for each travel query, mirroring
human planning patterns. Additionally, we integrate Strategy Block and
Knowledge Block into our framework: Strategy Block facilitates information
collection, while Knowledge Block provides essential information for detailed
planning. Through our extensive experiments, we demonstrate that our framework
significantly improves the planning capabilities of LLM agents, enabling them
to tackle the travel planning task with improved efficiency and effectiveness.
Our experimental results showcase the exceptional performance of the proposed
framework; when combined with GPT-4-Turbo, it attains $10\times$ the
performance gains in comparison to the baseline framework deployed on
GPT-4-Turbo.

##  1 Introduction

Recently, large language models (LLMs), such as GPTs [1, 34] and LLaMAs [44],
have demonstrated remarkable potential in handling diverse tasks across
various domains, showcasing impressive generalization capabilities. [46] and
[33] show that LLM agents could play card games with human-like proficiency
and, in some instances, even outperform most human players in some games [10].
Moreover, [61], [9], and [11] revealed that LLM agents could solve daily tasks
such as cooking and door-opening. Additionally, LLM agents can also establish
complex software systems through collaboration with different agent roles [38,
18].

Nevertheless, LLM agents still struggle with certain tasks [45, 51, 23], which
are relatively solvable to humans. We categorize these challenges as Multi-
Phases Planning Tasks. These tasks are difficult because their solutions can
be divided into multiple interconnected phases, each requiring shared
information and having dependencies with each other. For example, creating a
travel plan [54] requires humans to plan for multiple days, encompassing
various aspects. This involves addressing different elements in different
phases of the planning process. Similarly, solving GitHub issues involves
checking various functions and systematically locating and resolving code
problems one by one [23]. The procedures for solving these tasks are often not
readily available online and cannot be easily converted into instruction fine-
tuning data formats to train LLMs. Consequently, tackling these types of tasks
remains a significant challenge for today’s LLM agents [45, 51, 23, 54].

Figure 1: Our human-like planning framework consists of three key parts. In
the Outline Generation phase, LLM agents produce rough plans and identify key
information related to the query, establishing the foundation for detailed
future planning. In the Information Collection phase, LLM agents gather the
essential data required for comprehensive planning. Finally, in the Plan
Making phase, LLM agents explore the potential plan space and return a well-
structured, reasonable plan.

We categorize the stages of solving such tasks into three main phases,
detailed as follows: Outline Generation Phase: When confronted with a Multi-
Phase Problem, individuals typically begin by sketching a preliminary outline
outlining the path toward a solution. Information Collection Phase: Generally,
to solve this type of task, the information provided in the task description
alone is completely insufficient. Therefore, individuals must proactively
identify the required information and ascertain how to obtain it effectively.
Plan Making Phase: Armed with the necessary information, individuals proceed
to formulate a plan. Multi-phase problems inherently entail numerous
constraints, encompassing factors such as human resources, physical
limitations, and more. Consequently, devising effective plans necessitates
meticulous consideration of diverse constraints. Given the challenges posed by
Multi-Phase Problems, we have chosen travel planning as our focal point for
several reasons. Firstly, it is a time-consuming and often challenging problem
for many people, demonstrating its difficulty. Secondly, travel planning
inherently involves a multitude of constraints, requires long-term
strategizing, and demands significant travel-related information gathering.
These complexities make travel planning a fitting subject for exploring and
improving planning strategies.

Previous reasoning strategies [59, 49, 42] have shown limited effectiveness in
addressing the complexities of travel planning [54]. However, by understanding
and emulating the processes humans use to tackle Multi-Phases Problems, we can
develop more robust planning frameworks to enable LLM agents to tackle such
challenges effectively. Our proposed framework aims to capture the essence of
human reasoning processes, particularly in the context of Multi-Phase
Problems, to enhance the performance of LLM agents.

Our framework is composed of three main components. The first is Outline
Generation. When tackling a complex problem, people often create a rough plan
to guide them through the process. Similarly, we generate a travel planning
outline using multi-agent collaboration, where each agent contributes to a
specific part of our travel outline, establishing a foundational guide for
planning. The second component is Information Collection. Just as people
gather relevant travel information, LLM agents require sufficient data to
create reliable plans. This step ensures that agents have all the necessary
details for making informed decisions. The third component is Plan Making.
After collecting adequate information, the LLM agent creates detailed daily
plans. Given the inherent complexity of travel planning, we employ a plan
search and evaluation method. Multiple plans are generated in each iteration,
and an evaluation agent identifies the best plan and flags errors in each
option. Our framework demonstrates that this structured approach leads to
impressive performance on this complex task, improving the complex task
planning ability of LLM agents. The major findings of our work are summarized
as follows:

  * •

Our work underscores the challenges LLM agents face with Multi-Phase Planning
tasks, highlighting the necessity of employing a novel reasoning framework for
effective problem-solving.

  * •

We found that utilizing a human reasoning framework for complex tasks is a key
factor in enhancing LLM agents’ performance on these tasks.

  * •

By identifying key factors contributing to effective problem-solving and
integrating them into a human-like planning framework, we enable LLM agents to
exhibit human-like reasoning, resulting in impressive performance
improvements, significantly enhancing LLM agents’ performance in travel
planning tasks.

  * •

Through extensive experimentation, we validate the effectiveness of our
approach, demonstrating its capability to tackle complex planning problems
with remarkable success.

##  2 Related Works

###  2.1 Reasoning Strategy for LLM Agents

Planning without Additional Components: In recent times, the development of
large language models (LLMs) agents has enabled numerous tasks to be addressed
by directly inputting questions into these models. However, LLM agents often
struggle with some problem-solving, particularly in areas like mathematics and
other intricate planning tasks. To address this limitation, recent research
has developed various reasoning strategies aimed at enhancing the problem-
solving abilities of LLM agents [20]. One common strategy, employed by works
like [49, 59, 41], leverages the divide-and-conquer approach. This involves
breaking down a complex task into simpler subtasks and allowing the LLM agents
to tackle each subtask sequentially. This method has shown improved
performance across a range of tasks. Other works [47, 58, 3, 63, 52, 16]
employ similar strategies using tree searches, such as Monte Carlo Tree Search
(MCTS), A*, Breadth-First Search (BFS), and Depth-First Search (DFS). They
generate multiple alternative plans via various sampling methods and select
the optimal plan through different selection techniques. Reflecting on and
refining plans based on prior experiences also significantly improves the
planning process of LLM agents [42, 13, 31, 19]. By reevaluating their plans,
LLM agents can avoid recurring errors, thereby enhancing their problem-solving
capabilities.

Planning with Addition Components: The planning capabilities of LLM agents can
also be improved through interactions with other components. For instance,
[28], [14], [8], and [7] combine symbolic planners with LLM agents, utilizing
the natural language generation capabilities of LLMs to create formalized task
descriptions. Other works [29, 24, 57, 32] enhance the problem-solving ability
of LLMs by integrating additional memory modules. A particularly effective
approach involves enabling LLM agents to access various APIs and tools [35,
39, 40, 48, 56, 30, 37, 60]. Some tools can be directly included in the
context [43, 41, 50] as prompt. However, improving an LLM agent’s tool-using
abilities often requires generating appropriate datasets and fine-tuning the
models [40, 37, 56]. By providing access to external tools, LLM agents can
tackle tasks that are otherwise challenging, such as computational problems
and formula verification. Our planning framework integrates these diverse
reasoning strategies to improve overall planning capabilities, allowing for
more sophisticated problem-solving and strengthening the effectiveness of LLM
agents in handling complex tasks.

###  2.2 Multi-Agents Framework

Multi-agent frameworks [15] have garnered significant interest from
researchers due to their flexibility across a broad range of tasks. These
frameworks can be utilized in simulations of various kinds, including game
simulations [55, 46, 33], economic simulations [26, 27], and societal
simulations [12, 53, 36, 62, 2]. Moreover, multi-agent frameworks demonstrate
superior performance compared to single-agent systems in solving diverse
tasks. For instance, [25], [17], and [6] have developed general multi-agent
frameworks that enhance task performance. For specific tasks, works like [38,
17, 18] propose specialized frameworks that enable agents to autonomously
develop software. In addition, multi-agent systems have been effectively
employed for reasoning in embodied environments [9, 61] and have demonstrated
their potential in scientific research [4, 5]. Our framework also incorporates
a multi-agent system to enhance overall planning capabilities, leveraging the
collaborative and complementary strengths of multiple agents for more
effective problem-solving.

##  3 Method

###  3.1 Task Description:

We introduce the task setting of TravelPlanner [54], where the objective is
for LLM agents to provide a reasonable travel plan based on a given query,
while adhering to the specified constraints and leveraging commonsense
knowledge. Each day’s plan includes details about the current city,
attractions, accommodation, transportation, and dining options (See Appendix
A.6 for travel plan examples). A reasonable plan must comply with commonsense
principles and meet all requirements specified in the travel query, such as
avoiding repeated visits to the same attractions and ensuring the plan stays
within the specified budget. An initial analysis of GPT-4-Turbo’s performance
in [54] revealed that the most advanced model still struggles with efficient
information collection and valid plan making. The details are as follows:

  * •

Generating unreasonable travel routes, such as a number of cities visited that
is inconsistent with the query or not returning to the original city on the
last day.

  * •

Utilizing unreasonable transportation methods, like combining self-driving and
flight in one trip.

  * •

The plan made may include unreal information (hallucination) in the plans.

  * •

LLM agents may omit essential information when making the plan and disregard
the constraints specified in the query, leading to problematic and incomplete
plans.

To address the issues of unreasonable travel routes and transportation
methods, we employ the Outline Generation phase (Section 3.3) to create
logical outlines and appropriate transportation options. To alleviate the
hallucination problem, we enhance the Information Collection phase (Section
3.4) to ensure more accurate data gathering. Finally, to deal with the missing
information during plan creation, we utilize the Plan Making phase (Section
3.5) to support the development of correct and comprehensive plans.

###  3.2 Framework Overview

Our framework, as shown in Fig 1, employs a human-like reasoning approach to
address travel planning problems. Based on the specific query, we first
generate an outline for the query (Section 3.3). Following the Outline
Generation, LLM agents equipped with Strategy block and Knowledge Block
proceed with Information Collection (Section 3.4). During this stage, once
sufficient information is gathered for a single day’s plan, the agent creates
the daily plan in Plan Making phase (Section 3.5). The final travel plan is
composed of each daily plan.

###  3.3 Outline Generation

Each travel query (details in Appendix A.5) involves numerous elements and
spans several days, making it challenging to generate a comprehensive plan all
at once. When humans tackle such tasks, they typically begin by drafting a
rough outline. This outline should encompass the route, transportation
options, and key points in the query, facilitating the planning process
(details in Appendix A.7). To replicate this approach, we introduce the Route
Generation phase, which incorporates transportation evaluation to determine
the route and transportation components. Additionally, we integrate two
additional agents to generate the key points of the travel, enhancing the
usefulness of the outlines.

Figure 2: This is our transportation evaluation process for each route
generated by PathFinder Agent. We will evaluate whether this route is
reasonable under the limitation from the specific travel planning query, e.g.,
cannot take flight. If not reasonable, we will redo the route by giving
feedback. Otherwise, if the route is not reasonable for some kind of
transportation, we will add the constraint.

Route Generation: Drawing inspiration from human reasoning patterns, we’ve
developed the Route Generation component of Outline Generation. Firstly, the
query is first passed to the PathFinder Agent. This agent generates a rough
route for the entire trip, including city transfers and exploratory travel
information. This preliminary route serves as a guide for the subsequent
Information Collection and Plan Making phases, providing a clear roadmap for
the journey and making the planning process more structured and transparent.

Transportation Evaluation: When constructing a travel route, humans typically
do not initially verify transportation availability between target cities,
assessing options later based on preferences and availability. Similarly,
we’ve observed that LLM agents may generate invalid routes due to a lack of
detailed transportation information between cities. To address this issue, we
propose adding an evaluation stage after route generation. As shown in Fig 2,
during this stage, each route is evaluated to determine its rationality from a
transportation perspective. If the route is perfect, we don’t need to do
anything. If the route cannot be traveled by driving, we inform the agent
accordingly. Similarly, if the route cannot be reached by any means of
transportation, we provide the feedback to the PathFinder Agent and prompt the
agent to revise the route accordingly.

Keypoints Generation: When humans receive a query, they focus on the key
points and ignore irrelevant details. Inspired by this, we introduced the
Keypoints Agent (Fig 1 left) to identify the critical points in the query that
need to be considered by the Plan Agent. Moreover, as [54] demonstrated, LLM
agents tend to overlook common sense when planning, such as navigating the
same restaurant or attraction. To address this issue, the Commonsense Agent
(Fig 1 left) generates basic guides related to travel planning. This agent
doesn’t have access to the specific query, as we believe these guides are
common sense and universally applicable to every query.

These three parts combine with each other to generate a useful and detailed
travel outline, building a strong foundation for the planning task.

###  3.4 Information Collection

After the Outline Generation phase, humans proceed to collect necessary
travel-related information, such as specific attractions and restaurants.
Collecting sufficient information is crucial as it can significantly decrease
the probability of hallucination. In our framework, the Information Collection
process begins with the Thought Agent generating the next steps based on the
Strategy Block. Subsequently, the Tool Agent utilizes the output of the
Thought Agent to generate a suitable function expression. The result of this
function, primarily comprising travel information, is then recorded in the
Knowledge Block with a description from the Description Agent. Finally, the
selected information is forwarded to the Plan Agent for generating the daily
travel plan.

Strategy Block: In the information collection process, humans typically
remember the types of information they have gathered, which guides their
subsequent steps. To emulate this procedure, we introduce a component called
the Strategy Block as shown in Fig 1, primarily used by the Thought Agent.
First, the outline is stored in the Strategy Block to guide the information
collection process. The Strategy Block also informs the Thought Agent which
day it is in the travel plan and short descriptions of the collected data.
Besides, we streamline the prompts for better workflow management. Compared to
the scratchpad approach in [54], our design provides more task-specific
details and better supports the Thought Agent in collecting information.

Tool Agent: We’ve observed that as the planning process progresses, the
context length can become too long, leading to potential oversight of tool
documents placed at the beginning of input messages. Thus, we introduce the
Tool Agent tasked with generating the correct function expression format
messages. The Tool Agent’s input includes only the tool document, the query,
and some previous content from the Strategy Block to ensure it can generate
the correct information based on the context.

Figure 3: This image illustrates the Knowledge Block workflow. The left part
is the Knowledge Write process, where a function’s result is written to the
top of the Knowledge Block. About Knowledge Read, when the knowledge needs to
be popped out but is below the threshold, the Knowledge Block pops out enough
items to meet the threshold. If the number exceeds the threshold, the items in
past two days are popped out.

Knowledge Block: Humans will make their plans by examining the detailed
information they have previously collected. To mimic this process, we
introduce the Knowledge Block, as shown in Fig 1, primarily used by the Plan
Agent. In Fig 3 left part, this block will automatically record all
information and the description of each information, which is generated by
Description Agent. However, when travel plans span long periods, the extensive
information can make it difficult to extract pertinent details and may even
exceed the LLM’s context window length. To address this, we implement a stack-
like structure within the Knowledge Block. When the Plan Agent needs detailed
information, the block only “pops out” the information collected over the past
two days, restricted by a minimal threshold. As shown in Fig 3 If the number
of knowledge items collected exceeds the threshold, the block pops out the
same number of items. If it exceeds the threshold, all knowledge items from
the past two days are popped out.

Lastly, we found that presenting the data in a DataFrame format makes it
difficult to read and interpret. Therefore, we have reorganized the
information into a more reader-friendly structure. We directly specify each
data’s name just before presenting the data, like: Flight Number: F3502691
(see Appendix A.7.2 for more details).

###  3.5 Plan Making

When humans make plans, they typically do not plan everything at once;
instead, they approach it step by step. Therefore, unlike the approach in [54]
that plans the entire trip at once, we adopt a daily planning strategy. This
method requires less information at each step, making the planning process
easier. Each time the Thought Agent determines that sufficient information has
been collected for a specific day, the Tool Agent calls the DailyPlanner tool,
and the Plan Agent creates the daily plan.

Plan Search: When examining the results of Plan Agent, we observe that
creating a plan often introduces various errors. These errors can stem from
insufficient information collected during the information collection stage or
from oversights during the planning stage. To address this issue, we propose a
plan search method.

Each time the Plan Agent generates a daily plan, it creates several plans
based on the same information. An Evaluate Agent reviews each plan, converting
them into JSON format and using code to identify and rank errors. Based on
these evaluations, we select the best plan. If significant errors are found,
these errors are recorded in the Strategy Block, prompting further information
collection, and the plans with errors are discarded. This discard process
occurs only once per day’s plan to prevent unlimited planning iterations.
Through daily planning and plan search strategies, the agent can generate more
reasonable and reliable travel plans.

##  4 Experiment

Table 1: Our experimental results on the validation dataset of the
TravelPlanner benchmark demonstrate that our framework enables GPT-3.5 to
surpass previous GPT-4’s performance. Additionally, our framework increases
the Final Pass Rate of GPT-4-Turbo $10\times$ compared to previous algorithms.
For each model, using our framework results in significantly improved
performance.

Model |  Delivery Rate |  | Commonsense Pass Rate
---
| Hard Constraint Pass Rate
---
Final Pass Rate
| Micro | Macro | Micro | Macro
TravelPlanner Result
Mistral-7B-32K [21] | $8.9$ | $5.9$ | $0$ | $0$ | $0$ | $0$
Mixtral-8×7B-MoE [22] | $49.4$ | $30.0$ | $0$ | $1.2$ | $0.6$ | $0$
GPT-3.5-Turbo [34] | $86.7$ | $54.0$ | $0$ | $0$ | $0$ | $0$
GPT-4-Turbo [1] | 89.4 | 61.1 | 2.8 | 15.2 | 10.6 | 0.6
Our Results
Mistral-7B-32K [21] | $39.4$ | $24.0$ | $1.1$ | $0.5$ | $0.6$ | $0.6$
Mixtral-8×7B-MoE [22] | $67.8$ | $40.2$ | $0$ | $0$ | $0$ | $0$
GPT-3.5-Turbo [34] | 100.0 | 75.1 | $15.6$ | $15.5$ | $4.4$ | $2.2$
GPT-4-Turbo [1] | $91.7$ | $74.6$ | 24.4 | 35.7 | 16.7 | 6.7

###  4.1 Experiment Setup

We utilize the TravelPlanner benchmark proposed by [54], which includes a
variety of travel queries with different travel lengths and difficulty levels.
This benchmark has a train, valid, and test set. The valid set has 180 queries
and the test set has 1000 queries. We use the valid set to examine our
framework as running the experiment on the test set is too expensive. We use
gpt-3.5-turbo-1106, gpt-4-1106-preview, mixtral, mistral-7B-32K in our
experiments [1, 22, 21, 34]. We ran open-source models on 4 NVIDIA RTX A6000.

Hyperparameters Settings: To ensure the reproducibility of our experimental
results, we set the temperature to 0 for all processes except for Plan Agent,
where it is set to 0.7 to allow the Plan Agent to generate diverse plans
enabling a larger search space.

For the travel route generation, we limit the maximum number of retries to 3
to prevent infinite remaking of the travel route. The Plan Agent generates 3
different plans each time. If all generated plans contain errors, we revert to
the information collection stage only once. If errors persist after this
second attempt, we proceed with the process regardless. About the Knowledge
Block, we set the minimal number of pop-outs is 5.

Metrics: We use the same metrics proposed in TravelPlanner [54].

  * •

Delivery Rate: Measures if agents can successfully deliver a final plan within
a set number of steps (max 45). Failure includes dead loops, numerous failed
attempts, or exceeding the step limit.

  * •

Commonsense Constraint Pass Rate: Assesses if agents can incorporate
commonsense into their plans without explicit instructions, across eight
dimensions.

  * •

Hard Constraint Pass Rate: Evaluates if a plan meets all explicitly given hard
constraints, testing agents’ adaptability to diverse user queries.

  * •

Final Pass Rate: Indicates the proportion of plans that meet all constraints
(delivery, commonsense, and hard constraints), reflecting agents’ overall
proficiency in producing practical plans.

  * •

Micro Pass Rate: The Micro Pass Rate evaluates the proportion of constraints
that are successfully passed by an agent’s plans, as shown in Formula 1. It is
calculated by taking the total number of successfully met constraints across
all plans and dividing it by the total number of constraints applied to all
plans.

  * •

Macro Pass Rate: The Macro Pass Rate assesses the proportion of plans that
satisfy all of their constraints. It calculates the ratio of the number of
plans that meet all applicable commonsense or hard constraints to the total
number of plans evaluated.

$\quad\text{Micro Pass Rate}=\frac{\sum_{p\in P}\sum_{c\in
C_{p}}1_{\text{passed}(c,p)}}{\sum_{p\in P}|C_{p}|},\text{Macro Pass
Rate}=\frac{\sum_{p\in P}1_{\text{passed}(C_{p},p)}}{|P|}$ (1)

$P$ is the set of all plans. $C_{p}$ is the set of constraints applicable to a
specific plan $p$. $1_{\text{passed}(c,p)}$ is an indicator function that
returns 1 if constraint $c$ is passed in plan $p$, and 0 otherwise.
$1_{\text{passed}(C_{p},p)}$ is an indicator function that returns 1 if all
constraints in $C_{p}$ are passed in plan $p$, and 0 otherwise.

###  4.2 Experiment Result

Figure 4: The error distribution about GPT-4-Turbo’s Delivery Rate Failure.

.

As shown in Table 1, we tested four LLMs on the dataset to verify our
framework’s effectiveness. All models exhibited increased Delivery Rate, with
Mistral-7B showing an improvement of 8.9% to 39.4% and GPT-3.5-Turbo achieving
100% Delivery Rate. This indicates that GPT-3.5-Turbo can generate a travel
plan for every query, strongly demonstrating that our framework enables better
planning for the task.

However, we observed limited improvement for GPT-4-Turbo on Delivery Rate. To
understand the reason, we visualized the error distribution from GPT-4-Turbo
in Fig 4. We found that the primary issue was the repeated use of the same
function three times, with the most frequent being AccommodationSearch. The
agent repeatedly attempted to find the ’correct’ room type, often overlooking
some information, which led to redundancy. Additionally, GPT-4-Turbo aimed to
find accommodations that could fit all travelers in one room, ignoring the
possibility of booking multiple rooms. The lower Delivery Rate will cause a
lower Micro Commonsense pass rate. We conjecture that GPT-4-Turbo’s stronger
reasoning capabilities hindered its progress by overcomplicating the task.

Figure 5: The error distribution of GPT-3.5-Turbo and GPT-4-Turbo.

Commonsense Pass Rate: Our framework demonstrates impressive improvements in
the Commonsense Pass Rate across all models. For GPT-3.5-Turbo, the Micro Pass
Rate increased from 54.0% to 75.1%, and the Macro Pass Rate increased from 0%
to 15.6%. For GPT-4-Turbo, the Micro Pass Rate increased from 61.1% to 74.6%,
while the Macro Pass Rate saw a significant boost from 2.8% to 24.4%. Although
Mistral and Mixtral showed improvements in their Micro Pass Rates, their Macro
Pass Rates did not see significant increases.

Analyzing the commonsense error distribution in Fig 5, we find that
GPT-3.5-Turbo and GPT-4-Turbo exhibit very similar error patterns. The top
three errors are Hallucinated Information, Necessary Information Absent, and
Invalid Accommodation, accounting for nearly 60% of all errors. Hallucinated
Information indicates that the LLM agents still generate unreal information.
Necessary Information Absent suggests that the Plan Agent might not strictly
follow instructions or the information collection phase misses key details.
Invalid Accommodation implies that some points in the accommodation data are
overlooked.

Hard Constraints Pass Rate:  Additionally, our framework enhances the Pass
Rate for Hard Constraints. For GPT-4-Turbo, the Micro Pass Rate increased from
15.2% to 35.7%, and the Macro Pass Rate increased from 10.6% to 16.7%. While
this improvement is not as substantial as that of the Commonsense Pass Rate,
it is still a notable enhancement. For GPT-3.5-Turbo, the improvements were
slightly lower, with increases from 0% to 15.5% for the Micro Pass Rate and
from 0% to 4.4% for the Macro Pass Rate. The last two models, Mistral and
Mixtral, showed only slight improvements. Examining the error distribution in
Fig 5, we observe that the main errors mainly appear in accommodation
planning. This suggests that the concentrated information about each
accommodation is challenging for LLM agents to correctly extract in making
accurate plans.

Regarding the Final Pass Rate, GPT-3.5-Turbo’s result using our framework
surpasses the GPT-4-Turbo’s result in the previous algorithm, showcasing the
effectiveness of our framework. Notably, GPT-4-Turbo’s Final Pass Rate
increased $10\times$ compared to the previous algorithm. This significant
improvement underscores our framework’s capability to enhance the performance
and reliability of LLM agents in complex travel planning tasks.

###  4.3 Ablation Study

Table 2: Ablation study results on GPT-3.5-Turbo.

|  Delivery Rate |  | Commonsense Pass Rate
---
| Hard Constraint Pass Rate
---
Final Pass Rate
| Micro | Macro | Micro | Macro
Original Result [54] | $86.7$ | $54.0$ | $0$ | $0$ | $0$ | $0$
No Outlines | $93.3$ | $59.2$ | $1.1$ | $3.8$ | $1.1$ | $0$
No Strategy | $100$ | $74.9$ | $13.9$ | $12.1$ | $2.2$ | $0.6$
No Knowledge | $99.4$ | $66.0$ | $8.3$ | $10.2$ | $1.1$ | $0$
No Plan Search | $99.4$ | $67.4$ | $0.6$ | $1.9$ | $0$ | $0$
Our Framework | 100.0 | 75.1 | 15.6 | 15.5 | 4.4 | 2.2

We analyzed the impact of each component of our framework by removing them one
at a time, as shown in Table 2. When the outline generation component was
removed, we observed a significant drop in all metrics, although they remained
better than the TravelPlanner results. This indicates that Outlines Generation
is a crucial element of our framework.

About Strategy Block, we removed some information from the Strategy Block,
such as specific daytime guidance, Knowledge Block Information, Budget
Requirements, and key points from the query. However, the rest of this block
remains necessary because, without it, the information collection process
cannot proceed. From the results, we observed that all metrics decreased.
However, the degree of decrease in Delivery Rate and Micro Commonsense Pass
Rate was not very substantial. This outcome is expected, as this component
does not significantly impact the feasibility of creating a plan but rather
the quality of the plans produced.

For the Knowledge Block, removing it entirely is impractical. Therefore, we
adopted the method from [54], using an extra tool to record information. To
prevent exceeding the maximum token limit, we implemented a maximum length
limitation. Removing this block caused a notable decline in all metrics except
Delivery Rate, echoing findings from the Strategy Block ablation study. These
outcomes underscore the critical role of the Knowledge Block.

Additionally, when we removed the Plan Search component, we observed a
dramatic drop in all metrics except Delivery Rate. The decrease was
particularly severe in the Macro Commonsense Pass Rate compared to other
ablation studies, highlighting the critical role of the Plan Search component.

Our ablation study results confirm that each block of our framework is
essential. Removing any part will significantly decrease performance across
various metrics. These parts are vital for managing information and enhancing
plan quality, ensuring a robust and efficient planning process.

##  5 Conclusion and Future Work

In this paper, we presented a novel human-like reasoning framework for travel
planning using LLM Agents. Our approach integrates several key components:
Outline Generation, Information Collection with blocks, and Plan Search. These
components synergistically mimic human problem-solving strategies, enabling
LLM agents to handle multi-phase tasks more effectively. Extensive experiments
on the TravelPlanner benchmark demonstrated significant improvements across
multiple models with our framework.

Future work can focus on applying our planning framework to real-world
planning problems in practical environments. Additionally, the current tools
within our framework are limited. Enhancing the framework by incorporating
more advanced tools, such as those with additional parameters and
capabilities, could further improve its planning and problem-solving
effectiveness. This expansion could enable more sophisticated and adaptable
solutions, making our framework even more robust in addressing complex, real-
world challenges.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai.  Using large language models to simulate multiple humans and replicate human subject studies.  In International Conference on Machine Learning, pages 337–371. PMLR, 2023.
  * [3] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al.  Graph of thoughts: Solving elaborate problems with large language models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024.
  * [4] Daniil A Boiko, Robert MacKnight, and Gabe Gomes.  Emergent autonomous scientific research capabilities of large language models.  arXiv preprint arXiv:2304.05332, 2023.
  * [5] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller.  Chemcrow: Augmenting large-language models with chemistry tools.  arXiv preprint arXiv:2304.05376, 2023.
  * [6] Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi.  Autoagents: A framework for automatic agent generation.  arXiv preprint arXiv:2309.17288, 2023.
  * [7] Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al.  Binding language models in symbolic languages.  arXiv preprint arXiv:2210.02875, 2022.
  * [8] Gautier Dagan, Frank Keller, and Alex Lascarides.  Dynamic planning with a llm.  arXiv preprint arXiv:2308.06391, 2023.
  * [9] Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus.  Collaborating with language models for embodied reasoning.  arXiv preprint arXiv:2302.00763, 2023.
  * [10] Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al.  Human-level play in the game of diplomacy by combining language models with strategic reasoning.  Science, 378(6624):1067–1074, 2022.
  * [11] Zipeng Fu, Tony Z Zhao, and Chelsea Finn.  Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.  arXiv preprint arXiv:2401.02117, 2024.
  * [12] Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Li.  S3: Social-network simulation system with large language model-empowered agents.  arXiv preprint arXiv:2307.14984, 2023.
  * [13] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen.  Critic: Large language models can self-correct with tool-interactive critiquing.  arXiv preprint arXiv:2305.11738, 2023.
  * [14] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati.  Leveraging pre-trained large language models to construct and utilize world models for model-based task planning.  Advances in Neural Information Processing Systems, 36:79081–79094, 2023.
  * [15] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang.  Large language model based multi-agents: A survey of progress and challenges.  arXiv preprint arXiv:2402.01680, 2024.
  * [16] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.  Reasoning with language model is planning with world model.  arXiv preprint arXiv:2305.14992, 2023.
  * [17] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al.  Metagpt: Meta programming for multi-agent collaborative framework.  arXiv preprint arXiv:2308.00352, 2023.
  * [18] Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui.  Agentcoder: Multi-agent-based code generation with iterative testing and optimisation.  arXiv preprint arXiv:2312.13010, 2023.
  * [19] Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, and Xing Xie.  Recommender ai agent: Integrating large language models for interactive recommendations.  arXiv preprint arXiv:2308.16505, 2023.
  * [20] Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen.  Understanding the planning of llm agents: A survey.  arXiv preprint arXiv:2402.02716, 2024.
  * [21] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [22] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [23] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.  Swe-bench: Can language models resolve real-world github issues?  arXiv preprint arXiv:2310.06770, 2023.
  * [24] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  * [25] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.  Camel: Communicative agents for" mind" exploration of large scale language model society.  2023\.
  * [26] Nian Li, Chen Gao, Yong Li, and Qingmin Liao.  Large language model-empowered agents for simulating macroeconomic activities.  arXiv preprint arXiv:2310.10436, 2023.
  * [27] Yang Li, Yangyang Yu, Haohang Li, Zhi Chen, and Khaldoun Khashanah.  Tradinggpt: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance.  arXiv preprint arXiv:2309.03736, 2023.
  * [28] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone.  Llm+ p: Empowering large language models with optimal planning proficiency.  arXiv preprint arXiv:2304.11477, 2023.
  * [29] Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, and Guannan Zhang.  Think-in-memory: Recalling and post-thinking enable llms with long-term memory.  arXiv preprint arXiv:2311.08719, 2023.
  * [30] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al.  Llava-plus: Learning to use tools for creating multimodal agents.  arXiv preprint arXiv:2311.05437, 2023.
  * [31] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.  Self-refine: Iterative refinement with self-feedback.  Advances in Neural Information Processing Systems, 36, 2024.
  * [32] Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen.  Generation-augmented retrieval for open-domain question answering.  arXiv preprint arXiv:2009.08553, 2020.
  * [33] Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton.  Welfare diplomacy: Benchmarking language model cooperation.  arXiv preprint arXiv:2310.08901, 2023.
  * [34] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in neural information processing systems, 35:27730–27744, 2022.
  * [35] Aaron Parisi, Yao Zhao, and Noah Fiedel.  Talm: Tool augmented language models.  arXiv preprint arXiv:2205.12255, 2022.
  * [36] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein.  Generative agents: Interactive simulacra of human behavior.  In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023.
  * [37] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.  Gorilla: Large language model connected with massive apis.  arXiv preprint arXiv:2305.15334, 2023.
  * [38] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun.  Communicative agents for software development.  arXiv preprint arXiv:2307.07924, 2023.
  * [39] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al.  Tool learning with foundation models.  arXiv preprint arXiv:2304.08354, 2023.
  * [40] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  Advances in Neural Information Processing Systems, 36, 2024.
  * [41] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.  Advances in Neural Information Processing Systems, 36, 2024.
  * [42] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [43] Dídac Surís, Sachit Menon, and Carl Vondrick.  Vipergpt: Visual inference via python execution for reasoning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023.
  * [44] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [45] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  arXiv preprint arXiv:2305.16291, 2023.
  * [46] Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang.  Avalon’s game of thoughts: Battle against deception through recursive contemplation.  arXiv preprint arXiv:2310.01320, 2023.
  * [47] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  arXiv preprint arXiv:2203.11171, 2022.
  * [48] Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig.  What are tools anyway? a survey from the language model perspective.  arXiv preprint arXiv:2403.15452, 2024.
  * [49] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.
  * [50] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  arXiv preprint arXiv:2303.04671, 2023.
  * [51] Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li.  Smartplay: A benchmark for llms as intelligent agents.  arXiv preprint arXiv:2310.01557, 2023.
  * [52] Hengjia Xiao and Peng Wang.  Llm a*: Human in the loop large language models enabled a* search for robotics.  arXiv preprint arXiv:2312.01797, 2023.
  * [53] Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Kai Shu, Adel Bibi, Ziniu Hu, Philip Torr, Bernard Ghanem, and Guohao Li.  Can large language model agents simulate human trust behaviors?  arXiv preprint arXiv:2402.04559, 2024.
  * [54] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su.  Travelplanner: A benchmark for real-world planning with language agents.  arXiv preprint arXiv:2402.01622, 2024.
  * [55] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu.  Language agents with reinforcement learning for strategic play in the werewolf game.  arXiv preprint arXiv:2310.18940, 2023.
  * [56] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.  Gpt4tools: Teaching large language model to use tools via self-instruction.  Advances in Neural Information Processing Systems, 36, 2024.
  * [57] Zhun Yang, Adam Ishay, and Joohyung Lee.  Coupling large language models with logic programming for robust and general reasoning from text.  arXiv preprint arXiv:2307.07696, 2023.
  * [58] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [59] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  arXiv preprint arXiv:2210.03629, 2022.
  * [60] Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang.  Easytool: Enhancing llm-based agents with concise tool instruction.  arXiv preprint arXiv:2401.06201, 2024.
  * [61] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan.  Building cooperative embodied agents modularly with large language models.  arXiv preprint arXiv:2307.02485, 2023.
  * [62] Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie.  Competeai: Understanding the competition behaviors in large language model-based agents.  arXiv preprint arXiv:2310.17512, 2023.
  * [63] Zirui Zhao, Wee Sun Lee, and David Hsu.  Large language models as commonsense knowledge for large-scale task planning.  Advances in Neural Information Processing Systems, 36, 2024.

##  Appendix A Appendix / supplemental material

###  A.1 Limitation

Although our framework has significantly improved performance compared to
previous studies, achieving a Final Pass Rate of 100% akin to human
performance remains elusive. One limitation we observed is that even when
provided with detailed information, LLM Agents still tend to overlook certain
key points. Addressing this challenge may require the integration of
alternative planning strategies. Additionally, effectively utilizing extensive
accommodation-related information to generate reasonable plans remains a
significant challenge, even for advanced models like GPT-4-Turbo. Furthermore,
the occurrence of hallucinations during the planning process presents another
obstacle that requires attention.

In future work, addressing these limitations could involve exploring novel
planning strategies that complement the capabilities of LLM Agents, such as
incorporating heuristics or domain-specific rules. Moreover, developing more
sophisticated models that can better comprehend and utilize complex
accommodation-related information could lead to more accurate and reliable
travel plans. Additionally, investigating techniques to mitigate the
occurrence of hallucinations and improve the overall robustness of LLM-based
planning systems could further enhance their performance. Overall, overcoming
these challenges will be crucial for advancing the state-of-the-art in LLM-
driven travel planning systems.

###  A.2 Impact

The social impact of our work is profound, as it pioneers a novel methodology
by emulating human reasoning processes to address Multi-Phases planning
challenges. Equipping LLM agents with human-like problem-solving abilities,
our framework enhances their efficacy in applications, such as travel
planning, thereby potentially revolutionizing various industries reliant on
complex decision-making. However, the advent of LLM agents with enhanced
planning capabilities also raises concerns about potential job displacement
and ethical implications surrounding data privacy and algorithmic bias,
necessitating careful consideration and regulation to mitigate adverse
consequences.

###  A.3 Ethics Statement

In our study, we introduce a novel human-like reasoning framework to enhance
LLM agents’ proficiency in Multi-Phases planning tasks, especially on
traveling planning problems. While our work aims to advance the capabilities
of LLM agents for beneficial purposes, we recognize the potential for misuse
or unintended consequences, such as the propagation of inaccurate information.
To address these ethical considerations, we advocate for ongoing research into
methods for detecting and mitigating biases in LLMs, as well as the promotion
of responsible use and transparency in the deployment of such frameworks. We
urge users and researchers to remain vigilant of these ethical risks and
prioritize the ethical considerations in the development and application of
our framework.

###  A.4 Example

###  A.5 Travel Query Example:

1\. Could you please create a 5-day travel itinerary for one person, starting
in Albuquerque and visiting 2 cities in Texas from March 25th to March 29th,
2022? The travel plan should work within a budget of $2,100. 2\. Could you
help create a 7-day travel plan for a group of 3, departing from Greensboro
and touring 3 different cities in Georgia from March 10th to March 16th, 2022?
We have a new budget of $4,000 for this trip. We’d also appreciate if our
accommodations have smoking areas. 3\. Could you help create a 5-day itinerary
for a travel plan departing from Grand Junction and heading to 2 cities in
Arizona from March 19th to March 23rd, 2022? It’s a plan for two people with a
budget of $2,100. Our accommodations should allow visitors and our preference
is for private rooms. Additionally, we do not require any flight
transportation. 4\. Could you create a 7-day travel itinerary for 2 people,
departing from Albuquerque and visiting 3 cities in Texas from March 8th to
March 14th, 2022? Our budget is set at $5,000. We require accommodations that
allow smoking and are preferably not shared rooms. We would prefer to avoid
any flights for our transportation. Figure 6: Few examples of the travel query

###  A.6 Travel Plan Example:

"day": 1, "current_city": "from Buffalo to Atlanta", "transportation": "Flight
Number: F3502691, from Buffalo to Atlanta, Departure Time: 18:48, Arrival
Time: 20:51", "breakfast": "-", "attraction": "Georgia Aquarium, Atlanta;World
of Coca-Cola, Atlanta;", "lunch": "-", "dinner": "Chaina Ram Sindhi
Confectioners, Atlanta", "accommodation": "Spacious private room close St.
Barnabas Hospital, Atlanta" "day": 2, "current_city": "Atlanta",
"transportation": "-", "breakfast": "Baba Au Rhum, Atlanta", "attraction":
"Atlanta Botanical Garden, Atlanta;High Museum of Art, Atlanta;", "lunch":
"Barkat, Atlanta", "dinner": "Taste of Vishal, Atlanta", "accommodation":
"Spacious private room close St. Barnabas Hospital, Atlanta" "day": 3,
"current_city": "from Atlanta to Buffalo", "transportation": "Flight Number:
F3502694, from Atlanta to Buffalo, Departure Time: 15:47, Arrival Time:
17:42", "breakfast": "Asian Bistro, Atlanta", "attraction": "Piedmont Park,
Atlanta;", "lunch": "Beliram Degchiwala, Atlanta", "dinner": "-",
"accommodation": "-" Figure 7: An example of the travel plan

###  A.7 Outline Example:

The First Day: from Buffalo to Atlanta. Exploring Atlanta. The Second Day:
Exploring Atlanta. The Third Day: from Atlanta to Buffalo. 1.Departure and
Return Dates: The travel must commence on March 2nd, 2022, and conclude with a
return to Buffalo on March 4th, 2022. 2.Solo Travel: The plan should be
tailored for a single traveler, ensuring accommodations and activities are
suitable for one person. 3.Budget Limit: The total cost of the trip, including
transportation, accommodation, meals, and activities, must not exceed $1,100.
4.Destination Specific: All travel arrangements must be made with Atlanta as
the sole destination for the duration of the trip. 5.Time Frame: The entire
travel plan, including all experiences and bookings, must fit within the 3-day
window specified. 1\. Accommodations should be booked for the correct dates
corresponding to the traveler’s stay in each city, ensuring no gaps or
overlaps in lodging. 2\. Dining options for breakfast, lunch, and dinner
should be reasonably spaced throughout the day and geographically convenient
to either the accommodation or planned attractions. 3\. Transportation
arrangements must align with the traveler’s schedule, allowing sufficient time
for check-in, security clearance at airports, and transfers between different
modes of transport. 4\. Attractions should be planned with consideration of
their operating hours and any required advance reservations or tickets,
avoiding scheduling conflicts. 5\. The sequence of visiting attractions should
be logical and efficient, minimizing backtracking and excessive travel time
between locations. 6\. Ensure that the transportation method selected is
appropriate for the distance and time required to reach the scheduled
activities, such as not planning long-distance road trips between breakfast
and lunch. 7\. Dining reservations should be made with consideration of
dietary restrictions, restaurant capacity, and the expected duration of the
meal to maintain the day’s itinerary flow. 8\. Verify that the accommodation
has the necessary amenities and services required for the traveler’s comfort
and convenience, such as Wi-Fi if needed for planning or work purposes. Figure
8: An example of the outline

####  A.7.1 Strategy Block Content:

Below is a preliminary outline of your trip, which can serve as a reference
for collecting information: The First Day: from Ontario to Honolulu. Exploring
Honolulu. The Second Day: Exploring Honolulu. The Third Day: from Honolulu to
Ontario. You now need to gather relevant information to specify the travel
plan for the First day. Remember that the total daily expenses of your trip
(the sum of expenses for each person) do not exceed 1066. Hard Constraints.
Total Budget: The trip must not exceed a total cost of $3,200, including all
transportation, accommodation, meals, and activities. {Detail Omit} Thought 1:
To begin planning your first day, I will collect information on a flight from
Ontario to Honolulu for March 4th, 2022. After that, I will look for dining
options, attractions to visit, and accommodation in Honolulu. Let’s start with
the flight information. Action 1: FlightSearch[Ontario, Honolulu, 2022-03-04]
Observation 1: Masked due to limited length. Make sure the data has been
written in Notebook. Successfully recorded in Notebook: Flight Information for
Ontario to Honolulu on March 4, 2022 Thought 2:... {Omit Other Process} Action
5: DailyPlanner[Please outline the plan for the first day of a 3-day trip from
Ontario to Honolulu on March 4th, 2022, including flight, dining, attractions,
and accommodation.] Observation 5: Travel Plan: Day 1: Current City: from
Ontario to Honolulu Transportation: Flight Number: F3584294, from Ontario to
Honolulu, Departure Time: 10:11, Arrival Time: 14:46 Breakfast: - (Assuming
breakfast is eaten before departure or on the flight) Attraction 1: Aloha
Tower, Honolulu (Estimated visit time: 1 hour) Lunch: Bait El Khetyar,
Honolulu (Cuisines: Cafe, Seafood, Average Cost: 28) Attraction 2: Iolani
Palace, Honolulu (Estimated visit time: 1.5 hours) Attraction 3: Honolulu
Museum of Art (HoMA), Honolulu (Estimated visit time: 2 hours) Dinner: Crystal
Restaurant, Honolulu (Cuisines: Tea, Pizza, Bakery, Fast Food, Chinese,
Average Cost: 57) Accommodation: Park, Subway & All Conveniences; Room type:
Entire home/apt; Price: 754; City: Honolulu Outline: The First Day: from
Ontario to Honolulu. Exploring Honolulu. The Second Day: Exploring Honolulu.
The Third Day: from Honolulu to Ontario. Remember that the total daily
expenses of your trip (the sum of expenses for each person) do not exceed
1066. You should gather the necessary information to plan your trip for the
Second day. Each time you use the DailyPlanner tool, the planner can only
access information queried during the previous 2 days. However, if the number
of queries in the past 2 days is less than 5, then it will return the last 5
queried pieces of information. Thought 6: ... {Omit Future Steps} Figure 9: An
example of the content of strategy block

####  A.7.2 Knowledge Block Pop out Content:

1: Flight Number: F3502691; Price: 240; DepTime: 18:48; ArrTime: 20:51;
OriginCityName: Buffalo; DestCityName: Atlanta 2: Flight Number: F3514187;
Price: 322; DepTime: 06:51; ArrTime: 08:40; OriginCityName: Buffalo;
DestCityName: Atlanta 3: Flight Number: F3555201; Price: 265; DepTime: 12:44;
ArrTime: 14:33; OriginCityName: Buffalo; DestCityName: Atlanta ...
\------------------------------------- 1: Attraction Name: Atlanta Botanical
Garden; City: Atlanta 2: Attraction Name: World of Coca-Cola; City: Atlanta 3:
Attraction Name: Georgia Aquarium; City: Atlanta ...
\-------------------------------------- 1: Restaurant: Adda; City: Atlanta;
Cuisines: Tea, Mexican, Bakery; Average Cost: 22; Rating: 3.8 2: Restaurant:
Baba Au Rhum; City: Atlanta; Cuisines: Desserts, Pizza, Mexican, BBQ, Fast
Food; Average Cost: 27; Rating: 4.5 3: Restaurant: Barkat; City: Atlanta;
Cuisines: Bakery, Indian, Mediterranean, Desserts; Average Cost: 78; Rating:
3.4 ... \-------------------------------------- 1: Accommodation: Fantastic
Room in Bushwick; Room type: Private room; Price: 1069.0; Minimum number of
nights stay: 2.0; review rate number: 3.0; House rules: No children under 10;
One room can accommodate how many people: 2; City: Atlanta 2: Accommodation:
Sunny, Friendly, Brooklyn Apartment; Room type: Private room; Price: 874.0;
Minimum number of nights stay: 1.0; review rate number: 4.0; House rules: No
pets; One room can accommodate how many people: 2; City: Atlanta 3:
Accommodation: 1bd in a sunny 2 bd Ft. Greene Apt; Room type: Private room;
Price: 1056.0; Minimum number of nights stay: 1.0; review rate number: 4.0;
House rules: No visitors & No pets; One room can accommodate how many people:
1; City: Atlanta ... Figure 10: An example of format information in Knowledge
Block.

###  A.8 Prompt

####  A.8.1 PathFinder Agent Prompt:

Figure 11: PathFinder Agent Prompt

####  A.8.2 Thought Agent Prompt:

Figure 12: Thought Agent Prompt

####  A.8.3 Tool Agent Prompt:

Figure 13: Tool Agent Prompt

####  A.8.4 Plan Agent Prompt:

Figure 14: Plan Agent Prompt


# Study of bound and resonant states of NS molecule in the R-matrix approach

F. Iacob1<EMAIL_ADDRESS>Thomas Meltzer2 J. Zs Mezei3,4
<EMAIL_ADDRESS>I. F. Schneider4,5<EMAIL_ADDRESS>J.
Tennyson2,3,4,6<EMAIL_ADDRESS>1Physics Faculty, West University of
Timisoara, 300223,Timisoara, Romania  2Department of Physics and Astronomy,
University College London, WC1E 6BT London, UK  3HUN-REN Institute for Nuclear
Research (ATOMKI), H-4001 Debrecen, Hungary  4LOMC-UMR6294, CNRS, Université
Le Havre Normandie, 76600 Le Havre, France  5LAC-UMR9188, CNRS Université
Paris-Saclay, F-91405 Orsay, France  6HAS-Distinguished Guest Scientists
Fellowship Programme-2022

###### Abstract

The bound and resonance states along with corresponding autoionization widths
for nitrogen sulphide (NS) molecule are determined using electron NS+ cation
scattering calculations. The calculations are performed for
${}^{2}\Sigma^{+}$, ${}^{2}\Pi$ and ${}^{2}\Delta$ total symmetries using the
ab initio R-matrix method for both bound and continuum states. Calculations
are performed on a grid of 106 points for internuclear separations between
1.32 and 3 Å. The resonance states yield dissociative potential curves which,
when considered together with their widths, provide input for models of
different electron-cation collision processes including dissociative
recombination, and rotational and vibrational excitation. Curves and couplings
which will lead directly to dissociative recombination are identified.

###### pacs:

33.80. -b, 42.50. Hz

††preprint: APS/123-QED

##  I Introduction

Sulphur is the tenth and nitrogen is the fifth most abundant element in the
Universe, thus their chemistry is of key importance for astronomical
environments. It is also known to play a significant role in planetary
geochemistry and polymer chemistry. NS emissions are accessible to
astronomical observation and NS was among the first diatomic molecules to be
observed in space Somerville . The start of nitrogen sulphide (or more
precisely mononitrogen monosulphide) chemistry, dates from the discovery of a
compound named as “thiazyl” radical $(\mbox{S}=\mbox{N}\cdot)$ by Demarçay
Demarcay . It was identified inside a polymeric structure [SN]x known as
polythiazyl. Later it was proved that this non-metallic compound has
superconducting properties at low temperatures and the study of this molecule
began to command considerable attention Preuss .

The situation was not the same for NS+ cation. Although it is now known to be
ubiquitous in the interstellar medium (ISM), it was only recently detected
NS+disc . A year later, the first detection of NS+ in a photodissociation
region of the Horsehead Nebula was reported Riviere19 .

A challenge for astrochemistry is to understand the mechanisms and rates of
formation and destruction of both neutral and cationic molecules. Electron
collisions with cations are a common mechanism that leads to their
dissociation. However, despite its astrochemical importance, no estimate of
the dissociative recombination (DR) rate of NS+ is available NS+disc .
Modelling this mechanism requires the calculation of the doubly excited states
of the compound neutral system.

To our knowledge, there are no previous studies of e + NS+ scattering and
relatively few theoretical studies on the NS+ target. These studies mostly
concern NS electronic ground state. O’Hare OHare70  performed a self-
consistent field (SCF) calculation at the equilibrium geometry, Re=1.4957 Å,
and estimated a dissociation energy of $D_{0}$=4.8 eV; other studies of
equilibrium geometry of the X ${}^{1}\Sigma^{+}$ state of NS+ can be found in
Kapf  and reference within. The low-lying valence states of NS+ were explored
by Karna et al. Karna86  using configuration interaction (CI) calculations.
Most important for this work is the more recent study of NS+ metastable states
by Yaghlane and Hochlaf Ben09 .

The purpose of this article is to provide reliable data which can be used as
input to models capable of giving good estimates of the DR rate as well as
rates for other important processes such as electron-impact vibrational
excitation. Our study of electron scattering from the NS+ cation uses the
R-matrix method to give ab initio estimates of the bound and continuum states
of NS. We identify Feshbach resonances that can provide a route to DR by using
a model comprising the ionic core in its ground $X\,^{1}\Sigma^{+}$ state and
three lowest excited states, a$\,{}^{3}\Sigma^{+}$, b$\,{}^{3}\Pi$ and
c$\,{}^{3}\Delta$. Besides potential energy curves (PEC) of the resonances, we
provide their autoionization widths of these doubly excited states; we also
provide PECs for bound, singly excited Rydberg states of NS. These Rydberg
states are significantly more excited than those computed in the previous
study of electronically excited states of NS Lie85 , where only
${}^{2}\Sigma^{+}$ symmetry states were considered. These highly excited
Rydberg states of neutral NS provide an important component of the DR
mechanism for low-energy electron collisions with cations such as those
encountered in the ISM.

The article starts with a section detailing our calculations which considers
the method, the target model, and how our scattering calculations are
performed. Section three presents and discusses our results which comprise
bound states, resonances, widths and effective principal quantum numbers.
Finally, we summarise the main results of this article in the conclusion with
pointers towards future work.

##  II Calculations

###  II.1 Method

The R-matrix method is widely used for studies of scattering problems Burke .
It was originally formulated for nuclear physics and given an ab initio
variational formulation for electron collision problems in atomic and
molecular physics. A detailed discussion of its application to electron-
molecule problems can be found in the extensive review tenn .

Our calculations use the UKRmol+ code UKRmol+ , a new implementation of the
time-independent UK R-matrix electron–molecule scattering code, which uses the
electronic structure code MOLPRO molpro  to provide target wave functions.

The R-matrix method is based on the division of space into an inner and an
outer region. The inner region is a sphere, here with radius of 10 a0, centred
on the centre-of-mass of the NS+ target. In the inner region we perform two
separate calculations: a target one which considers the $N$ interacting target
electrons, and an $N+1$ electron calculation in which the scattering electron
interacts with all target electrons. Then these two calculations are used in
turn to provide R-matrices on the boundary which are then used in the outer
region to calculate scattering properties.

In the inner region the $(N+1)$-electron system of the target and the
colliding electron are described by the wave function:

$\displaystyle\Psi_{k}($ $\displaystyle x_{1}$
$\displaystyle,\cdots,x_{N+1})=\mathcal{A}\sum_{i,j}a_{i,j,k}\,\Phi_{i}^{N}(x_{1},\cdots,x_{N})\times$
(1) $\displaystyle
u_{i,j}(N+1)+\sum_{i}b_{i,k}\,\chi_{i}^{N+1}(x_{1},\cdots,x_{N+1}).$

Here $\Phi_{i}^{N}$ are the wave functions describing the $i^{\rm th}$ target
state. The $u_{i,j}$ represent the extra continuum orbitals. $\Phi_{i}^{N}$
and $\chi_{i}$ are constructed to vanish on the R-matrix boundary. Hence, the
$\chi_{i}$, which represent extra configurations obtained by placing the
scattering electron in a target orbital, are known as $L^{2}$ configurations.
The wave function of the $N+1$ electron system has to obey the Pauli
principle, which is achieved by the action of $\mathcal{A}$ which represents
an anti-symmetrization operator. The coefficients $a_{i,j,k}$ and $b_{i,k}$
are variational coefficients of expansions morgan . To construct the R-matrix
on the boundary we used all the solutions of the inner region problem.

Within the framework of the R-matrix method, there are a variety of different
scattering models and procedures that can be used tenn . Close-coupling (CC)
expansions are used here which involve including several target states in Eq.
(1). The use of the CC method is essential for describing electronic
excitation and is also best for studying Feshbach resonances, which is the
major goal of this paper.

###  II.2 NS+ target

A good description of the target is required when calculating the resonant
scattering states of a molecule. A state averaged multi-configuration self-
consistent field (MCSCF) model was used to generate the first four states of
NS+. These were computed with MOLPRO using a Gaussian-type orbital (GTO) cc-
pVQZ basis set. Several speed and accuracy tests were performed with different
bases to decide the optimal one. The ground state of NS+ is $X$
${}^{1}\Sigma^{+}$ and the equilibrium bond length was determined at $1.44\pm
0.01$ Å.

Figure 1: Potential energy curves for the ground ($X^{1}\Sigma^{+}$) and three
lowest excited states (a$\,{}^{3}\Sigma^{+}$, b$\,{}^{3}\Pi$,
c$\,{}^{3}\Delta$) of NS+ target. Comparison is given with the work of Ben
Yaghlane and Hochlaf Ben09

The low-lying excited states are all triplets: a$\,{}^{3}\Sigma^{+}$,
b$\,{}^{3}\Pi^{+}$, and c$\,{}^{3}\Delta^{+}$. Whilst NS+ belongs to the C∞v
point group, both MOLPRO and UKRmol+ only allow the calculation to be
performed in C2v symmetry. In this case the molecular orbitals are labeled
according to their symmetry properties as belonging to one of the four
irreducible representations (A1, A2, B1, and B2). In the case of a linear
molecule in the $C_{2v}$ point group, results obtained in the B1 and B2
representations are degenerate. The 22 electrons of NS+ are organized in
(10,4,4,1) orbitals of which: (4,1,1,0) are frozen, (4,2,2,0) are used for the
complete active space (CAS) and (2,1,1,1) are used as virtual orbitals in the
scattering calculation. We note that the fewer frozen orbitals we consider
and/or the larger the CAS, the higher accuracy we get but, on the other hand,
this high accuracy requires more computer time.

The NS+ target potential energy curves we obtained are plotted in figure 1.
Our calculation is displayed using solid lines and compared with the
calculations by Ben Yaghlane and Hochlaf Ben09 , displayed using dashed lines.
It can be seen that the agreement with the ground and first excited potential
energy curve of the cation is very good. The energy gap between the ground
state and the first excited state, which is important for determining the
positions of the resonance states, is the same in both studies. Slight
differences appear for the two higher excited states, b$\,{}^{3}\Pi$,
c$\,{}^{3}\Delta$, due to the use of a larger basis set (aug-cc-pV5Z basis
set) in Ben09 .

###  II.3 Scattering calculations

For the scattering calculations, we used the NS+ MCSCF molecular orbitals and
target CAS presented in section II.2, and the same cc-pVQZ basis set. These
were supplemented by continuum orbitals $u_{i,j}$ in order to represent the
scattering electron. A truncated partial wave expansion around the centre-of-
mass with $l\leq 4$ generated as a GTO expansion Faure  for an R-matrix radius
of 10 a0. Use of an orthogonalisation deletion threshold of $2\times 10^{-7}$
resulted in the removal of (10,6,6,4) orbitals from the continuum. The 4 (6 in
C2v symmetry) target states shown in Figure 1 were used in the CC model.

Scattering calculations were performed as a function of bond length, $R$, on a
grid from 1.32 to 2.30 Å with the step 0.01 Å and from 2.30 to 3.0 Å with a
step 0.05 Å giving a total of 106 points. Although the C2v point group was
used for calculation, results were extracted for ${}^{2}\Sigma^{+}$,
${}^{2}\Pi$, and ${}^{2}\Delta$ symmetry of the neutral system.

In the outer region the R-matrix was propagated Morgan84  to 150 a0. Module
RESON Tenn84  was used to automatically detect resonances and fit them to a
Breit-Wigner profile to determine their position ($E_{r}$) and width
($\Gamma$) using grids of eigenphase sums computed for each resonance.

Figure 2:  NS ${}^{2}\Sigma^{+}$ bound states: Left panel PECs. Right panel
effective principal quantum numbers. Bound states belong to series that
converge to $X\,^{1}\Sigma^{+}$ symmetry of ion. The partial wave
characterizing the Rydberg $\sigma$-wave electrons is indicated using colours:
blue $s$, red $p$, green $d$.

##  III Results and discussion

In this section, we present results for the bound states and for the
autoionizing resonance positions and widths of the neutral NS molecule.

###  III.1 Bound states

Bound electronic states of NS were found using the method of Sarpal et al.
sarpal : R-matrices were propagated to 30 a0 and wave functions computed using
an improved Runge-Kutta-Nystrom algorithm zhang11 . Effective principal
quantum numbers ($\nu$) as a function of internuclear separation were
calculate from the Rydberg states assuming as threshold of the ground state of
the ion.

Bound states were found for ${}^{2}\Sigma^{+}$, ${}^{2}\Pi$ and ${}^{2}\Delta$
total symmetries. These states are a mixture of Rydberg states which follow
the shape of the NS+ X ${}^{1}\Sigma^{+}$ state and valence states which do
not. Both sets of states are depicted in our figures. In this section we
concentrate characterizing the Rydberg series but below we also consider those
bound state curves which link with the key, dissociative resonances at large
$R$.

The curves and the effective principal quantum numbers for states we assign as
Rydberg-like are shown in figures 2, 3 and 4. In each figure, the uppermost
dashed black curve gives the $X\,^{1}\Sigma^{+}$ ground state of NS+.

Figure 3:  NS ${}^{2}\Pi^{+}$, bound states: Left panel PECs. Right panel
effective principal quantum numbers. Bound states belong to series that
converge to $X\,^{1}\Sigma^{+}$ symmetry of ion. The partial wave
characterizing the Rydberg $\pi$-wave electron is indicated using colours:
green $p$-wave, blue $d$-wave.  Figure 4: NS ${}^{2}\Delta^{+}$, bound states:
Left panel PECs. Right panel effective principal quantum numbers. Bound states
belong to series that converge to $X\,^{1}\Sigma^{+}$ symmetry of ion. The
partial wave characterizing the $\delta-wave$ electron of the states is
indicated using colours: red $d$-wave, blue $f$-wave.

If one allocates a dominant partial wave as characterizing the Rydberg
electron of each states, one gets different Rydberg series for the various
total symmetries. Note our calculations neglect Rydberg series arising partial
waves with $\ell>4$; such states are expected to have quantum defects close to
zero. Below we consider the major (i.e. low $\ell$) partial waves for each
symmetry.

For ${}^{2}\Sigma^{+}$ symmetry we consider $s$, $p$ and $d$ Rydberg
electrons; for ${}^{2}\Pi$ symmetry we consider $p$ and $d$ electrons and for
${}^{2}\Delta$ symmetry we consider $d$ and $f$ electrons. In general the
effective principal quantum numbers of the states show smooth and generally
slow variation with $R$. However, in places the structure of the Rydberg
series are complicated by the presence of so-called intruder states which
arise from Rydberg series associated with excited states of the ion. Crossings
by these intruder states appear as local discontinuities in the $\nu$ as a
function of $R$.

###  III.2 Resonance curves

The R-matrix calculation is made in the fixed nuclei approximation and hence
the PECs are adiabatic. Resonances were characterised up to approximately 4 eV
above the equilibrium energy of the X$\,{}^{1}\Sigma^{+}$ state of ion, which
restricts the internuclear distance range to $R\approx 1.2-1.8$ Å.

Figures 5, 6, and 7 shows the resonances that are found with emphasis on the
PECs that can lead to dissociation as DR occurs along these repulsive PECs.
The right panel of each figure shows the real part of the $\nu$ for each
resonance as a function of $R$; being derived assuming that the resonance is
associated with the first, a ${}^{3}\Sigma^{+}$, excited state. In each
figure, the curves can be matched by their colours. Again we concentrate on
states of low $\ell$ as higher $\ell$ states have quantum defects close to
zero and, more importantly, very weak electronic couplings meaning that in
general they do not play a significant role in DR.

Figure 5 shows that the lowest two curves show almost constant effective
principal quantum number for $R>1.4$ Å. The inflection point at $R=1.36$ Å is
due to the NS+ c$\,{}^{3}\Delta$ curve, which for $R<1.36$ Å crosses below the
a$\,{}^{3}\Sigma$ state. The $\nu$ for the highest, black curve in Fig. 5 had
to be given special consideration. It showed strong variation with $R$ when
taken relative to the a$\,{}^{3}\Sigma^{+}$ state. Assuming this resonance to
be an intruder we plotted its effective principal quantum number relative to
both the b$\,{}^{3}\Phi$ and c$\,{}^{3}\Delta$ states but again found strong
variations with $R$. Finally, we took $\nu$ relative to the
d$\,{}^{5}\Sigma^{+}$ curve, as calculated by Ben Yaghlane and Hochlaf Ben09 ;
this curve is the extra one shown on this figure. This showed much flatter
variation although there is a pronounced slope which is possibly caused by the
fact that this is not a like-for-like comparison.

The ${}^{2}\Pi$ Rydberg states curves presented in figure 6 mostly showing
little variation in $\nu$ as for $R>1.4$ Å. However, the ${}^{2}\Pi$ symmetry
shows avoided crossings between the green with turquoise curves, and the
turquoise and blue curves. The approximately constant behaviour of $\nu$ with
$R$ is maintained with the corresponding change of colours. In the figure 7
the effective principal number shows an inflexion point at $R<1.36$ Å due to
the crossing of NS+ c$\,{}^{3}\Delta$ curve below the a$\,{}^{3}\Sigma$ one.
The effective principal number corresponding to the NS∗∗ 3 ${}^{2}\Delta$
state shows a strong variation with $R$ and a shift at $R=1.6$ Å. A comparison
with a higher excited curve might show an effective quantum number with a
flatter dependence on $R$.

Figure 5: NS ${}^{2}\Sigma^{+}$ symmetry relative to resonances curves (left
panel) and effective principal quantum numbers (right panel) for some specific
curves of special interest for DR. Figure 6: NS ${}^{2}\Pi$ symmetry
resonances curves (left panel) and effective principal quantum numbers (right
panel) for some specific curves of special interest for DR. Figure 7: NS
${}^{2}\Delta$ symmetry resonances curves (left panel) and effective principal
quantum numbers (right panel) for some specific curves of special interest for
DR.

Figure 8 shows the PECs with ${}^{2}\Sigma^{+}$ symmetry of neutral NS in an
adiabatic representation. However, following the colours one can see the
diabatic states. The curves situated above the PEC of the ion ground state are
the resonances and those below it represent bound states. Dissociative PECs of
importance for DR comprise of a resonant state in the continuum which cross
the ion ground state to become bound states where they then, in the diabatic
picture, pass through the Rydberg series associated with the ion ground state.
The PECs are smooth except in a vicinity of avoided crossings. Figures 9 and
10 show similar curves for both ${}^{2}\Pi$ and ${}^{2}\Delta$ symmetry
respectively. These curves are the ones which form the input for a model of
DR.

Figure 8: NS resonance and the bound states extension for the
${}^{2}\Sigma^{+}$ molecular symmetry as a function of internuclear distance.
Figure 9: NS resonance and the bound states extension for the ${}^{2}\Pi$
molecular symmetry as a function of internuclear distance. Figure 10: NS
resonance and the bound states extension for the ${}^{2}\Delta$ molecular
symmetry as a function of internuclear distance.

###  III.3 Widths and quantum defects

Complex quantum defects $\mu=\alpha+i\beta$ were obtained from the fitted
position and width using the formulae:

$E_{r}=E_{t}-\frac{Ry}{\nu^{2}},\;\;\Gamma=\frac{2\beta\,Ry}{\nu^{3}},$ (2)

where the effective principal quantum number $\nu=n-\alpha$, Et is the energy
of the threshold to which the Rydberg series converges and $Ry$ is the Rydberg
constant.

The widths are presented in figure 11 as functions of $R$; the symmetry is
marked on each graph. One may see how the behaviour of widths change when
avoided crossings occur. In general, we expect the width of a resonance to
decrease as the number of open channels decreases which occurs when a
resonances passes through a state of the ion. Note that the widths should
vanish once the resonance states cross the ion ground state and become bound.

Figure 11: Resonance widths as a function of internuclear distance in
agreement with the resonances from figures 8, 9, and 10. The corresponding
symmetry of the resonant states is indicated in the ﬁgure.

##  IV Conclusions

Resonance positions and auto-ionization widths were calculated for e-NS+
system depending on the internuclear separation. The Rydberg series that
converge to ion ground state were computed together with the corresponding
effective principal quantum numbers. The use of a dense grid produces numerous
resonances and bound states in great details, which facilitate the
identification of dissociative states and singly excited Rydberg manifolds.
Figures 8, 9, and 10 summarize our results and provide the curves and
electronic couplings which will provide the input for future nuclear dynamics
(dissociative recombination and ro-vibrational transitions) studies. To our
knowledge, this is the first time when relevant molecular data sets were
calculated to study electron induced reactive elementary processes in NS+. The
data produced in this work will be used for calculating dissociative
recombination and ro-vibrational transition cross sections and rate
coefficients relevant to the astrochemistry of sulphur and nitrogen, using
stepwise Multichannel Quantum Defect Theory, that has proven its power for
many diatomic and polyatomic molecular systems Little14 ; Niyonzima17 ;
Niyonzima18 ; Pop21 ; Epee22 ; slava18 .

## Acknowledgements

We are grateful to Kalyan Chakrabarti for helpful discussions. This article is
based upon work performed as part of COST Action: CM1401 - Our Astro-Chemical
History, CA17126 - TUMIEE, CA18212 - MD-GAS. The authors acknowledge support
from La Région Normandie, FEDER, and LabEx EMC3 via the projects PTOLEMEE,
Bioengine COMUE Normandie Université, the Institute for Energy, Propulsion and
Environment (FR-IEPE), and to Agence Nationale de la Recherche (ANR) via the
project MONA. This work was supported by the Programme National “Physique et
Chimie du Milieu Interstellaire” (PCMI) of CNRS/INSU with INC/INP co-funded by
CEA and CNES. JZsM thanks the financial support of the National Research,
Development and Innovation Fund of Hungary, under the K 18 and FK 19 funding
schemes with project numbers K 128621 and FK 132989. JZsM is greateful for the
hospitality of West University Timişoara within the Visiting@WUT grant
programme. JT and JZsM are indebted to the Distinguished Guest Scientists
Fellowship Programme-2022 of the Hungarian Academy of Sciences. TM was funded
by EPSRC (Grant No. EP/M507970/1).

## Data availability

Upon a reasonable request, the data supporting this article will be provided
by the corresponding author.

## References

  * (1) Somerville W B 1977 Rep. Prog. Phys. 40, 483.
  * (2) Demarçay E 1880 Compt. Rend. 91, 854.
  * (3) Preuss K E 2007 Dalton Trans. 23, 2357.
  * (4) Cernicharo J, et al. 2018 ApJL 853, L22.
  * (5) Rivière-Marichalar P, et al. 2019 A&A 628, A16.
  * (6) O’Hare P A G 1970 J. Chem. Phys. 52, 2992.
  * (7) Karpfen A., Schuster P., Petkov J., Lischka H., 1978, J. Chem. Phys., 68, 3884
  * (8) Karna S P and Grein F 1986 Chem. Phys. 109, 35.
  * (9) Ben Yaghlane S and Hochlaf M 2009 J. Phys. B: At. Mol. Opt. Phys. 42, 015101.
  * (10) Lie G C, Peyerimoff S D and Buenker R J 1985 J. Chem. Phys. 82, 2672 and references therein.
  * (11) Burke P G 2011 R-Matrix Theory of Atomic Collisions Springer: Heidelberg, Berlin.
  * (12) Tennyson J 2010 Phys. Rep. 491, 29.
  * (13) Masin Z, Benda J, Gorfinkiel J D, Harvey A G and Tennyson J 2020 Comput. Phys. Commun. 249, 107092.
  * (14) Werner H-J, Knowles P J, Knizia G, Manby F R and Schütz M 2012 WIREs Comput Mol Sci 2, 242.
  * (15) Morgan L A, Tennyson J and Gillan C J 1998 Computer Phys. Commun. 114, 120.
  * (16) Faure A, Gorfinkiel J D, Morgan L A and Tennyson J. 2002 Gtobas: Comput. Phys. Commun. 144, 224.
  * (17) Morgan L A 1984 Comput. Phys. Commun. 31, 419.
  * (18) Tennyson J and Noble C J 1984 Comput. Phys. Commun. 33 421\.
  * (19) Sarpal B K, Branchett S E, Tennyson J and Morgan L A 1991 J. Phys. B: At. Mol. Opt. Phys. 24, 3685.
  * (20) Zhang R, Baluja K L, Franz J and Tennyson J 2011 J. Phys. B: At. Mol. Opt. Phys. 44, 035203.
  * (21) Little D A, Chakrabarti K, Mezei J Zs, Schneider I F and Tennyson J 2014 Phys. Rev. A bf 90, 052705.
  * (22) Niyonzima S, Ilie S, Pop N, Mezei J Zs, et al. 2017 At. Data Nucl. Data Tables bf 115-116, 287.
  * (23) Niyonzima S, Pop N, Iacob F, Larson Å, et al. 2018 Plasma Sources Sci. Technol. bf 27, 025015.
  * (24) Pop N, Iacob F, Niyonzima S, Abdoulanziz A, Laporta V, Reiter D, Schneider I F and Mezei J Zs 2021 At. Data Nucl. Data Tables 139, 101414.
  * (25) Epée Epée M D, Motapon O, Pop N, Iacob F, Roueff E, Schneider I F and Mezei J Zs 2022 MNRAS 512, 424.
  * (26) Kokoouline V, Ayouz M, Mezei J Zs, Hassouni K and Schneider I F 2018 Plasma Sources Sci. Technol. 27, 115007.