# JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog
Policy Learning

Wai-Chung Kwan1, Huimin Wang2, Hongru Wang 1, Zezhong Wang 1, Xian Wu 2,
Yefeng Zheng 2, Kam-Fai Wong 1

###### Abstract

Dialogue policy learning (DPL) is a crucial component of dialogue modelling.
Its primary role is to determine the appropriate abstract response, commonly
referred to as the “dialogue action”. Traditional DPL methodologies have
treated this as a sequential decision problem, using pre-defined action
candidates extracted from a corpus. However, these incomplete candidates can
significantly limit the diversity of responses and pose challenges when
dealing with edge cases, which are scenarios that occur only at extreme
operating parameters. To address these limitations, we introduce a novel
framework, JoTR. This framework is unique as it leverages a text-to-text
Transformer-based model to generate flexible dialogue actions. Unlike
traditional methods, JoTR formulates a word-level policy that allows for a
more dynamic and adaptable dialogue action generation, without the need for
any action templates. This setting enhances the diversity of responses and
improves the system’s ability to handle edge cases effectively. In addition,
JoTR employs reinforcement learning with a reward-shaping mechanism to
efficiently finetune the word-level dialogue policy, which allows the model to
learn from its interactions, improving its performance over time. We conducted
an extensive evaluation of JoTR to assess its effectiveness. Our extensive
evaluation shows that JoTR achieves state-of-the-art performance on two
benchmark dialogue modelling tasks, as assessed by both user simulators and
human evaluators 111Our code, models and other related resources are publicly
available at https://github.com/KwanWaiChung/JoTR.

## Introduction

Figure 1: This illustrates dialogue policy learning in pipeline dialogue
system, where NLU/NLG stands for Natural Language Understanding/Generation,
and DST refers to Dialogue State Tracking.

Dialogue policy learning (DPL) is a fundamental component of pipeline-based
dialogue systems. As illustrated in Figure 1, its primary role is to select
the appropriate dialogue action to manage the conversation flow, essentially
acting as the decision-making core of the system. Previous research in this
field has predominantly focused on treating DPL as a sequential decision
problem. The optimization of the policy is typically achieved using
reinforcement learning (RL), a machine learning approach that learns from its
interactions with the environment, based on predefined dialogue action
candidates (Lipton et al. 2016; Li et al. 2017; Peng et al. 2018; Takanobu,
Zhu, and Huang 2019; Wang, Peng, and Wong 2020a; Li et al. 2020; Kwan et al.
2023). A dialogue action usually comprises one or more combinations of domain
name, intent type, and slot name, collectively referred to as an “atomic
action” (Li, Kiseleva, and de Rijke 2020). Traditional DPL approaches filter
action candidates from a corpus, placing emphasis on more frequent actions to
enhance the efficiency of the RL agent. However, this approach has its
limitations. It may underperform in less common situations and can restrict
response diversity and naturalness due to the rigid templates of allowable
actions (Wang and Wong 2021). This limitation stems from the fact that policy
optimization is based on incomplete action candidates, which include only a
subset of possible atomic actions or their combinations. To address this
issue, Li, Kiseleva, and de Rijke (2020) proposed using a GRU-based decoder to
predict atomic actions sequentially in a single turn. This approach allows the
DPL model to learn the action structure from data, resulting in more flexible
and powerful system responses. However, this method has a significant
drawback: the dialogue state and atomic action space grow exponentially with
the number of domains, intents, or slots (Lee et al. 2019), making it
challenging to train the model effectively. Alternatively, Wang and Wong
(2021) proposed a multi-agent reinforcement learning framework where atomic
actions were represented as the cross product of three spaces, with each part
assigned to a different agent. While this approach showed promising results,
it had the significant drawback of allowing only one atomic action per turn.
This limitation necessitates more turns to achieve user goals and can lead to
unnatural utterances. A recent work that used a transformer to generate
actions is closely related to our research (Geishauser et al. 2022). However,
their work primarily focused on continual learning and did not address the
issue of action efficiency, which is a critical aspect of DPL. Our research
aims to build upon these previous studies, addressing their limitations and
enhancing the efficiency and effectiveness of DPL.

In this paper, we introduce a novel approach, the Joint Transformer
Reinforcement Learning Framework (JoTR), designed to address the limitations
inherent in dialogue action candidates. The primary innovation of JoTR lies in
its ability to generate actions with a word-level policy directly, a
significant departure from the traditional reliance on fixed, human-defined
templates. This is achieved by leveraging the capabilities of a text-to-text
transformer, a model that has shown remarkable success in various natural
language processing tasks. Simultaneously, JoTR employs reinforcement learning
and reward shaping, two powerful techniques in machine learning, to optimize
the word-level policy. This results in more efficient responses that require
fewer interaction turns, a critical aspect in enhancing the user experience in
dialogue systems. Furthermore, JoTR demonstrates improved learning efficiency
compared to traditional methods, making it a promising approach for dialogue
policy learning (DPL). We evaluate the performance of JoTR on two benchmark
multi-domain dialogue modeling tasks, providing a comprehensive assessment of
its effectiveness. Our research is primarily focused on developing a system
that generates flexible and efficient responses for DPL, a critical aspect of
dialogue systems.

Our work offers several key contributions to the field. Firstly, we formulate
the DPL as a conditional sequence generation problem, a novel approach that
allows for more dynamic action generation. Secondly, we introduce a
transformer reinforcement learning framework, designed to learn the word-level
DPL and optimize it with a reward-shaping mechanism. This approach enhances
the model’s ability to adapt and improve over time. Thirdly, we validate the
effectiveness of our proposed models through multi-domain testing on a
simulator. This rigorous testing ensures that our model can handle a variety
of dialogue scenarios effectively. Lastly, we implement a rule-based simulator
for the benchmark Schema-Guided Dialogue (SGD) dataset (Rastogi et al. 2020),
a widely-used resource in dialogue system research. This implementation
provides a practical tool for further research and development in the field.

## Related Work

##### Dialogue Policy Learning

The mainstream method for constructing a task-oriented dialogue (TOD) system
is the pipeline approach, which divides the system into four interconnected
modules: natural language understanding (to identify user intentions),
dialogue state tracking (to monitor the dialogue status), dialogue policy
learning (to decide the subsequent system action), and natural language
generation (to generate dialogue responses) Kwan et al. (2023). Reinforcement
learning has been the main stream approach to optimize the dialogue policy
(Levin, Pieraccini, and Eckert ; Singh et al. ; Gašić et al. 2010). To address
the challenges of large state-action spaces and low exploration efficiency
when applying reinforcement learning to DPL, hierarchical reinforcement
learning has been used to break down the complex task into several subtasks
and learn separate policies for these subtasks (Budzianowski et al. 2017; Peng
et al. 2017; Kristianto et al. 2018; Tang et al. 2018). Alternatively, Other
researchers have adopted reward learning and reward shaping to provide denser
rewards and foster faster learning (Su et al. 2015, 2016; Wang, Peng, and Wong
2020b). Our approach aligns with this line of work and incorporates reward
shaping to refine the dialogue policy. Recently, a few studies have started to
apply multi-agent reinforcement learning for dialogue policy learning (Liu and
Lane ; Zhang et al. 2020). Takanobu, Liang, and Huang (2020) propose a joint
learning process for the dialogue system and the user agent, and introduced
the Hybrid Value Network to enhance the reward for each role. Wang and Wong
(2021) further expand on this approach by partitioning the action space into
three subspaces and training a separate agent for each using multi-agent
reinforcement learning. These works frame model DPL as a classification
problem where the policy chooses a suitable dialogue action from a predefined
action list. Contrasting to these methodologies, our approach obviates the
need for a predefined action list, generating the dialogue action instead.

##### Pre-trained Language Model

Recent studies have demonstrated significant advancements in TOD by fine-
tuning pre-trained language models (Budzianowski and Vulić 2019; Hosseini-Asl
et al. 2020; Lee 2021; Su et al. 2022; Zhao et al. 2022). These studies have
effectively unified the learning process of various modules within the
pipeline approach. Peng et al. (2021) introduces SOLOIST, a model that
leverages GPT-2 (Radford et al. 2019), a pre-trained auto-regressive
transformer decoder model. This model generates the belief state, dialogue
action, and response in a sequential manner. SOLOIST is first fine-tuned on a
large amount of dialogue corpus with a multi-task objective, then fine-tuned
on the target dataset. He et al. (2022) further extends this approach by
incorporating unlabelled dialogue data through semi-supervised learning. A
notable drawback of sequential subtask generation is the error accumulation
from earlier modules to the later modules in the pipeline. To mitigate this,
PPTOD first formulates the belief state, then concurrently generates the
dialogue action and response, thus preventing error propagation. These
approaches require a large dialogue corpus to further pre-training and fine-
tuning of the pre-trained language model. In contrast, our work only requires
on small amount of data to warm up the model to obtain the state-of-the-art
results. Additionally, our work utilizes reinforcement learning to fine-tune
the model, which does not require labelled data.

## JOTR

Figure 2: The joint transformer and reinforcement learning framework
illustration consists of: 1) (Left Part) Text Encoding - The encoder processes
user act, system act, belief state, and database query results to form the
state. 2) (Right Part) Model Optimization - The state directs action
generation, with the Action Interpreter generating structured dialogue
actions. The transformer-based policy model undergoes interactive optimization
through reinforcement learning from scratch.

We utilize a transformer-based model to directly generate dialogue actions,
distinguishing it from conventional approaches that rely on selecting dialogue
actions from a predefined set of atomic dialogue actions and their
combinations. The architecture of our model is illustrated in Figure 2. In
this section, We first provide a formal definition for DPL, then present the
overview of our approach.

### Problem Definition

The goal of DPL is to learn a policy that interacts with a user through the
generation of dialogue actions, given the belief state and the database
results, to satisfy the user goal $G=(C,R)$, where $C$ denotes the user
constraints (e.g. an air ticket to Seattle) and $R$ represents the information
required by the user (e.g. the price of the air ticket). As mentioned in the
previous section, the belief state keep tracks of the user’s constraints
throughout the dialogue. The belief state is defined as a list of domain,
slot, value triplets (e.g. [(flight, destination, Seattle), (flight, day,
tomorrow)]). The dialogue action is represented as a list of domain, intent,
slot, value quadruples (e.g. [(flight, request, time, ?)]). An external
database provides relevant entries to the dialogue policy based on the belief
state. Figure 1 provides an illustrative example of such a process.

### Dialogue State Text Encoding

The encoder generates state embeddings
$e_{u},e_{s},e_{b},e_{d}\in\mathbb{R}^{d}$ by encoding the flattened textual
representations of four elements: user action, system action, belief state,
and the database result. These linearized textual representations are referred
to as the dialogue state text. The user action is a sequence of tokens derived
from atomic dialogue action triplets, each comprising the domain $D$, intent
$I$, and slot $S$. This sequence is represented as
$D_{1},I_{1},S_{1},\ldots,D_{N^{u}},I_{N^{u}},S_{N^{u}}$, where $N^{u}$
denotes the number of atomic user dialogue actions. For instance, ”hotel
inform price hotel request name” is a valid example. The system action,
similar to user action, is represented as
$D_{1},I_{1},S_{1},\ldots,D_{N^{s}},I_{N^{s}},S_{N^{s}}$, with $N^{s}$
indicating the number of atomic system dialogue actions. The belief state is a
sequence formed by concatenating the belief state triplets, where each triplet
is composed of the domain $D$, slot $S$, and value $V$. This sequence can be
represented as $D_{1},S_{1},V_{1},\dots,D_{N^{b}},S_{N^{b}},V_{N^{b}}$. A
typical example could be ”hotel price expensive hotel area north”. The
database result is represented as $D_{1},Q_{1},\dots,D_{N^{d}},Q_{N^{d}}$,
with $N^{d}$ denoting for the number of queried domains and $Q$ denoting the
number of matched entities in the database. An illustrative example would be
“hotel 4 restaurant 2”.

To obtain the state embeddings $e_{u},e_{s},e_{b},e_{d}$, the [CLS] token, a
common sentence representation placeholder, is prefixed to each dialogue state
text (Devlin et al. 2018). The output representation of the [CLS] token of
each dialogue state text is used as the state embedding. Initial experiments
revealed inferior performance if the model is only fed with the state
embeddings, likely due to the model’s confusion about the varying types of
state information being encoded. Therefore, a context embedding was
constructed for each dialogue state text. The context embeddings are added
with the state embeddings individually to produce the state
$s\in\mathbb{R}^{4\times d}$.

### Word-level Dialogue Policy

We have formulated the problem of dialogue policy learning (DPL) as a Markov
Decision Process (MDP) on the word level. In this process, the system agent
observes the current dialogue state $s$, executes an action $a$ (generated by
the policy function), receives a response, a reward $r$, and the updated
dialogue state $s^{\prime}$. This cycle continues until the conversation ends.
The action $a$ is textually represented as a sequence of words
$w_{1:N}=w_{1}\ldots w_{N}$. The policy function can be detailed as a series
of conditional probabilities:

$\displaystyle\pi_{\theta}(a|s)=\prod_{i=1}^{N}\mathcal{P}_{\theta}(w_{i}|w_{1:i-1},s),$
(1)

where $\mathcal{P}$ is approximated with a transformer encoder-decoder network
parameterized by $\theta$, representing the probability of the word $w_{i}$
condition on the preceding words and state. As shown in Figure 2, the
transformer decoder generates the dialogue action text word by word, beginning
with the start signal “[start]”, conditioned on the dialogue state, and
proceeds until it encounters the stop signal “[end]”. Additionally, an action
interpreter decodes the dialogue action text into structured format,
populating slot values from the database, yielding the final dialogue action.
This process involves verifying whether the dialogue action text adheres to
the domain, intent, slot order, discarding any words that violate these
conditions. The policy $\pi$, parameterized by $\theta$, is optimized using
reinforcement learning to minimize the negative expected cumulative future
rewards:

$\displaystyle\mathcal{L}_{\theta}=-\mathbb{E}_{a_{t}\sim\pi_{\theta}(\cdot|s_{t})}\left[\sum_{t=1}^{T}r(s_{t},a_{t})\right],$
(2)

where $s_{t}$ and $a_{t}$ are the state and dialogue action turn $t$, and $T$
is the maximum turn. In practice, the expected gradient for a dialogue session
can be approximated by using a Monte Carlo sample from $\mathcal{P}_{\theta}$.
For each session example, the gradient is approximated as:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\theta}\approx-\sum_{t=1}^{T}r(s_{t},a_{t})\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})=$
(3)
$\displaystyle-\sum_{t=1}^{T}r(s_{t},w^{t}_{1:N^{t}})\sum_{i=1}^{N^{t}}\nabla_{\theta}\log\mathcal{P}_{\theta}(w_{i}^{t}|w_{1:i-1}^{t},s_{t}),$

where $N^{t}$ is the length of the action text at turn $t$.

### JoTR for Efficient Policy Training

We employ Proximal Policy Optimization (PPO) (Schulman et al. 2017) to
optimize the policy. More specifically, we minimize the objective function for
each session example.

$\displaystyle\mathcal{L}_{\theta}$
$\displaystyle=\sum_{t=1}^{T}-\hat{\mathbb{E}}_{t}\left[\min\left(\frac{\pi_{\theta}\left(a_{t}\mid
s_{t}\right)}{\pi_{\theta_{\text{old }}}\left(a_{t}\mid
s_{t}\right)}\hat{A}_{t}^{\phi},\right.\right.$ (4)
$\displaystyle\left.\left.\operatorname{clip}\left(\frac{\pi_{\theta}\left(a_{t}\mid
s_{t}\right)}{\pi_{\theta_{\text{old }}}\left(a_{t}\mid
s_{t}\right)},1-\epsilon,1+\epsilon\right)\hat{A}_{t}^{\phi}\right)\right]$
$\displaystyle=\sum_{t=1}^{T}-\hat{\mathbb{E}}_{t}\left[\min\left(\frac{\prod_{i=1}^{N}\mathcal{P}_{\theta}(w_{i}|w_{1:i-1},s_{t})}{\prod_{i=1}^{N}\mathcal{P}_{\theta_{\text{old}}}(w_{i}|w_{1:i-1},s_{t})}\hat{A}_{t}^{\phi},\right.\right.$
$\displaystyle\left.\left.\operatorname{clip}\left(\frac{\prod_{i=1}^{N}\mathcal{P}_{\theta}(w_{i}|w_{1:i-1},s_{t})}{\prod_{i=1}^{N}\mathcal{P}_{\theta_{\text{old}}}(w_{i}|w_{1:i-1},s_{t})},1-\epsilon,1+\epsilon\right)\hat{A}_{t}^{\phi}\right)\right],$

where $\hat{A}_{t}^{\phi}=r(s_{t},a_{t})+\gamma
V^{\phi}(s_{t+1})-V^{\phi}(s_{t})$ is the advantage estimation and
$V^{\phi}(s_{t})$ is the value function estimated by the critic parameterized
by $\phi$.

To improve the efficiency and quality of the dialog response, we integrate
reward-shaping into the reinforcement learning fine-tuning process. The goal
is to prevent the policy from generating protracted yet predominantly
irrelevant dialogue actions during optimization with PPO. To achieve this, we
propose a reward-shaping function assigning supplementary rewards to guide the
model to learn. Formally, we replace the $r(s_{t},a_{t})$ with
$\hat{r}(s_{t},a_{t},G)$ defined as

$\displaystyle\hat{r}(s_{t},w_{1:N^{t}}^{t},G)=r(s_{t},a_{t})+F(G,w_{1:N^{t}}^{t}),$
(5)

where $G$ denotes the user goal and $F$ represents the shaping reward. We
design $F$ to provide different reward based on the follow: (1) If the system
informs a slot present in the user’s request slot list, it receives an
additional $\lambda$ reward. Conversely, informing other slots receives an
additional -1 reward. (2) If the system requests a slot included in the user’s
inform slot list, it receives an additional $\lambda$ reward. However,
requesting other slots results in an additional -1 reward. $\lambda$ is a
hyperparameter that controls the aggressiveness of the dialogue agent to
inform or request additional slots. We try $\lambda$ with value 3, 4, 5, 6, 7.
We find that the range of 3 to 5 yielded favourable results during validation.
Higher $\lambda$ values encourage the dialogue agent to attempt many actions
in a single turn, where one successful inform or request action offsets the
negative rewards incurred by other irrelevant actions. We pick $\lambda=3$ for
all the experiments.

## Experiments and Results

Experiments are carried out on MultiWOZ 2.0 (Budzianowski et al. 2018),
utilizing a publicly accessible agenda-based user simulator (Zhu et al. 2020),
and on the SGD dataset with our developed rule-based simulator. Furthermore,
we incorporate human evaluations, in which evaluators interact with various
models and assess the success of the dialogue upon its completion. While all
models, except SimpleTOD, are optimized in the dialogue action space,
SimpleTOD takes the utterance dialogue history as input and generates both the
dialogue action and the system utterance.

## Evaluation Metrics

We focus on three evaluation metrics to align with previous work (Wang and
Wong 2021): success rate, average number of turns, and average reward. A
dialogue session is considered successful if it fulfils all the user requests,
and reserves an entity that meets the user’s specifications if necessary. The
average number of turns is calculated by counting the number of interactions
between the two parties, with each full interaction counted as two turns. The
average reward is the total cumulative reward obtained in each dialogue
session. Additionally, since SimpleTOD uses dialogue utterances as input, an
NLG component is needed to convert the user’s dialogue actions into
utterances. However, the performance of SimpleTOD is significantly dependent
on the NLG component. Therefore, we also evaluate it against a testing corpus,
which we believe is a superior method than using a user simulator equipped
with NLU and NLG modules that could potentially introduce noise.

Figure 3: The learning curve of various models on MultiWOZ and SGD, with the
mean and standard deviation illustrated over 5 runs. We didn’t reach JOIE’s
400k frames, as JoTR attains a 0.93 success rate with only 50k frames,
surpassing JOIE, and training 400k frames for its variants is costly.

## Training and Implementation Details

We implement all the models in PyTorch (Paszke et al. 2019) and Transformers
(Wolf et al. 2020). We use a randomly initialized transformer encoder with 1
hidden layer, 1 head, and hidden size of 256 as the dialogue state text
encoder of JoTR. In JoTR${}_{\text{pretrained}}$, we use DistilBert (Sanh et
al. 2019) with pretrained weights from Huggingface as the initial weights for
the dialogue state text encoder. For the encoder-decoder model, we use a
transformer with 1 hidden layer, 1 attention head, and hidden size of 256 for
both the encoder and decoder. The total number of parameters is 5M. All
variants of JoTR and MLP${}_{\text{ppo}}$ are warmed up before reinforcement
learning by first performing supervised learning on a training set to be
consistent with previous work (Wang and Wong 2021). We use the same set of 10K
dialogue turns sampled randomly from the original training set to warm up all
models. A separate, non-overlapping set of 3K dialogue turns is used for
validation in the warm-up phase. The same set of hyperparameters are used on
MultiwWOZ and SGD in both pretraining and PPO fine-tuning. For the warm-up
phase, the models are trained using a batch size of 32 and a learning rate of
$3\times 10^{-4}$. The models are trained for 80 epochs but we include an
early stopping mechanism that halted training when no improvement is observed
in the validation set over five consecutive epochs. In PPO training, we use an
actor learning rate of $5\times 10^{-7}$ and critic learning rate of $1\times
10^{-4}$. The critic is a transformer with identical architecture as the
encoder-decoder model. The models are trained with a total of 50K frames. All
models are trained on a single RTX 3090 GPU, which take about 10 hours to
train. The maximum interaction turn allowed is 40. The main reward provided by
the environment is -1 in every turn, and a reward of 80 or -40 at the end for
successful or failed dialogue respectively.

The prompt for ChatGPT is shown in Table 1. Following previous work (Wang et
al. 2022b), we provide one formatting example in the zero-shot setting. In our
preliminary experiments, we find that the task definition and the output
specification significantly influence ChatGPT’s ability to understand the task
and generate a structured output.

Prompt:
---
You are a dialogue agent to assist me with my queries and provide me with
relevant information from a database. My questions are formatted as tuples of
(domain, intent, slot, slot value) accompanied by the number of matching
results that satisfy my constraint from the database, e.g., ”4 matches”.
Your responses should be formatted as one or several tuples of (domain,
intent, slot) to provide me with the necessary information.
The domain is selected from attraction, hospital, ….
The intent is selected from inform, request, ….
The slot includes addr: the address of the hotel, attraction, restaurant,
hospital, or police station….
Example 1:
USER: [(train, inform, depart, london kings cross)] 3 matches.
ASSISTANT: [(train, inform, id)]
…
Example 2:
USER: [(restaurant, inform, area, centre)]…
Output:
ASSISTANT: [(restaurant, request, day), (restaurant, request, time),
(restaurant, request, people)]
Table 1: A zero-shot ChatGPT prompt example for dialogue act prediction. It
consists of task definition, output specification, formatting example and
dialogue history.

## Simulator Details

We implement an agenda-based simulator for SGD, following the prevalent
approach for user simulator design as outlined in previous works (Schatzmann
et al. 2007; Wang and Wong 2021; Kwan et al. 2023). The agenda-based simulator
samples a user goal according to the distribution of slots in the training set
of SGD. The user goal is kept unknown to the dialogue agent. It maintains a
stack (i.e. user agenda) that stores all the user actions that the user needs
to inform perform to achieve his/her goal during the conversation. It acts to
the system’s actions according to some pre-fined rules.

### Baseline Agents

We compare our model, JoTR, to five other models. 1) JOIE (Wang and Wong
2021), the current state-of-art(SOTA) on MultiWOZ, is a collaborative multi-
agent model that generates a single dialogue action without a predefined
action list. 2) MLP${}_{\text{ppo}}$, an agent optimized with PPO with fixed
action candidates. 3) SimpleTOD (Hosseini-Asl et al. 2020), a GPT-2-based
agent trained with supervised learning to generate dialogue actions along with
belief states and responses based on the dialogue history. 4) DASP
(Jhunjhunwala, Bryant, and Shah 2020), an LSTM-based agent that trained with
human supervision to select among N-best action candidates based on the
dialogue history. 5) ChatGPT is a large language model developed on the
foundation of InstructGPT (Ouyang et al. 2022). It functions as an efficient
conversational agent, capable of interpreting user prompts and producing
logically consistent replies. ChatGPT has demonstrated impressive results
across a diverse range of natural language processing assignments. We use it
to generate dialogue actions based on the dialogue action history using a
zero-shot prompt as shown in Table 1.

Figure 4: An illustrative dialogue example featuring responses from various
models is provided. The system’s dialogue actions (text on the right)
highlighted in yellow emphasize JoTR’s capability to manage complex and out-
of-domain user actions, which other models struggle to handle. The dialogue
actions in pink showcase JoTR’s proficiency in preemptively informing relevant
slots, an area where other models fall short. and

To further demonstrate the advantages of our model, JoTR, we also compare it
to three variants. JoTR${}_{\text{w/o rs}}$ does not use reward shaping.
JoTR${}_{\text{w/o ppo}}$ is only pretrained with supervised learning and not
further fine-tuned with PPO. JoTR${}_{\text{pretrained}}$ uses a pre-trained
BERT as the context encoder but with the weights fixed.

### Main Results

Model | MultiWOZ | SGD
---|---|---
Succ.$\uparrow$ | Turn$\downarrow$ | Rew.$\uparrow$ | Succ.$\uparrow$ | Turn$\downarrow$ | Rew.$\uparrow$
JOIE$\dagger$ | 0.91 | 18.90∗ | 40.82 | 0.51 | 11.10∗ | 15.32
MLP${}_{\text{ppo}}$ | 0.56 | 30.72 | -26.76 | 0.54 | 23.43 | 16.50
SimpleTOD$\ddagger$ | 0.62 | - | - | 0.50 | - | -
DASP$\dagger,\ddagger$ | 0.85 | - | - | 0.70 | - | -
ChatGPT | 0.73 | 13.10 | 41.05 | 0.50 | 11.04 | 15.48
JoTR | 0.93 | 9.94 | 68.46 | 0.79 | 15.23 | 49.25
JoTR${}_{\text{w/o rs}}$ | 0.89 | 9.95 | 66.42 | 0.72 | 16.53 | 38.84
JoTR${}_{\text{w/o ppo}}$ | 0.67 | 18.44 | 32.18 | 0.55 | 24.76 | 14.62
JoTR${}_{\text{pretrained}}$ | 0.76 | 14.19 | 44.87 | 0.64 | 19.25 | 28.18
Table 2:  Dialogue act modeling performance is measured by success rate
(Succ.), turn, and reward (Rew.) All the RL-based agents use the same reward
values and assignments during testing. Results with $\dagger$ are from
original papers. We reproduce JOIE to obtain the performance in SGD. Models
with $\ddagger$ are tested against a corpus. The best results are in bold. *In
our settings, a user-system utterance pair counts as 2 turns, while in the
JOIE paper, it’s 1 turn.  11footnotetext:
https://github.com/salesforce/simpletod

The learning process demonstrated in Figure 3 showcases the superior
performance and efficiency of the proposed JoTR model. Notably, JoTR
outperforms the previous SOTA model JOIE (0.93 vs 0.91 for MultiWOZ and 0.79
vs 0.51 for SGD) despite only trained with 50K frames, compared to JOIE’s 400K
frames (Wang and Wong 2021). Moreover, the ability of JoTR to improve
significantly with such short training makes it more suitable for real-world
applications. Note that we implement JOIE and obtain the result in SGD with
50K frames while the result in MultiWOZ is from the original paper.

Table 2 shows that JoTR requires significantly fewer turns than JOIE to
satisfy the user goal. This efficiency can be attributed to JoTR’s capacity to
generate multiple atomic actions in one turn, in contrast to JOIE’s single
action prediction. This characteristic not only reduces the total number of
interaction turns but also enhances JoTR’s practicality for everyday use. All
JoTR variants outperform MLP${}_{\text{ppo}}$. Notably, JoTR${}_{\text{w/o
ppo}}$ surpasses MLP${}_{\text{ppo}}$ without additional RL fine-tuning,
indicating that the strong learning capacity of transformer is effectively
harnessed for learning the specific structural properties of dialogue actions.
In comparison to SimpleTOD, JoTR performs better due to its use of dialogue
actions as input, which reduces noise and complexity compared to SimpleTOD’s
language utterances. Furthermore, JoTR’s encoder-decoder model structure aids
in capturing context information more effectively than SimpleTOD’s decoder-
only model. Lastly, JoTR performs significantly better than ChatGPT. Most
errors made by ChatGPT can be categorized into two categories: 1.
hallucination on domain, slot and values. 2. Violations of output format
constraints.

### Ablation Study

#### The Effectiveness of Reward Shaping

The success rate increase from 0.89 to 0.93 in MultiWoz and 0.72 to 0.79 in
SGD when reward shaping is applied. It manifests that reward shaping plays a
big role in achieving high success rate. Furthermore, Figure 3 demonstrates
that JoTR consistently maintains a higher success rate throughout the fine-
tuning process. This observation highlights the advantage of using a dense and
well-designed reward in RL fine-tuning, corroborating previous findings (Wang
et al. 2022a).

#### The Necessity of RL Fine-Tuning

JoTR${}_{\text{w/o ppo}}$ underperforms significantly without continuously
being optimized with RL, exhibiting a reduction of as much as 28% in the
success rate on MultiWOZ. This underlines the critical role of RL fine-tuning
in refining the behavior of the policy model through the reward.

#### The Importance of Training from Scratch

JoTR significantly outperforms JoTR${}_{\text{pretrained}}$ (0.93 vs 0.76 in
Multiwoz, 0.79 vs 0.64 in SGD). As evidenced by Figure 3,
JoTR${}_{\text{pretrained}}$ exhibits a notably lower success rate initially.
This can be attributed to the distinct structure of the dialogue actions’
input space, which markedly differs from the natural language space where the
model was originally pretrained. This discrepancy cannot be bridged
effectively by supervised training during the warm-up phase or reinforcement
learning in the fine-tuning phase.

### Case Study

Figure 5: An example user utterance accompanied by the language responses
corresponding to the dialogue actions predicted by various agents. We used the
rule-based NLG component in ConvLab2333https://github.com/thu-coai/ConvLab-2to
generate the language response given the predicted dialogue action.

As demonstrated in Figure 5, we observe that when the user requests a slot
combination that is never seen in the training set, both JoTR and
JoTR${}_{\text{w/o rs}}$ are able to successfully informed all requested
slots. This demonstrates their robust ability to generate effective and
efficient dialogue actions. In contrast, JoTR${}_{\text{w/o ppo}}$,
JoTR${}_{\text{w/o pretrained}}$, and SimpleTOD were unable to inform all
requested slots, potentially a reflection of their inferior performance
relative to JoTR. JOIE only informed one requested slot, likely due to its
design limitation of generating a single action per turn. Moreover,
MLP${}_{\text{ppo}}$ could not inform the complete slots as well, since the
dialogue action for informing address, postcode, and phone number is not found
within its predefined action set. Lastly, ChatGPT responded inappropriately,
for reasons elaborated in the preceding section.

We also provide a full dialogue example of various models interacting with the
user simulator in Figure 4. The user requested for the address, postcode and
phone of the park in the second turn. There is not a single training example
that request these three slots simultaneously, which serves to showcase
ability of different models to respond with complex and out-of-domain user
actions. In consistent with Figure 5, JoTR and JoTR${}_{\text{w/o rs}}$ were
able to inform all three slots while other models can’t (highlighted in
yellow) Furthermore, when the user requested for a guesthouse in turn three,
JoTR is able to provide the phone number without being explicitly asked while
JoTR${}_{\text{w/o rs}}$ and other models failed to do so as (highlighted in
pink). This illustrates the rewarding shaping can incentivize the model to
provide additional information preemptively. In this example, we can also see
the dialogues of other models are significantly longer than those of JoTR.
Therefore, JoTR is able to achieve the user’s goal efficiently.

### Human Evaluation

Model | Succ.(MultiWOZ)$\uparrow$ | Succ.(SGD)$\uparrow$
---|---|---
JOIE | 0.56 | 0.53
MLP${}_{\text{ppo}}$ | 0.52 | 0.56
SimpleTOD | 0.62 | 0.50
DASP | - | -
ChatGPT | 0.66 | 0.52
JoTR | 0.92 | 0.76
JoTR${}_{\text{w/o rs}}$ | 0.84 | 0.70
JoTR${}_{\text{w/o ppo}}$ | 0.66 | 0.56
JoTR${}_{\text{pretrained}}$ | 0.68 | 0.60
Table 3:  Human evaluation results. We use the models trained with 50K frames
for all agents.

We further conduct a human evaluation to validate the simulation results using
the models trained with 50K frames. We recruit 3 volunteer student helpers as
evaluators to interact with different models. For each model, we held 50
dialogue sessions. In each session, an evaluator is assigned a randomly
selected model and user goal. The evaluators are instructed to interact with
the model in accordance with the user goal, with a maximum of 20 turns per
session, aligning with the settings used in the experiments in previous
sections. At the end of each session, the evaluators assessed the success or
failure of the dialogue. The results are illustrated in Table 3, which are
consistent with the previous results using a user simulator.

## Conclusion

We introduced JoTR, a versatile framework for dialogue policy learning using
joint text-to-text transformer reinforcement learning. It trains word-level
policies that can generate dialogue actions without the need for predefined
templates. Empirical results from two benchmark datasets show that our model,
which does not rely on predefined action templates, outperforms the strongest
baseline in terms of both policy learning efficiency and dialogue action
quality as determined by simulated and human evaluations.

## References

  * Budzianowski et al. (2017) Budzianowski, P.; Ultes, S.; Su, P.-H.; Mrkšić, N.; Wen, T.-H.; Casanueva, I.; Rojas-Barahona, L. M.; and Gašić, M. 2017.  Sub-domain Modelling for Dialogue Management with Hierarchical Reinforcement Learning.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , 86–92. Saarbrücken, Germany: Association for Computational Linguistics.
  * Budzianowski and Vulić (2019) Budzianowski, P.; and Vulić, I. 2019.  Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , 15–22. Hong Kong: Association for Computational Linguistics.
  * Budzianowski et al. (2018) Budzianowski, P.; Wen, T.-H.; Tseng, B.-H.; Casanueva, I.; Ultes, S.; Ramadan, O.; and Gašić, M. 2018.  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gašić et al. (2010) Gašić, M.; Jurčíček, F.; Keizer, S.; Mairesse, F.; Thomson, B.; Yu, K.; and Young, S. 2010.  Gaussian Processes for Fast Policy Optimisation of POMDP-based Dialogue Managers.  In _Proceedings of the SIGDIAL 2010 Conference_ , 201–204. Tokyo, Japan: Association for Computational Linguistics.
  * Geishauser et al. (2022) Geishauser, C.; van Niekerk, C.; Lin, H.-c.; Lubis, N.; Heck, M.; Feng, S.; and Gašić, M. 2022.  Dynamic Dialogue Policy for Continual Reinforcement Learning.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , 266–284. Gyeongju, Republic of Korea: International Committee on Computational Linguistics.
  * He et al. (2022) He, W.; Dai, Y.; Zheng, Y.; Wu, Y.; Cao, Z.; Liu, D.; Jiang, P.; Yang, M.; Huang, F.; Si, L.; et al. 2022.  Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 36, 10749–10757.
  * Hosseini-Asl et al. (2020) Hosseini-Asl, E.; McCann, B.; Wu, C.-S.; Yavuz, S.; and Socher, R. 2020.  A Simple Language Model for Task-Oriented Dialogue.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 20179–20191. Curran Associates, Inc.
  * Jhunjhunwala, Bryant, and Shah (2020) Jhunjhunwala, M.; Bryant, C.; and Shah, P. 2020.  Multi-Action Dialog Policy Learning with Interactive Human Teaching.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , 290–296. 1st virtual meeting: Association for Computational Linguistics.
  * Kristianto et al. (2018) Kristianto, G. Y.; Zhang, H.; Tong, B.; Iwayama, M.; and Kobayashi, Y. 2018.  Autonomous Sub-domain Modeling for Dialogue Policy with Hierarchical Deep Reinforcement Learning.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , 9–16. Brussels, Belgium: Association for Computational Linguistics.
  * Kwan et al. (2023) Kwan, W.-C.; Wang, H.-R.; Wang, H.-M.; and Wong, K.-F. 2023.  A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-oriented Dialogue Policy Learning.  _Machine Intelligence Research_ , 1–17.
  * Lee et al. (2019) Lee, S.; Zhu, Q.; Takanobu, R.; Zhang, Z.; Zhang, Y.; Li, X.; Li, J.; Peng, B.; Li, X.; Huang, M.; and Gao, J. 2019.  ConvLab: Multi-Domain End-to-End Dialog System Platform.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 64–69. Florence, Italy: Association for Computational Linguistics.
  * Lee (2021) Lee, Y. 2021.  Improving End-to-End Task-Oriented Dialog System with A Simple Auxiliary Task.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , 1296–1303. Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * (14) Levin, E.; Pieraccini, R.; and Eckert, W. ????  Learning dialogue strategies within the Markov decision process framework.  In _1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings_ , 72–79.  148 citations (Semantic Scholar/DOI) [2021-11-09].
  * Li et al. (2017) Li, X.; Chen, Y.-N.; Li, L.; Gao, J.; and Celikyilmaz, A. 2017.  End-to-end task-completion neural dialogue systems.  _arXiv preprint arXiv:1703.01008_.
  * Li, Kiseleva, and de Rijke (2020) Li, Z.; Kiseleva, J.; and de Rijke, M. 2020.  Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems.  _arXiv preprint arXiv:2009.09781_.
  * Li et al. (2020) Li, Z.; Lee, S.; Peng, B.; Li, J.; Shayandeh, S.; and Gao, J. 2020.  Guided Dialog Policy Learning without Adversarial Learning in the Loop.  _arXiv preprint arXiv:2004.03267_.
  * Lipton et al. (2016) Lipton, Z. C.; Gao, J.; Li, L.; Li, X.; Ahmed, F.; and Deng, L. 2016.  Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks & Replay Buffer Spiking.  _CoRR abs/1608.05081_.
  * (19) Liu, B.; and Lane, I. ????  Iterative policy learning in end-to-end trainable task-oriented neural dialog models.  In _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 482–489.  67 citations (Semantic Scholar/DOI) [2022-03-29].
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35: 27730–27744.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Peng et al. (2021) Peng, B.; Li, C.; Li, J.; Shayandeh, S.; Liden, L.; and Gao, J. 2021.  Soloist: Building Task Bots at Scale with Transfer Learning and Machine Teaching.  _Transactions of the Association for Computational Linguistics_ , 9: 807–824.
  * Peng et al. (2018) Peng, B.; Li, X.; Gao, J.; Liu, J.; Chen, Y.-N.; and Wong, K.-F. 2018.  Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6149–6153. IEEE.
  * Peng et al. (2017) Peng, B.; Li, X.; Li, L.; Gao, J.; Celikyilmaz, A.; Lee, S.; and Wong, K.-F. 2017\.  Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2231–2240. Copenhagen, Denmark: Association for Computational Linguistics.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Rastogi et al. (2020) Rastogi, A.; Zang, X.; Sunkara, S.; Gupta, R.; and Khaitan, P. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 8689–8696.
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schatzmann et al. (2007) Schatzmann, J.; Thomson, B.; Weilhammer, K.; Ye, H.; and Young, S. 2007.  Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers_ , 149–152. Rochester, New York: Association for Computational Linguistics.
  * Schulman et al. (2017) Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017.  Proximal Policy Optimization Algorithms.  _arXiv:1707.06347 [cs]_.
  * (30) Singh, S.; Kearns, M.; Litman, D.; and Walker, M. ????  Reinforcement Learning for Spoken Dialogue Systems.  In _Advances in Neural Information Processing Systems_ , volume 12. MIT Press.
  * Su et al. (2016) Su, P.-H.; Gašić, M.; Mrkšić, N.; Rojas-Barahona, L. M.; Ultes, S.; Vandyke, D.; Wen, T.-H.; and Young, S. 2016.  On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2431–2441. Berlin, Germany: Association for Computational Linguistics.
  * Su et al. (2015) Su, P.-H.; Vandyke, D.; Gašić, M.; Mrkšić, N.; Wen, T.-H.; and Young, S. 2015.  Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems.  In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , 417–421. Prague, Czech Republic: Association for Computational Linguistics.
  * Su et al. (2022) Su, Y.; Shu, L.; Mansimov, E.; Gupta, A.; Cai, D.; Lai, Y.-A.; and Zhang, Y. 2022\.  Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 4661–4676. Dublin, Ireland: Association for Computational Linguistics.
  * Takanobu, Liang, and Huang (2020) Takanobu, R.; Liang, R.; and Huang, M. 2020.  Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 625–638. Online: Association for Computational Linguistics.
  * Takanobu, Zhu, and Huang (2019) Takanobu, R.; Zhu, H.; and Huang, M. 2019.  Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 100–110. Hong Kong, China: Association for Computational Linguistics.
  * Tang et al. (2018) Tang, D.; Li, X.; Gao, J.; Wang, C.; Li, L.; and Jebara, T. 2018.  Subgoal Discovery for Hierarchical Dialogue Policy Learning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2298–2309. Brussels, Belgium: Association for Computational Linguistics.
  * Wang, Peng, and Wong (2020a) Wang, H.; Peng, B.; and Wong, K.-F. 2020a.  Learning Efficient Dialogue Policy from Demonstrations through Shaping.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 6355–6365.
  * Wang, Peng, and Wong (2020b) Wang, H.; Peng, B.; and Wong, K.-F. 2020b.  Learning Efficient Dialogue Policy from Demonstrations through Shaping.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 6355–6365. Online: Association for Computational Linguistics.
  * Wang et al. (2022a) Wang, H.; Wang, H.; Wang, Z.; and Wong, K.-F. 2022a.  Integrating Pretrained Language Model for Dialogue Policy Evaluation.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6692–6696. IEEE.
  * Wang and Wong (2021) Wang, H.; and Wong, K.-F. 2021.  A Collaborative Multi-agent Reinforcement Learning Framework for Dialog Action Decomposition.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 7882–7889. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Wang et al. (2022b) Wang, Y.; Mishra, S.; Alipoormolabashi, P.; Kordi, Y.; Mirzaei, A.; Naik, A.; Ashok, A.; Dhanasekaran, A. S.; Arunkumar, A.; Stap, D.; Pathak, E.; Karamanolakis, G.; Lai, H.; Purohit, I.; Mondal, I.; Anderson, J.; Kuznia, K.; Doshi, K.; Pal, K. K.; Patel, M.; Moradshahi, M.; Parmar, M.; Purohit, M.; Varshney, N.; Kaza, P. R.; Verma, P.; Puri, R. S.; Karia, R.; Doshi, S.; Sampat, S. K.; Mishra, S.; Reddy A, S.; Patro, S.; Dixit, T.; and Shen, X. 2022b.  Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 5085–5109. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Le Scao, T.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Zhang et al. (2020) Zhang, Z.; Liao, L.; Zhu, X.; Chua, T.-S.; Liu, Z.; Huang, Y.; and Huang, M. 2020\.  Learning Goal-oriented Dialogue Policy with opposite Agent Awareness.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , 122–132. Suzhou, China: Association for Computational Linguistics.
  * Zhao et al. (2022) Zhao, X.; He, B.; Wang, Y.; Li, Y.; Mi, F.; Liu, Y.; Jiang, X.; Liu, Q.; and Chen, H. 2022.  UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialogues.  In _Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering_ , 13–22. Dublin, Ireland: Association for Computational Linguistics.
  * Zhu et al. (2020) Zhu, Q.; Zhang, Z.; Fang, Y.; Li, X.; Takanobu, R.; Li, J.; Peng, B.; Gao, J.; Zhu, X.; and Huang, M. 2020.  Convlab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems.  _arXiv preprint arXiv:2002.04793_.


# Multi-alpha Boson Gas state in Fusion Evaporation Reaction and Three-body
Force

Taofeng Wang1∗, Ziming Li1, R. B. Wiringa2, Minliang Liu3, Jiansong Wang3,
Yanyun Yang3, Qinghua He4, Zhiyu Sun3, Chengjian Lin5, M. Assié6,Y. Ayyad7,
D. Beaumel6, Zhen Bai3, Fangfang Duan3, Zhihao Gao3, Song Guo3, Yue Hu1,
Wei Jiang8, F. Kobayashi9, Chengui Lu3, Junbing Ma3, Peng Ma3,
P. Napolitani10, G. Verde11,12, Jianguo Wang3, Xianglun Wei3, Guoqing Xiao3,
Hushan Xu3, Biao Yang8, Runhe Yang3, Yongjin Yao1, Chaoyue Yu3,
Junwei Zhang3, Xing Zhang3, Yuhu Zhang3, Xiaohong Zhou3 1School of Physics,
Beihang University, Beijing 100191, China
2Physics Division, Argonne National Laboratory, Argonne, Illinois 60439, USA
3Institute of Modern Physics, Chinese Academy of Sciences, Lanzhou 730000,
China
4Department of Nuclear Science $\&$ Engineering, College of Material Science
and Technology,
Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
5China Institute of Atomic Energy, P.O. Box 275 (10), Beijing 102413, China
6IJCLab, Université Paris-Saclay, CNRS/IN2P3, 91405 Orsay, France
7Facility for Rare Isotope Beams, Michigan State University, East Lansing,
Michigan 48824, USA
8State Key Laboratory of Nuclear Physics and Technology, School of Physics,
Peking University, Beijing 100871, China
9Graduate School of Engineering Science, Osaka University, 1-3 Machikaneyama,
Toyonaka, Osaka 560-8531, Japan
10IPN, CNRS/IN2P3, Université Paris-Sud 11, Université Paris-Saclay, 91406
Orsay Cedex, France
11INFN Sezione di Catania, via Santa Sofia 64, I-95123 Catania, Italy
12Laboratoire des 2 Infinis - Toulouse (L2IT-IN2P3), Université de Toulouse,
CNRS, UPS, F-31062 Toulouse Cedex 9, France

###### Abstract

The experimental evidence for the $\alpha$ Boson gas state in the
11C+12C$\rightarrow$23Mg∗ fusion evaporation reaction is presented. By
measuring the $\alpha$ emission spectrum with multiplicity 2 and 3, we provide
insight into the existence of a three-body force among $\alpha$ particles. The
observed spectrum exhibited distinct tails corresponding to $\alpha$ particles
emitted in pairs and triplets consistent well with the model-calculations of
AV18-UX and chiral effective field theory of NV2-3-la*, indicating the
formation of $\alpha$ clusters with three-body force in the Boson gas state.

###### pacs:

Valid PACS appear here

††preprint: APS/123-QED

##  I Introduction

Fusion evaporation is a fascinating phenomenon that occurs when two atomic
nuclei collide and merge to form a compound nucleus, which subsequently
undergoes a de-excitation process by emitting particles. This process can be
analyzed and understood from various aspects, including the excitation energy,
spin of the compound nucleus and multiplicity of the emitted identical
particles [1-8]. By studying the emission patterns of particles and their
correlations with these parameters, ones can gain insights into the nuclear
structure, reaction mechanisms, and the properties of compound nuclei. This
knowledge is crucial for various fields, including nuclear astrophysics,
nuclear energy production, and the synthesis of superheavy elements.

The excitation energy determines the available energy for particle emission
and governs the probability and types of particles emitted during the de-
excitation process. The angular momentum associated with the colliding nuclei
is transferred to the compound nucleus upon fusion. The spin of the compound
nucleus influences the selection rules for subsequent particle emission. The
conservation of angular momentum dictates the possible spin states of the
emitted particles and their relative intensities. The spin distribution of the
compound nucleus is crucial in determining the probabilities and types of
particles emitted during the de-excitation process. The multiplicity of
emitted particles is determined by the excitation energy and spin of the
compound nucleus. Higher excitation energies and larger spin values generally
lead to higher multiplicities.

The $\alpha$ particle Boson gas dilute state [9, 10] refers to a specific
condition in fusion evaporation reactions where the $\alpha$ particles behave
like a gas of non-interacting particles. This approximation is particularly
applicable when the density of $\alpha$ particles within the colliding nuclei
is low, and the inter-alpha particle interactions can be neglected. The
$\alpha$ particle Boson gas model provides a useful framework for
understanding the initial stages of the reaction and the subsequent
evaporation process involving $\alpha$ particles. This is typically the case
in heavy-ion reactions involving light nuclei or when the incident energy is
sufficiently high to overcome the nuclear interaction barriers. Under these
conditions, the probability of inter-alpha particle collisions becomes small
enough that the $\alpha$ particles can be treated as quasi-independent
particles within the Boson gas model.

The Boson gas model assumes that the $\alpha$ particles within the colliding
nuclei follow a statistical distribution known as the Bose-Einstein
distribution [9], which characterizes the occupancy of energy levels within
the system. The Bose-Einstein distribution determines the average energy,
momentum, and occupation probability of the particles within the Boson gas
[9]. Within the $\alpha$ particle Boson gas model, the dilute state assumption
allows for a simplified treatment of the collision dynamics and subsequent de-
excitation process involving $\alpha$ particles emission.

The low $\alpha$ particle density and negligible inter-alpha particle
interactions enable a decoupling of the collision and evaporation stages,
simplifying the analysis. During the fusion stage, when the colliding nuclei
approach each other, the attractive nuclear force acts on the nucleons
individually, including the $\alpha$ particles. As a result, the $\alpha$
particles are treated as independent entities during the fusion process.
Following the formation of the compound nucleus, which consists of the fused
nuclei and any emitted particles, it rapidly equilibrates internally through
various nucleon-nucleon interactions, allowing it to reach a high excitation
energy. The compound nucleus then undergoes a de-excitation process, where it
emits particles, including $\alpha$ particles, until it reaches a more stable
state.

The three-body force among $\alpha$ particles [11-21] refers to the additional
nuclear force that arises from the interactions among multiple $\alpha$
particles in a system. It accounts for the collective interaction between the
constituent nucleons of the $\alpha$ particles, extending beyond the pairwise
interactions. When multiple $\alpha$ particles come close to each other, their
mutual interactions give rise to a three-body force that cannot be fully
explained by the pairwise interactions alone.

The three-body force affects various aspects of systems involving multiple
$\alpha$ particles. The additional attraction arising from the three-body
force enhances the stability of $\alpha$ clusters, contributing significantly
to their binding energy. Furthermore, the three-body force influences the
spatial arrangement and structure of systems involving multiple $\alpha$
particles. The interplay between the pairwise interactions and the three-body
force determines the overall spatial arrangement and clustering patterns of
$\alpha$ particles.

##  II Experimental procedure

The present experimental measurement was performed at the Radioactive Ion Beam
Line at the Heavy Ion Research Facility in Lanzhou (HIRFL-RIBLL) [22], as
shown in Fig. 1. A 60 MeV/nucleon 12C beam was transfed to bombard a 3.5 mm
9Be target to produce about 25 MeV/nucleon 11C secondary beam with a purity of
about 99$\%$ and an intensity of about 104 particles per second [23]. The beam
particles were identified in terms of $B\rho-$TOF$-\Delta E$ method with the
magnets and two plastic scintillator detectors in the beam line [22]. The 11C
secondary beam were bombarded on a 50 mg/cm2 carbon target to produce the
breakup reaction.

Figure 1:  The experimental equipment consisting of three PPACs for determing
the reaction position of beam particle on the target, the reaction products
were detected by a DSSD silicon detector combined with a 2$\times$2 CsI(Tl)
scintillators array, the decay $\gamma$s were detected by five LaBr3 (Ce) and
one NaI scintillator detectors.

Three parallel plate avalanche chambers (PPACs) with 50$\times$50 mm2 active
area and position resolutions of about 1 mm (FWHM) in both the $X$ and $Y$
directions were placed in front of the target to track the incident 11C beam
[22] and to subsequently get the reaction vertex in the target. $\alpha$
particles are detected by the zero-degree telescope system which consists of a
double-sided silicon strip detector (DSSD, of 148 $\mu$m in thickness and
50$\times$50 mm2 in cross-sectional area) with 32 strips on both front and
back sides, and a 2$\times$2 photodiode (PD) readout CsI (Tl) scintillator
(25$\times$25$\times$30 mm3 size for each unit) array. Each CsI (Tl)
scintillator is covered by two layers of high reflection Tyvek papers and a 10
$\mu$m aluminum coated Mylar film as window. The PD is coupled to CsI (Tl)
scintillator with the photoconductive silicone grease. The angular coverage of
the zero-degree telescope is about 0-9∘. Five LaBr3 (Ce) and one NaI
scintillator detectors were placed around the target to measure the decayed
$\gamma$s from the excited fragments. DSSD was utilized to record the
$\Delta$E energy and the position of the detecting fragments, therefore, the
emission angle may be obtained by combining with the reaction vertex in the
target. CsI (Tl) detection system provides the residual E energy of the
fragments. Particle identifications (PID) for $\alpha$s were performed using
$\Delta$E-E contour, as shown in Fig. 2. The energy resolution with sigma of
this $\Delta$E-E detection system is estimated to be $\sim$0.8 MeV from
numerical simulation.

Figure 2: Particle identification (PID) with $\Delta E$ strip-silicon detector
and $E$ CsI array.

##  III  Data analysis procedure

###  III.1 Analysis with statistical model and chiral effective field theory

The $\alpha$ spectra with the multiplicity equal to 2 and 3 depending on
$p_{c.m.}$ the $\alpha$ momentum in the center of mass system were sorted and
exhibitted in Fig. 3, in which the center value of the maxmium amplititude
with $p_{c.m.}$ momentum of 3$\alpha$ spectrum moves to the higher compare to
that of 2$\alpha$. This aspect reflects the higher local temperature of
3$\alpha$ according to Maxwell-Boltzmann theory of the root mean square speed
$\sqrt{\overline{v}^{2}}$$\approx$1.73$\sqrt{RT/M_{\alpha}}$, where $R$ is the
universal gas constant, $T$ is the gas temperature, $M_{\alpha}$ is the mass
of $\alpha$ particle.

The absolute slope value of the spectrum tail of 2$\alpha$ is larger than that
of 3$\alpha$ as shown in Fig. 3. The formation of such cluster configurations
can be understood in terms of a loosely multi-$\alpha$ bound system [24, 25].
Assuming a square well potential between them, the density distribution of
$\alpha$ is $\rho(r)=|\Psi(r)|^{2}\propto\frac{e^{-2\kappa r}}{r^{2}}$, where
$\Psi(r)$ is the $\alpha$ particle wave function. Its slope of the density
distribution tail in $r$ position coordinate space is determined by the
quantity $\kappa$. The momentum distribution of $\alpha$ obtained from the
Fourier transform for wave function $\Psi(r)$ can be expressed by
$f(p)=C/(p_{i}^{2}+\kappa^{2})$ in the momentum space.
$\kappa_{2\alpha}$$>$$\kappa_{3\alpha}$ is observed according to $f(p)$ from
the comparison of the the slopes of $3\alpha$ and $2\alpha$ in Fig. 3, it
hints the long tail distribution of $3\alpha$ in position coordinate space.

Figure 3: The spectra of 2$\alpha$ and 3$\alpha$ compared with the statistical
mode of PACE4 calculation [28] as well as the calculations with AV18-UX
potential combined by three-body force [26], besides togethers with the
calculation with chiral effective field theory of NV2-3-la* model [27].

The fusion evaporation process begins with the collision of two atomic nuclei
with enough energy to overcome the repulsive Coulomb barrier between them.
This collision leads to the formation of a compound nucleus, which is a
temporary, highly excited state of the combined system. Upon forming the
compound nucleus, the incident kinetic energy is converted into internal
energy, causing the compound nucleus to become even more excited. The
excitation energy is distributed among the nucleons within the nucleus,
resulting in the collective motion of the nucleons, such as vibrations and
rotations.

In the fusion evaporation process, the formation of a compound nucleus
involves the interaction between the colliding atomic nuclei and the
subsequent rearrangement of nucleons within the compound nucleus. This
rearrangement results in the establishment of a mean field and the presence of
local $\alpha$ two-body and three-body interactions. The formed mean field
refers to the average potential experienced by the nucleons within the
compound nucleus. It arises due to the collective interaction of all nucleons
and is influenced by the nuclear structure and the density distribution of
nucleons. The mean field is responsible for governing the motion and behavior
of individual nucleons within the compound nucleus.

In addition to the mean field, the fusion evaporation process also involves
local $\alpha$ two-body and three-body interactions. These interactions arise
due to the formation and subsequent decay of $\alpha$ particles within the
compound nucleus. This three-body interaction can influence the energies and
angular momenta of the nucleons involved and can influence the decay channels
and final states of the compound nucleus.

The local $\alpha$ two-body and three-body interactions are typically
incorporated into theoretical models and simulations of the fusion evaporation
process. These interactions are often described using effective nuclear
potentials or effective field theories, which aim to capture the essential
features of the $\alpha$-particle interactions within the compound nucleus. In
Fig. 3 the calculations of the adopted Aagonne v18 two-nucleon potential and
Urbana X three-nucleon potentials [26] as well as the calculations with
Nv2-3-la∗ chiral effective field theory ($\chi$EFT) [27] including three-
nucleon force are consistent well to the tail slope of 3$\alpha$. The fusion
evaporation statistical model PAEC4 [28] for the calculation are partly
consistent to the tail slopes of 2$\alpha$ and 3$\alpha$ spectra. The three-
body interaction among $\alpha$ particles arises from the complex interplay
between the nucleons within each $\alpha$ particle and the collective
interactions among the $\alpha$ particles themselves. The three-body force
cannot be fully explained by the pairwise interactions alone and requires a
more advanced theoretical treatment to capture its effects accurately.
$\chi$EFT has been utilized to study the three-body force among alpha
particles. It combines the principles of chiral symmetry and effective field
theory and provides a systematic expansion for nuclear interactions, including
the three-body force. By applying $\chi$EFT, ones have been able to derive
interactions among alpha particles that accurately reproduce experimental data
and provide valuable insights into the three-body force. These aspects
indicate that the effects of three-body interaction among $\alpha$ particles
prefer to the formation of multiple $\alpha$ particles dilute state in the
final fusion process of 11C+12C$\rightarrow$23Mg∗ reaction.

Figure 4: The cross section ratio of $\sigma_{3\alpha}/\sigma_{2\alpha}$
dependent on the $\alpha$ momentum $p_{c.m.}$, together with the calculations
of NV2+3+la* and AV18+UX models, besides a phenomenological curves with
different widths reflect $\alpha$ three-body force dependent on $p_{c.m.}$.

###  III.2 Enhancement of three-body force with $p_{c.m.}$

In fusion evaporation reactions, the cross section ratio of the 3$\alpha$
spectrum to the 2$\alpha$ spectrum can provide valuable information about the
dynamics and properties of the reaction. This ratio of
$\sigma_{3\alpha}/\sigma_{2\alpha}$ is dependent on the $\alpha$ momentum
$p_{c.m.}$ and exhibits an increasing trend as shown in Fig. 4. The cross
section ratio of the 3$\alpha$ spectrum to the 2$\alpha$ spectrum refers to
the relative number of events or probabilities of observing three $\alpha$
particles versus two $\alpha$ particles in the final state of a fusion
evaporation reaction. The dependence of this ratio on the $\alpha$ momentum
$p_{c.m.}$ arises from the underlying dynamics of the fusion evaporation
process. As the $\alpha$ momentum increases, the kinetic energy available for
the system also increases. This higher kinetic energy allows for more
energetic collisions and greater excitation of the compound nucleus formed
during the reaction.

The strong nucleon-nucleon interaction was formulated by the Yukawa theory of
multi-meson exchange in the short distance, and one pion meson exchange in the
large distance case. Two nucleon One Pion Exchange Potential (OPEP)[29] was
described by

$\begin{split}V^{1\pi}_{2N}=g^{2}\frac{m_{\pi}}{12M^{2}\hbar}\bm{\tau}(1)\cdot\bm{\tau}(2)[\bm{\sigma}_{1}\cdot\bm{\sigma}_{2}+S_{12}(1+\frac{2}{x}+\frac{3}{x^{2}})\frac{e^{-x}}{x}],\end{split}$
(1)

where $x=\frac{m_{\pi}c}{\hbar}r$, $r$ is the distance of two particles,
$M=\frac{1}{2}(m_{p}+m_{n})$ is the reduced mass of nucleon,
$\sigma_{i}$($\tau_{i}$) is the Cartesian component of spin(isospin) operators
of nucleon $i$, $S_{12}$ denotes the non-central interaction of tensor force
term. On the other hand, three-body one pion exchange potential was shown as
[11, 30]

$\begin{split}V^{1\pi}_{3N}=-\frac{g_{A}}{8f^{2}_{\pi}}\frac{c_{D}}{f^{2}_{\pi}\Lambda_{\chi}}\sum_{i\neq
j\neq
k}\frac{\bm{\sigma}_{j}\cdot\bm{Q}_{j}}{Q^{2}_{j}+m^{2}_{\pi}}(\bm{\tau}_{i}\cdot\bm{\tau}_{j})(\bm{\sigma}_{i}\cdot\bm{Q}_{j}),\end{split}$
(2)

where $\bm{Q}_{j}=\bm{k}^{{}^{\prime}}_{i}-\bm{k}_{i}$ is the transfer
momentum, i.e. the difference between the initial and final single-particle
momenta ($\bm{k}^{{}^{\prime}}_{i}$ and $\bm{k}_{i}$ respectively). Since the
interactions among $\alpha$ particles origiante from the residule interactions
of two-nucleon force and three-nucleon force within $\alpha$ particle, the
mutual $\alpha$ interactions still can be expressed by two-body and three-body
force with Eq. (1) and (2), named by $V^{1\pi}_{2\alpha}$ and
$V^{1\pi}_{3\alpha}$. $V^{1\pi}_{2\alpha}$ is position $r$ dependence, while
$V^{1\pi}_{3\alpha}$ is momentum $Q_{j}$ dependence. Thereby, the potential
ratio of $V^{1\pi}_{3\alpha}$ to $V^{1\pi}_{2\alpha}$ is momentum $Q_{j}$
positive dependence that leads to the enhancement of three-body force effects
at high momentum transfer, it is clearly reflected by the plots of
$\sigma_{3\alpha}/\sigma_{2\alpha}$ with $p_{c.m.}$ in Fig. 4.

A phenomenological calculation has been parametrized in terms of the
difference between single-particle momentum and Fermi momentum $p_{F}$ and
distribution width. As shown in Fig. 4, the maximum of the increase ratio of
$\sigma_{3\alpha}/\sigma_{2\alpha}$ around Fermi momentum $p_{F}$ indicates
the three-body force is favorable of high momentum admixture.

$\begin{split}\sigma_{3\alpha}/\sigma_{2\alpha}=Ce^{-\frac{(p-p_{F})^{2}}{w^{2}}},\end{split}$
(3)

where $p$ equal to $\alpha$ momentum $p_{c.m.}$, $C$=0.085, the Fermi momentum
$p_{F}$=200 MeV/c, $w$=70, 80, 90 MeV/c, these plots are consistent well to
the cross section ratio $\sigma_{3\alpha}/\sigma_{2\alpha}$ of this
measurement. These phenomenological calculations indicate that the momentum
dependent three-body force among multi-alpha particles are essentially
enhanced in high momentum region, especially near Fermi momentum. The
fevorable $\alpha$ dilute Boson gas state, thereby, is formed in the final
process of fusion reaction.

Figure 5: The spectra of 2$\alpha$ and 3$\alpha$ compared with the models of
2$\alpha$+residule nucleus, 3$\alpha$+residule nucleus and all of Helium Boson
gas final state.

The increasing trend of the cross section ratio with $\alpha$ momentum
$p_{c.m.}$ is a consequence of the statistical nature of the fusion
evaporation process. As the excitation energy of the compound nucleus
increases, the available phase space for decay pathways involving three
$\alpha$ particles becomes more favorable. This increasing trend can be
explained by considering the probability distributions and partitioning of
energy and angular momentum in the compound nucleus. As the alpha momentum
increases, the compound nucleus has a higher probability of accessing energy
states that favor the emission of multiple alpha particles, leading to an
enhanced 3$\alpha$ spectrum compared to the 2$\alpha$ spectrum.

For deeply understanding this experimental results, firstly, the excess proton
of 11C nucleus leads to an increased Coulomb repulsion between the protons in
the collision, making the compound nucleus less stable. Secondly, the $\alpha$
particle is a tightly bound configuration in the nuclear structure. It has a
high binding energy, which means it requires a large amount of energy to break
apart. This stability makes the $\alpha$ particle emission a favorable decay
channel for the compound nucleus, as it is less likely to undergo further
reactions or break apart once emitted. Furthermore, the proton-rich projectile
in a fusion evaporation reaction leads to the formation of an $\alpha$ dilute
state in the compound nucleus due to the energetically favorable nature of
three-body $\alpha$ correlation .

Figure 6: The correlation of relative momentum $P_{\alpha_{i}\\_\alpha_{j}}$
and relative open emission angle $\Theta_{\alpha_{i}\\_\alpha_{j}}$ for
$\alpha_{1}$$\\_$$\alpha_{2}$, $\alpha_{2}$$\\_$$\alpha_{3}$ and
$\alpha_{3}$$\\_$$\alpha_{1}$ interactions.

###  III.3 Evidence of $\alpha$ Boson Gas state

The results of Boson gas model calculations are consistent with the
experimental 2$\alpha$ and 3$\alpha$ spectra in fusion evaporation reactions
as shown in Fig. 5, it suggests that the model accurately captures the
energetics and behavior of $\alpha$ particles within the compound nucleus.
This consistency indicates that the $\alpha$ cluster structure and $\alpha$
two-body and three-body interactions play a significant role in the reaction.
On the other hand, if calculations using models that include additional
residual particles, such as the 2$\alpha$+residue and 3$\alpha$+residue
models, deviate from the experimental 2$\alpha$ and 3$\alpha$ spectra, it
suggests that the involvement of these additional residule nucleus has a
limited impact on the observed spectra.

The deviation from the 2$\alpha$+residue and 3$\alpha$+residue models implies
that the multi-$\alpha$ particles play a dominant role in the reaction
dynamics, and their interactions are the primary driving force for the
observed spectra. This deviation highlights the importance of the $\alpha$
cluster structure and the dilute state in the fusion evaporation process of
23Mg nucleus.

##  IV Discussion and conclusion

The relative momentum $P_{\alpha_{i}\\_\alpha_{j}}$ of $\alpha$$\alpha$
clusters is probably larger than that of normal or long range correlated $pp$
pair in nuclei because of the larger Coulomb force between 2$\alpha$.
$P_{\alpha_{i}\\_\alpha_{j}}$ reflects the correlation strength of each
$\alpha_{i}\alpha_{j}$ pair in 3$\alpha$ configuration. Measurements for
$P_{\alpha_{i}\\_\alpha_{j}}$ can supply the significant clues for probing the
properties of the initial 3$\alpha$ state. The weighted vertex of 3$\alpha$
interaction corresponds to the magnitudes of the relative momentums and
relative emission angles of $\alpha_{1}$-$\alpha_{2}$,
$\alpha_{2}$-$\alpha_{3}$ and $\alpha_{3}$-$\alpha_{1}$, as shown in Fig. 6.
Generally, the position of weighted vertex reflects the configuration of
3$\alpha$ with an isosceles, equilateral or obtuse triangle shape. The
sequence of $i$ for $\alpha_{i}$ was sorted according to the measured energies
with the increase trend.

The correlation of relative momentum $P_{\alpha_{i}\\_\alpha_{j}}$ and the
relative open emission angle $\Theta_{\alpha_{i}\\_\alpha_{j}}$ are shown in
Fig. 6. The similar magnitudes of
$P_{\alpha_{1}\\_\alpha_{2}}$$\approx$$P_{\alpha_{2}\\_\alpha_{3}}$$\approx$216
MeV/c correspond to the approximately equal magnitudes of
$\Theta_{\alpha_{1}\\_\alpha_{2}}$$\approx$$\Theta_{\alpha_{2}\\_\alpha_{3}}$$\approx$118∘,
while the larger $P_{\alpha_{3}\\_\alpha_{1}}$ of 305 MeV/c corresponds to the
broader $\Theta_{\alpha_{3}\\_\alpha_{1}}$ of 151∘. The two equal smaller
relative momentum indicate the longer interaction range for $\alpha_{1}$ to
$\alpha_{2}$ and $\alpha_{2}$ to $\alpha_{3}$, while the larger relative
momentum denotes the short interaction range for $\alpha_{3}$ to $\alpha_{1}$.
Therefore, the connections of $\alpha_{1}$$\\_$$\alpha_{2}$,
$\alpha_{2}$$\\_$$\alpha_{3}$ and $\alpha_{3}$$\\_$$\alpha_{1}$ make up an
isosceles triangle.

In conclusion, the study of the $\alpha$ three-body force and its impact on
the dilute Boson Gas state of the fusion evaporation final process of compound
nucleus 23Mg has shed light on the complex dynamics involved in nuclear
reactions. Through the analysis of experimental data and theoretical
calculations, it has been demonstrated that the $\alpha$ three-body force
plays a significant role in determining the properties of excited compound
nuclei and the subsequent decay processes.

##  V Acknowledgement

We would like to acknowledge the staff of HIRFL for the operation of the
cyclotron. The author T. Wang appreciates for the financial supports from
China Scholarship Council. This work has also been supported by the National
Natural Science Foundation of China (No. 10175091 and No. 11305007).

## References

  * (1) J. F. Liang, D. Shapira, C. J. Gross, J. R. Beene, J. D. Bierman, A. Galindo-Uribarri, J. Gomez del Campo, P. A. Hausladen, Y. Larochelle, W. Loveland, P. E. Mueller, D. Peterson, D. C. Radford, D. W. Stracener, and R. L. Varner, Phys. Rev. Lett. 91, 152701 (2003).
  * (2) W. Loveland, A. M. Vinodkumar, R. S. Naik, P. H. Sprunger, B. Matteson, J. Neeway, M. Trinczek, M. Dombsky, P. Machule, D. Ottewell, D. Cross, K. Gagnon, and W. J. Mills, Phys. Rev. C 74, 064609 (2006).
  * (3) A. Lemasson, A. Shrivastava, A. Navin, M. Rejmund, N. Keeley, V. Zelevinsky, S. Bhattacharyya, A. Chatterjee, G. de France, B. Jacquot, V. Nanal, R. G. Pillay, R. Raabe, and C. Schmitt, Phys. Rev. Lett. 103, 232701 (2009).
  * (4) J. J. Kolata, A. Roberts, A. M. Howard, D. Shapira, J. F. Liang, C. J. Gross, R. L. Varner, Z. Kohley, A. N. Villano, H. Amro, W. Loveland, and E. Chavez, Phys. Rev. C 85, 054603 (2012).
  * (5) Z. Kohley, J. F. Liang, D. Shapira, C. J. Gross, R. L. Varner, J. M. Allmond, J. J. Kolata, P. E. Mueller, and A. Roberts, Phys. Rev. C 87, 064612 (2013).
  * (6) Varinderjit Singh, J. Vadas, T.K. Steinbach, B.B. Wiggins, S. Hudan, R.T. deSouza, Zidu Lin, C.J. Horowitz, L.T. Baby, S.A. Kuvin, Vandana Tripathi, I. Wiedenhöver, A.S. Umar, Phys. Lett. B, 765, 99 (2017).
  * (7) A. Lopez-Martens, A.V. Yeremin, M.S. Tezekbayeva, Z. Asfari, P. Brionnet, O. Dorvaux, B. Gall, K. Hauschild, D. Ackermann, L. Caceres, M.L. Chelnokov, V.I. Chepigin, M.V. Gustova, A.V. Isaev, A.V. Karpov, A.A. Kuznetsova, J. Piot, O.N. Malyshev, A.G. Popeko, Yu.A. Popov. Steinegger, Phys. Lett. B, 795, 271 (2019).
  * (8) Daniele Dell’Aquila, Brunilde Gnoffo, Ivano Lombardo, Luigi Redigolo, Francesco Porto, Phys. Lett. B, 837, 137642 (2023).
  * (9) F A Deeney and J P O’Leary, Eur. J. Phys., 33, 677 (2012).
  * (10) Subir Sachdev, Dilute Fermi and Bose Gases, Arxiv 1105.1793.
  * (11) Kai Hebeler, Three-nucleon forces: Implementation and applications to atomic nuclei and dense matter, Phys. Rept. 890, 1 (2021).
  * (12) P. Bocchieri and C. A. Orzalesi, The Ground-state Three-body Correlation Function in a Dilute Boson Gas with Hard-Sphere Interaction, IL NUOVO CIMENTO, Vol. XXXIII, No. 1, 1964.
  * (13) R. Rajaraman, Three-body problem in nuclear matter, Rev. of Mord. Phys., 39, 745 (1967).
  * (14) Guillaume Hupin, Sofia Quaglioni, and Petr Navratil, Phys. Rev. Lett., 114, 212502 (2015).
  * (15) A. Cipollone, C. Barbieri, and P. Navratil, Phys. Rev. Lett., 111, 062501 (2013).
  * (16) G. Hagen, M. Hjorth-Jensen, G. R. Jansen, R. Machleidt, and T. Papenbrock, Phys. Rev. Lett., 108, 242501 (2012).
  * (17) Takaharu Otsuka, Toshio Suzuki, Jason D. Holt, Achim Schwenk, and Yoshinori Akaishi, Phys. Rev. Lett., 105, 032501 (2010).
  * (18) Petr Navratil and W. Erich Ormand, Phys. Rev. Lett., 88, 152502 (2002).
  * (19) P. F. Bedaque, H.-W. Hammer, and U. van Kolck, Phys. Rev. Lett., 82, 463 (1999).
  * (20) V. Soma, A. Cipollone, C. Barbieri, P. Navratil, and T. Duguet, Phys. Rev. C, 89, 061301(R) (2014).
  * (21) K. Hebeler and A. Schwenk, Phys. Rev. C, 82, 014314 (2010).
  * (22) Z. Sun, W.-L. Zhan, Z.-Y. Guo, G. Xiao, and J.-X. Li, RIBLL, the radioactive ion beam line in Lanzhou, Nucl. Instrum. Methods Phys. Res., Sect. A, 503, 496 (2003).
  * (23) Z. M. Li, J. Zhu, T. F. Wang et al., Cluster structure of 11C investigated with a breakup reaction, Phys. Rev. C, 107, 014320 (2023).
  * (24) I. Tanihata, Neutron halo nuclei, J. Phys. G: Nucl. Part. Phys., 22, 157 (1996).
  * (25) I. Tanihata, H. Savajols, R. Kanungo, Recent experimental progress in nuclear halo structure studies, Prog. Part. Nucl. Phys., 68, 215 (2013).
  * (26) R. B. Wiringa, R. Schiavilla, Steven C. Pieper, and J. Carlson, Phys. Rev. C 89, 024305 (2014).
  * (27) M. Piarulli, S. Pastore, R. B. Wiringa, S. Brusilow, and R. Lim, Phys. Rev. C 107, 014314 (2023).
  * (28) J. Vadas,et. al., Phys. Rev. C 92, 064610 (2015).
  * (29) J. L. Friar, B. F. Gibson, and G. L. Payne, Phys. Rev. C 30, 1084 (1984).
  * (30) Humberto Garcilazo, Phys. Lett. B, 99, 195 (1981).